Skip to main content
Back

LLMs Are not Black Boxes Anymore

READ FULL PAPER

May 23, 2024 6 min read

Hey,

Welcome to today's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

.

IN TODAY'S SIGNAL

Top Paper

Kolena

  • Learn how to Master RAGs: a free webinar on the performance and pitfalls of RAG systems.

Trending Signals

Top of HuggingFace

  • MiniCPM-Llama3: A model that surpasses GPT-4V-1106 in OCRBench with 8B parameters and a 700+ score.

  • CogVLM2: A model excelling in TextVQA and DocVQA benchmarks with 8k content support.


  • AutoQuizzer: A tool that generates quizzes from URLs and evaluates LLMs' web and "closed book" performance.

  • PaliGemma Demo: A model that performs question answering and image captioning, fine-tuned on academic datasets.

  • Tokenizer Arena: A benchmark that visualizes and compares tokenizers of models like GPT-4 with bar plots.

Top Tweet

Read Time: 6 min 23 sec

Enjoying this newsletter?
Please forward it to a friend or colleague. It helps us keep this content free.

TOP PAPER

Safety

Anthropic releases groundbreaking AI safety research to map the "mind" of LLMs

⇧ 3128 Likes

What's New

Anthropic has just released a groundbreaking paper that inspects a Large Language Model from the inside for the very first time.


The Problem of Neural Networks as Black Boxes


Until now, such models acted like black boxes: input producing an output without a clear understanding of the reasons that lead to it.


More specifically, as you might know, these models are composed of neurons that are or are not activated by activation functions. The combination of neurons that get activated when a certain input is passed make up features, and the combination of features create an internal state of the model.


Well, most LLM neurons are uninterpretable, which stops us from mechanistically understanding the models.


A new technique to map the internal states of the model


The Anthropic team used a technique called "dictionary learning", borrowed from classical machine learning, which isolates patterns of neuron activations that recur across many different contexts. In turn, any internal state of the model can be represented in terms of a few active features instead of many active neurons, which means decomposing the model in features.


This technique was previously used on very tiny models and showed promising results. This is what motivated the researchers to apply the “dictionary learning” technique to Claude 3 Sonnet. And the results are mind-blowing.


The Anthropic team was able to extract millions of features which creates a conceptual map of its internal states halfway through the models computation. These features are remarkably abstract, often representing the same concept across contexts and languages, even generalizing to image inputs.


What the inside of an LLM looks like


What they found was that that the distance between similar features reflected their similarity. In other words, the model has a mapping of concepts that are as close to each other as their associated features.


For examples, looking near a feature related to the concept of "inner conflict", they find features related to relationship breakups, conflicting allegiances, logical inconsistencies, and so on.


But what’s even more interesting is that it is possible to artificially manipulate those features, and that emphasizing or deemphasizing certain features changes the behavior of the model. This is a groundbreaking result and a huge step forward for ensuring the safety of these models.


For example, researchers found a specific feature associated with blindly agreeing with the user. Artificially activating this feature completely alters the response and behavior or the model.

Why It Matters

This opens the opportunity to map all features of LLMs and be able to manipulate them to ensure more safety, for example by banning some features and activating others artificially.


However, there is still work to be done as mapping all existing features would require more computation that training the model itself. Plus, knowing the activations and their significance doesn’t give us the circuits they are involved in. And lastly, nothing says for sure that this technique would actually make AI models safer, even if this approach seems promising.

Community Response

Art Intelligence
Impressive. Any hope we could see the "features" displayed alongside every Claude answer? That would be incredibly useful for fine tuning a prompt. For example if an unrelated or unwanted 'feature' is activated by our prompt, we can change the prompt to explicitly avoid it.
Thomas Wolf
The new interpretability paper from Anthropic is totally based. Feels like analyzing an alien life form.
The Singularity Project
Mechanistic interpretability is never going to be complete for the same reason that human minds cannot be entirely mapped and predicted. Neither are purely computational in nature. Both rely upon a significant degree of non-computable emergence to produce Intelligence.
Read full paper

Webinar: Learn how to Master RAGs, Performance, Pitfalls and How to Avoid Them

Hallucinations, prompt injection, and PII leakage, oh my! Join Kolena’s webinar for a discussion on pitfalls in building a Retrieval-Augmented Generation (RAG) system. We’ll cover failure modes for a RAG system + benchmarking and evaluation techniques for mitigating these failures.


Register now to understand one of the fastest-growing use cases for LLMs, and how to unlock the opportunities it provides without falling prey to its wide range of challenges.

Register now

partner with us

TRENDING SIGNALS

Multimodal Models 

Microsoft releases an open-source version of sudo for Windows

⇧ 860 Likes

Language Models

Mistral releases new version of its flagship Model Mistral 70B v0.3

⇧ 614 Likes

ChatGPT

OpenAI partners with News Corp to integrate content to ChatGPT

⇧ 2911 Likes

Industry

Yann LeCun advises youth to not work on LLMs but next-gen AI systems

⇧ 7201 Likes

Industry

New feature on HuggingFace lets you directly launch local LM apps and calculate your TFLOPS

⇧ 2791 Likes

TOP OF HUGGINGFACE

Multimodal Models

MiniCPM-Llama3-V-2_5

⇩ 8123 Downloads

MiniCPM-Llama3-V 2.5 is the latest model in the MiniCPM-V series. The model is built on SigLip-400M and Llama3-8B-Instruct with a total of 8B parameters. It shows:

  • Leading Performance: With only 8B parameters, it surpasses widely used proprietary models like GPT-4V-1106, or Claude 3 and and greatly outperforms other Llama 3-based MLLMs.

  • Strong OCR Capabilities: can process images with any aspect ratio and up to 1.8 million pixels achieving an 700+ score on OCRBench, surpassing proprietary models such as GPT-4o. 

  • Efficient Deployment: systematically employs optimization technique such as quantization to achieve high-efficiency deployment on edge devices.

Multimodal Models

CogVLM2-llama3-chat-19B

⇩ 2645 Downloads

This is a new generation of CogVLM models, a family of lightweight multimodal models.


Compared to the first version, this model shows significant improvements in many benchmarks such as TextVQA, DocVQA. It supports an 8k content length and up to 1.8M pixel image resolution. This space also provide an open source model version that supports both Chinese and English.

AI Tools

AutoQuizzer

⇧ 84 Likes

This space lets you generate a quiz from a simple copy pasted URL. By feeding a URL, the model will automatically generate a quiz to let you test your knowledge and understanding.


You can play the quiz, or let the LLM play it.


The space also provides evaluations of LLMs playing the quiz, with both access to Web Browsing and in a "closed book" setting. 

AI Tools

PaliGemma Demo

⇧ 168 Likes

PaliGemma is an open vision-language model that was recently unveiled by Google.


This space includes models fine-tuned on top of PaliGemma on a mix of downstream tasks, such as question answering, captioning and so on. The fine-tuning is performed on a series of academic datasets.


You can try out the models directly on this space by proving text or image inputs.

Benchmarks

Tokenizer Arena

⇧ 40 Likes

After the Chatbot Arena, it's the turn of the Tokenizer Arena to be trending on HuggingFace. 


This space lets you compare and visualize the tokenizers of different models such as GPT-4, Phi-3 or Grok.


You can select multiple models, type a prompt and have a visualization of the number of tokens, the token IDs, and even the attention mechanism. A comparison of the different models is automatically generated in the form of a bar plot with a score for each model.

TOP TWEET

This recent tweet presents two new programming languages designed to bridge the gap between high-level programming and efficient execution on modern hardware accelerators like GPUs and Apple Silicon.


The first language mentioned is Bend, which compiles modern high-level language features like lambdas, closures, recursion, and algebraic data types to native multi-threading on Apple Silicon or NVIDIA GPUs. It supports complex constructs that are typically challenging for compilers to optimize. Bend compiles to a thread-safe runtime called HVM2, implemented in Rust.


The second language is Mojo, a CUDA-flavored, Python-like language that executes at C speed. Mojo is conceptually lower-level than Bend and allows more control over parallelism, making it well-suited for coding modern neural network accelerations by hand.


The tweet provides links to the open-source repositories for Bend and Mojo, as well as an example of the Llama2 model implemented in Mojo.

tw profile: Santiago Jim Fan @DrJimFan

AI is math. GPU is metal. Sitting between math and metal is a programming language. Ideally, it should feel like Python but scale like CUDA. I find two newcomers in this middle layer quite exciting:

 
 

2757 Likes   461 Retweets   53 Replies

Read tweet

Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.

Thank You.

Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?

Sponsor