Researchers Crack Open GPT-OSS-20b to Find Hidden Symbol Systems Inside
An 8-year interpretability project shows that LLM representations can be closely approximated by symbolic role-filler structures, enabling precise behavioral edits.
PRO- New paper shows LLM vector representations implicitly encode symbolic Tensor Product structure across seven major models.
- DISCOVER method replaces a network's representations with a closed-form symbolic equation, preserving behavior.
- Tested on GPT-OSS, Gemma-3, Qwen3, Llama-3.1, OLMo-2, Pythia, GPT-2 across arithmetic, logic, code, language tasks.
- Causal edits to role-filler pairs change model output predictably, averaging 90.3% intervention accuracy across 31 tests.
- DISCOVER generalizes to unseen role-filler combinations, evidence of systematic binding rather than memorization.
- Partial code released at github.com/tommccoy1/discover; full release pending approval.
Modern language models operate on continuous vectors yet excel at tasks that traditionally required discrete symbol manipulation: arithmetic, logic, code execution, syntax. A new paper from R. Thomas McCoy, Paul Soulos, Tal Linzen, and Paul Smolensky offers a concrete resolution to that apparent paradox. Inside those messy vectors, neural networks are quietly building something that looks like a symbol system, and you can swap the real vectors out for a closed-form symbolic equation without breaking the model.
The paper is the culmination of an eight-year project. Its central claim: the vector representations of a wide range of neural networks can be closely approximated by symbolic structures, meaning you can replace a network's entire representation-generating process with a closed-form equation instantiating a symbolic structure while the network's behavior remains largely unchanged. That holds not just for toy recurrent nets but for seven modern LLMs including Gemma-3-27b, GPT-OSS-20b, Qwen3-14b, OLMo-2-13B, and Llama-3.1-8b.
The binding problem vectors were never supposed to solve
Symbolic AI has a long history of representing sentences as trees, math as expressions, and logic as formulas. Neural networks encode everything as points in high-dimensional space instead. Addition is order-insensitive, so if you just sum embeddings, cats chase dogs and dogs chase cats look identical. Cognitive scientists call this the binding problem, and it's why the Linear Representation Hypothesis, the idea that concepts are directions in vector space, can't fully explain how models handle structure.
The authors' hypothesis is that networks solve this using Tensor Product Representations (TPRs), a formalism introduced by Smolensky in 1990. In a TPR, a structure is decomposed into fillers (the content, like the word "cats") paired with roles (the position, like "subject"). Each filler-role pair is combined via tensor product and the pairs are summed, producing a single vector that preserves who-goes-where.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
