Skip to main content
Back

StructLens Reveals Why Low-Resource Languages Stay Structurally Distant Inside LLMs

A new structural analysis of LLMs reveals how low-resource languages are processed differently inside the model, and what language-specific fine-tuning actually changes

Paper, Full Access
  • New structural analysis: Researchers use tree-based representations (StructLens) to compare how LLMs internally organize different languages, going beyond token-level cosine similarity.
  • Low-resource gap confirmed structurally: Languages like Guarani and Jinghpaw are structurally far more distant from English inside LLMs than German or Chinese, even when cosine similarity looks similar.
  • Post-training is a local operation: Continued pre-training on Japanese reshapes that language's internal structure without disturbing the relative relationships of other languages.
  • Three models tested: Gemma3 4B, Qwen3 8B, and OLMo3 7B (plus larger variants) all show the same pattern, ruling out architecture-specific artifacts.
  • Practical use: Findings inform CPT decisions and multilingual model diagnostics; StructLens is also useful for layer pruning.
  • Paper and StructLens framework: Available at arxiv.org/abs/2606.01800 and arxiv.org/abs/2603.03328.

Most research on how LLMs handle multiple languages has focused on individual token embeddings -- measuring whether the vector for the word "dog" in French sits close to "dog" in English. But language is more than a bag of tokens. It has structure: the way words relate to each other across a sentence is itself a signal. A new paper from researchers at Nara Institute of Science and Technology takes that structural view seriously, and what they found changes how we should think about multilingual LLMs.

The core finding is stark: low-resource languages are structurally more different from English than high- and mid-resource languages, and language-specific post-training alters their structures while preserving inter-language relationships. That second part -- the preservation of relationships -- is the real surprise.

Why token-level analysis wasn't enough

Previous interpretability work on multilingual models has mostly asked: do different languages share the same internal representations? Researchers would compare token embeddings using cosine similarity and call it a day. The problem is that this approach misses something fundamental.

Languages exhibit diverse morphosyntactic characteristics, and tokenizers do not account for these properties. Consequently, the semantic granularity of the minimal units processed by LLMs varies substantially across languages due to differences in tokenizers. This limits token-level analysis to understand how LLMs process and represent multiple languages.

A single English word might map to one token. The same concept in a morphologically rich language like Finnish or Guarani might fragment into five or six tokens. Comparing those token vectors directly is like comparing apples to fruit salad. This motivates an analysis that moves beyond the token level and adopts a holistic perspective that considers the entire input structure.

The tool: turning representations into trees

To get around this, the authors use StructLens, an analytical framework they developed in prior work. The idea is elegant: instead of comparing individual token vectors, build a tree that captures how all tokens in a sentence relate to each other inside the model, then compare those trees across languages.

StructLens is an analytical framework designed to reveal how internal structures relate holistically through their inter-token connection within a layer. It constructs maximum spanning trees based on the semantic representations in residual streams, analogous to dependency parsing, and leverages the tree properties to quantify structural similarity.