Skip to main content
Back

Biohub Opens ESMC-6B, the Biggest Protein AI Model Ever Released

Biohub open-sourced ESMC-6B, a 6-billion parameter protein language model with weights, sparse autoencoders, and fine-tuning support for downstream biology tasks.

  • Biohub released ESMC-6B, a 6B parameter protein language model with open weights on HuggingFace.
  • Trained on UniRef, MGnify, and JGI sequences clustered at 70% identity across roughly 2.5B clusters.
  • Two-stage training: 1M steps at 512 context, then 500K steps at 2048 context with 2.37e23 FLOPs.
  • Outperforms ESM-2 15B on contact prediction and powers ESMFold2 structure prediction.
  • Ships with layer-60 sparse autoencoders producing 16,384 interpretable features per residue.
  • Supports Flash Attention 2, LoRA fine-tuning via peft, and AWS SageMaker deployment.

Biohub just dropped the weights for ESMC-6B, the largest publicly released protein language model to date, along with its 300M and 600M siblings. ESMC has learned the rules of protein biology from training on billions of sequences, producing representations that feed downstream AI applications from therapeutic engineering to basic biology research. If you have worked with AlphaFold or ESM2 before, this is the next generation of the same idea, now scaled up and openly downloadable.

Scaling protein sequences like a language

ESMC treats amino acid sequences the way GPT treats text tokens: it learns by predicting masked residues in billions of protein chains. The ESMC family runs parallel to the ESM3 generative models for programmable protein engineering, focusing instead on representations of the underlying biology, and it scales up data and training compute to deliver large gains over ESM2.

The architecture is a stripped-down modern transformer. ESMC-6B has 6 billion parameters across 80 layers and 2.37e23 training flops, and Biohub additionally released overtrained 300M and 600M variants for local inference and finetuning. Under the hood it uses Pre-LayerNorm, rotary position embeddings, SwiGLU activations, and no bias terms in linear layers or LayerNorm, close to the LLaMA recipe adapted to protein sequences.

Emergent structure vs model scale for ESM2 and ESMC

How it was trained

Training pulled from UniRef, MGnify, and the Joint Genome Institute (JGI), with sequences clustered at 70% identity, yielding 83M, 372M, and 2B clusters respectively. The JGI slice is largely metagenomic data, meaning proteins from organisms nobody has ever cultured, which widens the evolutionary coverage the model sees.