Skip to main content
Back

STOMP Beats Multi-Objective AI Alignment Where Linear Mixing Always Fails

A new offline RL algorithm called STOMP extends direct preference optimization to multiple conflicting rewards, tested on protein engineering tasks.

Paper, Full Access
  • New offline RL algorithm STOMP extends DPO to multi-objective alignment using smooth Tchebysheff scalarization
  • Linear reward mixing provably misses non-convex Pareto regions; STOMP recovers them by scalarizing the optimization
  • Rewards are standardized by observed distributions so different-scale objectives can trade off meaningfully
  • Beats state-of-the-art baselines on hypervolume in 8 of 9 protein engineering settings
  • Tested on three protein language models across DHFR, PbRR, and alpha-amylase fitness datasets
  • Framework is domain-agnostic and applies to chatbot alignment with competing helpfulness and harmlessness rewards

Aligning a language model to one reward is a solved recipe. Aligning it to two rewards that pull in opposite directions, like a protein that must be both highly active and highly specific, is where the standard playbook quietly breaks. A new preprint from Aadyot Bhatnagar, Peter Mørch Groth, and Ali Madani proposes an algorithm called STOMP that fixes the geometry problem sitting underneath most multi-objective alignment methods, and shows it works on real wet-lab protein datasets.

Why linear reward mixing keeps failing

The most common way to handle multiple rewards is to collapse them into one number by taking a weighted sum. It is simple, and it plugs directly into methods like DPO or PPO. Prior work has largely relied on this linear scalarization, which provably fails to recover non-convex regions of the Pareto front.

In plain terms: the Pareto front is the set of trade-off points where you cannot improve one objective without hurting another. If that front curves inward anywhere, no choice of linear weights will ever land a model on those inward-curving points. Those are exactly the interesting trade-offs an engineer wants to explore, and linear mixing renders them unreachable.

Many real applications demand the simultaneous optimization of conflicting rewards: catalytic activity and specificity in protein engineering, helpfulness and harmlessness for chatbots, correctness and latency for code models.