STOMP Beats Multi-Objective AI Alignment Where Linear Mixing Always Fails
A new offline RL algorithm called STOMP extends direct preference optimization to multiple conflicting rewards, tested on protein engineering tasks.
Takeaways
- New offline RL algorithm STOMP extends DPO to multi-objective alignment using smooth Tchebysheff scalarization
- Linear reward mixing provably misses non-convex Pareto regions; STOMP recovers them by scalarizing the optimization
- Rewards are standardized by observed distributions so different-scale objectives can trade off meaningfully
- Beats state-of-the-art baselines on hypervolume in 8 of 9 protein engineering settings
- Tested on three protein language models across DHFR, PbRR, and alpha-amylase fitness datasets
- Framework is domain-agnostic and applies to chatbot alignment with competing helpfulness and harmlessness rewards
Aligning a language model to one reward is a solved recipe. Aligning it to two rewards that pull in opposite directions, like a protein that must be both highly active and highly specific, is where the standard playbook quietly breaks. A new preprint from Aadyot Bhatnagar, Peter Mørch Groth, and Ali Madani proposes an algorithm called STOMP that fixes the geometry problem sitting underneath most multi-objective alignment methods, and shows it works on real wet-lab protein datasets.
Why linear reward mixing keeps failing
The most common way to handle multiple rewards is to collapse them into one number by taking a weighted sum. It is simple, and it plugs directly into methods like DPO or PPO. Prior work has largely relied on this linear scalarization, which provably fails to recover non-convex regions of the Pareto front.
In plain terms: the Pareto front is the set of trade-off points where you cannot improve one objective without hurting another. If that front curves inward anywhere, no choice of linear weights will ever land a model on those inward-curving points. Those are exactly the interesting trade-offs an engineer wants to explore, and linear mixing renders them unreachable.
Many real applications demand the simultaneous optimization of conflicting rewards: catalytic activity and specificity in protein engineering, helpfulness and harmlessness for chatbots, correctness and latency for code models.