Sakana AI Unifies Two Rival AI Optimization Camps to Slash LLM Merging Costs

When you train a neural network, you usually have gradients to guide you. But a huge class of real problems -- tuning a robot in a physics simulator, querying a closed external API, searching for the best merge of two language models -- gives you nothing but a score. No gradients, no direction, just a number. That is the domain of Black-Box Optimization (BBO), and for decades it has been split into two camps that barely talked to each other. Sakana AI's new ICML 2026 paper, "Bridging Spherical Black-Box Optimizers", argues those camps were solving the same equation all along.

Two tribes, one equation

The BBO field has historically divided into two families:

• Parametric methods -- algorithms like Evolution Strategies (ES) and CMA-ES maintain a probability distribution over candidate solutions and update its parameters each step. They scale gracefully to thousands of dimensions but converge to a single solution, which means they can get stuck in local optima.
• Nonparametric (particle-based) methods -- algorithms like Consensus-Based Optimization (CBO) maintain a swarm of individual candidate solutions ("particles") that vote on where to move next. They can find multiple distinct solutions simultaneously, but they fall apart in high dimensions because the particles spread too thin.

The tradeoff felt fundamental. If you needed to search a high-dimensional space, you used ES. If you needed to escape local optima and find multiple good solutions, you used CBO. Nobody had a principled way to get both at once.

While Evolution Strategies, Consensus-Based Optimization, Optimization via Integration (OVI), and related methods have each been studied independently, their connections remained underexplored. The Sakana team unified these approaches within a common theoretical framework, revealing that they differ primarily in two design choices: fitness aggregation (controlling how sharply the algorithm focuses on the best candidates) and consensus scope (controlling whether the algorithm seeks one solution or many).

What the unification actually means

Think of it this way: every algorithm in both families is doing a weighted average of candidate solutions, then adding noise. The parametric methods do this averaging globally across all particles (one consensus), while particle-based methods do it locally per particle (many consensuses). The fitness aggregation knob controls how aggressively the algorithm ignores bad candidates -- turn it up and you get sharp, fast convergence; turn it down and you get a flatter, more exploratory search.

Once you see both families as dials on the same underlying update rule, you can build hybrids. Leveraging these insights, the authors introduce hybrid optimizers that interpolate between existing methods. Their ES-OVI hybrid allows explicit control over the preference for flat minima, enabling a trade-off between performance and robustness in continuous control tasks. Their CBO-OVI hybrids combine the higher-dimensional efficiency of parametric methods with the multimodal capabilities of particle-based approaches.

The two new algorithms that come out of this are:

• AdaPol -- adaptively adjusts the consensus scope during optimization, starting broad (multimodal) and narrowing as the search converges.
• SchedPol -- uses a fixed schedule to transition between multimodal and unimodal search phases, giving more predictable behavior.

The LLM merging problem that motivated all of this

The theoretical contribution would be interesting on its own, but Sakana had a concrete, expensive problem that made it urgent. Their earlier work on Evolutionary Model Merging -- published in Nature Machine Intelligence -- showed that you could automatically discover how to blend the weights of multiple fine-tuned LLMs to create a stronger combined model, without any additional training.

Model merging is a post-training method for LLMs that combines multiple fine-tuned versions of the same base model with different desirable characteristics. It interpolates the model parameters with learned per-layer weightings, resulting in a high-dimensional optimization problem that makes manual tuning impractical.

The catch: the original Evolutionary Model Merge used CMA-ES to find those weightings, with an objective function that evaluates the model on over 1,000 machine-translated Japanese samples from GSM8K -- making each evaluation costly. If you try to cut costs by evaluating on a smaller subset, a standard unimodal optimizer like CMA-ES overfits to that small sample and finds merge recipes that look great on the proxy but generalize poorly.

Multimodal search as a regularizer

Here is where the theory pays off. The CBO-OVI hybrids combine the higher-dimensional efficiency of parametric methods with the multimodal capabilities of particle-based approaches, achieving competitive results on language model merging under limited evaluation budgets.

By treating LLM merging as a multimodal problem -- actively searching for several distinct good merge recipes simultaneously -- AdaPol and SchedPol avoid committing too early to any single recipe that happens to score well on a small dataset. The diversity of solutions acts as an implicit regularizer. The best solution found across the multiple optima tends to generalize far better than the single optimum a standard ES would find.

A rough sketch of how you would set this up:

# Pseudocode: LLM merge with AdaPol
base_model = load("mistral-7b")
ft_models = [load("math-ft"), load("japanese-ft")]
# Each candidate x is a vector of per-layer merge weights
# AdaPol maintains a population of candidates (multimodal)
# and adaptively narrows consensus scope as search progresses
optimizer = AdaPol(dim=num_layers * num_models, population_size=32)
for step in range(budget):
    candidates = optimizer.ask()          # sample merge weight vectors
    scores = [eval_on_small_set(merge(base_model, ft_models, x))
              for x in candidates]        # expensive LLM evals
    optimizer.tell(candidates, scores)    # update distribution
best_x = optimizer.best_solution()
final_model = merge(base_model, ft_models, best_x)

Benchmarks and validation

The authors validate their methods on standard BBO benchmarks and higher-dimensional locomotion tasks, demonstrating that the hybrid methods can outperform their constituent algorithms. On the LLM merging task specifically, AdaPol and SchedPol find high-quality merge recipes using a fraction of the evaluation budget that CMA-ES requires, because the multimodal search identifies several distinct good solutions on the small proxy dataset rather than overfitting to one.

The locomotion experiments use continuous control environments (think MuJoCo-style tasks) where the ES-OVI hybrid's ability to prefer flat minima -- solutions that are robust to small perturbations -- translates directly into more reliable policies.

What this opens up

The practical implications go beyond LLM merging. Any setting where you need to optimize through an expensive black box with a limited evaluation budget is a candidate:

• Hyperparameter search for large models where each trial costs GPU-hours
• Neural architecture search over discrete-continuous mixed spaces
• Prompt optimization for closed APIs where you only see output scores
• Robotics policy search in simulators where you want robust, not just high-scoring, policies
• Drug discovery and scientific optimization loops with expensive wet-lab evaluations

The key limitation to keep in mind: the particle-based component still adds overhead compared to a pure ES. In very high dimensions (millions of parameters) with generous evaluation budgets, CMA-ES alone may still be the pragmatic choice. The hybrids shine specifically when evaluations are scarce and the loss landscape is multimodal -- which describes a surprisingly large fraction of real-world BBO problems.

The paper is accepted at ICML 2026. Code availability has not been announced separately from the paper itself, but Sakana AI has a track record of open-sourcing their optimization work, including the evolutionary model merge repository from their earlier Nature Machine Intelligence paper. The theoretical framework alone -- reducing the entire BBO zoo to two interpretable knobs -- gives practitioners a cleaner mental model for choosing and tuning gradient-free optimizers than anything previously available.