Sakana AI Bets Against Transformers With Its Frontier Intelligence Group

Sakana AI has formally introduced the Frontier Intelligence Group (FIG), an internal research collective at its Tokyo lab. FIG will investigate alternatives to today’s dominant AI recipe: Transformer models trained on vast data sets with large amounts of compute.

The announcement organizes five largely public research projects under a dedicated program with institutional support and a mandate for long-horizon experiments. Sakana released no new model or API, so the immediate change is organizational. The company is committing researchers and funding to architectures, learning rules, hardware, and objectives that could address limits in current systems.

Why Sakana is funding long shots

Transformers have powered most recent advances in generative AI since their introduction in 2017. They use attention mechanisms to model relationships among tokens, while larger data sets, parameter counts, and computing clusters have steadily expanded their capabilities.

Scaling that design leaves several problems unresolved. Frontier models can fabricate facts, break down on unfamiliar tasks, and consume substantial energy during training and inference. FIG will test whether different architectures and training methods can reduce those weaknesses. It also treats brain-like data and energy efficiency as a core goal, regardless of whether scaling eventually produces artificial general intelligence.

Sakana CTO Llion Jones, a co-author of the original Transformer paper, has argued that the architecture’s dominance has narrowed the range of ideas receiving serious investment. FIG gives speculative projects more time to mature, along with what the company calls the “freedom to fail.” Its stated mandate is to pursue research that would otherwise remain unexplored.

Five bets beyond the standard stack

• Continuous Thought Machines (CTM): The architecture separates synaptic processing from neuron-level computation and iterates over inputs across internal time steps. It represents information partly through synchronization among groups of neurons, producing richer temporal dynamics than a conventional recurrent neural network. The project tests whether biologically inspired timing can improve reasoning and adaptation.

• Augmented Lagrangian Predictive Coding: Standard backpropagation calculates gradients from a global loss and propagates them through the network. Predictive coding trains layers using local prediction errors, a structure that could suit distributed systems and neuromorphic chips. Earlier versions struggled with deep networks; Sakana reports that its augmented Lagrangian method trained networks containing as many as 1,000 layers.

• Sparser, Faster, Lighter Transformers: Sakana and NVIDIA developed a sparse data format and CUDA kernels for billion-parameter language models. Sakana says more than 95% of feedforward neurons can remain inactive for a given token, yet dense GPU operations still process the full layer. The custom format arranges sparsity around GPU execution patterns so hardware can skip inactive work, reducing memory use, latency, and energy consumption.

• The AI Picbreeder Experiment: The original Picbreeder website allowed people to evolve images by repeatedly selecting promising variations, often producing unexpected results through indirect paths. FIG tested whether frontier vision-language models could guide a similar open-ended search. Search interventions improved image quality and diversity, although the models still struggled to pursue inventive detours.

• Smart Cellular Bricks: Each physical block runs a small neural network and exchanges messages only with adjacent blocks. Working collectively, an assembly can identify whether it forms an object such as a chair or table and recover after damage. The project moves decentralized multi-agent learning into hardware with strict communication and computing limits.

Which ideas could reach developers first

The sparse Transformer project has the shortest path to existing production systems because it preserves the familiar model architecture and changes how sparse operations map onto GPUs. Adoption will depend on end-to-end benchmarks, hardware support, model-quality measurements, and integration with common inference frameworks.

Continuous Thought Machines and predictive coding require deeper changes to model and training design. FIG presents both as research systems, with no drop-in replacements for current PyTorch or JAX pipelines. Their practical case will depend on whether they can match established methods on accuracy while using less data, coordination, memory, or energy.

The Picbreeder work offers a nearer-term lesson for developers building autonomous agents: strong vision-language models still have difficulty sustaining open-ended exploration when progress requires temporary declines in apparent quality. Smart Cellular Bricks addresses a different engineering frontier, showing how small local models might coordinate robots, modular devices, or sensor networks without a central controller.

FIG gives Sakana a recruiting and funding structure for these projects outside its main model-development cycle. The group is hiring, particularly at the intersection of neuroscience and machine learning. Its broader impact will depend on reproducible benchmarks, released code, and hardware demonstrations that establish clear gains in cost, quality, or adaptability.