Sakana AI's Dream-Cubed Beats Real Minecraft Terrain 67% of the Time

Generative AI has conquered text, images, and video. Interactive 3D worlds, the kind you can walk around in, build in, and break, have remained stubbornly out of reach. Dream-Cubed, a new release from Sakana AI and NYU, attacks this problem by treating Minecraft's building blocks exactly like language tokens: discrete, composable units that a large transformer learns to predict.

The result is a family of diffusion models that can generate biome-accurate chunks of Minecraft terrain, inpaint missing regions, outpaint to arbitrary world sizes, and respond to hand-crafted block patterns as hard constraints, all at native block resolution, with outputs that load and play directly in the actual game.

Why Minecraft is the right testbed

Minecraft is the best-selling video game in history, and its world representation is unusually clean for machine learning. Every location is a voxel: a single categorical value drawn from a vocabulary of block types like stone, sand, or water. No continuous pixel values, no camera poses, no depth ambiguity. A 32x32x32 region is just a tensor of integer block IDs.

This maps almost perfectly onto how language models work. Words are discrete tokens from a fixed vocabulary; blocks are discrete tokens from a fixed block vocabulary. The compositional structure that makes transformers powerful for text turns out to be exactly the right inductive bias for 3D voxel worlds.

30 billion blocks of training data

Before training any models, the team had to solve a data problem. The paper describes a two-pronged collection strategy:

• Procedurally generated terrain: Over 1.8 million 32x32x32 chunks scraped from Minecraft's built-in world generator, labeled by biome (desert, ocean, jungle, village, and 11 others), totaling more than 30 billion individual blocks.
• Human-authored maps: 200,000+ chunks from six large, professionally crafted maps, obtained with the creators' explicit consent. These maps use a richer block vocabulary and contain complex architectural structures that procedural generation never produces.

Naive collection is badly imbalanced. Villages are the most structurally complex biome but also the rarest, appearing in less than 1% of naturally spawned chunks. The team built targeted collection pipelines to address this and showed that doubling village representation in the training set produced a significant FID improvement on that biome specifically.

Two diffusion formulations, one backbone

The models run on a 3D Diffusion Transformer (DiT) with roughly 280 million parameters. The team chose to work at native block resolution rather than compress spatial dimensions with a variational autoencoder, a common trick in image diffusion that trades spatial fidelity for speed. The slower approach is what makes fine-grained block-level control possible.

Two diffusion formulations were studied side by side:

• MD4 (discrete masked diffusion): A [MASK] token is added to the block vocabulary. During training, real blocks are progressively replaced with [MASK]. During generation, the model iteratively unmasks blocks by predicting what should fill each position. Conceptually close to masked language modeling, but in 3D.
• DDPM in embedding space (continuous diffusion): Each block type is mapped to a 16-dimensional vector using its in-game name as a text embedding (e.g., "stone" maps to an embedding). Gaussian noise is added and removed in this embedding space, and a nearest-neighbor lookup recovers the final block type.

Both approaches produce convincing biome-conditioned chunks, but MD4 has a structural advantage that makes it the more compelling choice for interactive use.

Controllability without extra training

Because MD4 works by unmasking tokens, you can inject any pattern of real blocks into the sampling process as pre-unmasked positions. The model treats them as hard constraints and fills in everything else around them.

Inpainting, outpainting, and user-conditioned generation are not separate fine-tuned capabilities. They fall out of the base training objective with no additional training required:

• Inpainting: Place a few blocks in a pattern (a spiral, a volcano cone, a wave), leave the rest masked, and the model completes a coherent chunk around your seed.
• Outpainting: Generate a chunk, then slide the window over by a few blocks. Fix the overlapping region from the already-generated chunk and sample the new region. Repeat indefinitely to grow the world in any direction.
• Large-scale conditioned worlds: Lay out a seed pattern spanning hundreds of blocks, slice it into overlapping 32x32x32 windows, and run outpainting with both the seed constraints and neighboring chunk contexts active simultaneously.

The outputs are valid Minecraft world files. A player can load them directly into the game and start building or exploring.

How it performs

No established metrics exist for 3D voxel worlds, so the team adapted FID (Fréchet Inception Distance, a standard image quality metric that compares the statistical distribution of generated images to real ones) by rendering chunks from above and computing FID on those renders.

They also ran a human preference study with actual Minecraft players, asking them to pick the better-looking chunk between a generated sample and a real one from the training set:

All three models were preferred over real chunks more than half the time. The best model, MD4 at patch size 2, won 67% of head-to-head comparisons against real Minecraft terrain. The team notes this may partly reflect the quality of their data curation pipeline, but it confirms that generated chunks are at least on par with the training distribution.

Limitations worth knowing

The human-authored map data required compressing thousands of block types down to a vocabulary of 171 to keep training tractable, so fine visual details get lost. The outpainting procedure is a sliding window, which means very long-range coherence across hundreds of chunks is not guaranteed. The models were also trained on Minecraft specifically; generalizing to other voxel games would require a new dataset and retraining.

What's available

Everything is open. The GitHub repository includes training code for both MD4 and DDPM variants, inference scripts for inpainting and outpainting, and FID evaluation utilities. Three pretrained model weights are on HuggingFace (MD4 patch 2, MD4 patch 4, and DDPM patch 2), along with the full 2-million-chunk Dream-Cubed dataset, released in accordance with the original map creators.

For game developers, the most immediately practical application is AI-assisted level design: sketch a rough block layout and let the model fill in a coherent, biome-appropriate world around it. For researchers, Dream-Cubed is the first large-scale, carefully curated benchmark for 3D voxel generation, a domain that has been conspicuously absent from the generative modeling literature despite being a natural fit for discrete sequence models.