---
title: "AlphaSignal Training Infra: Pretraining News"
url: "https://alphasignal.ai/topic/training-infra/pretraining"
description: "Latest AI and machine learning news about Pretraining (Training Infra) on AlphaSignal."
---

# AlphaSignal Training Infra: Pretraining News

Latest AI and machine learning news about Pretraining (Training Infra) on AlphaSignal.

## Articles

- [MSAR Cuts Transformer Training Compute by 37% With Smarter Residual Connections](https://alphasignal.ai/news/msar-cuts-transformer-training-compute-by-37-with-smarter-residual-connections.md) (2026-10-09)
- [Zyphra Cuts MoE Training Communication Costs by 2.63x Using Router Patterns](https://alphasignal.ai/news/zyphra-cuts-moe-training-communication-costs-by-2-63x-using-router-patterns.md) (2026-10-07)
- [Optimizer Rankings Flip as Batch Size Grows, Study Warns ML Engineers](https://alphasignal.ai/news/optimizer-rankings-flip-as-batch-size-grows-study-warns-ml-engineers.md) (2026-10-06)
- [Yousef Saad's Survey Bridges Math and LLMs With One Shared Language](https://alphasignal.ai/news/yousef-saad-s-survey-bridges-math-and-llms-with-one-shared-language.md) (2026-10-03)
- [SmolLM3 Retrieves Perfectly at 64x Its Training Length by Moving Dropout](https://alphasignal.ai/news/smollm3-retrieves-perfectly-at-64x-its-training-length-by-moving-dropout.md) (2026-10-03)
- [Meta AI's NAVA-WAM Teaches Robots From Unlabeled Video, Hitting 93% Success](https://alphasignal.ai/news/meta-ai-s-nava-wam-teaches-robots-from-unlabeled-video-hitting-93-success.md) (2026-10-02)
- [LOOM Pushes MoE Transformers to 12 Loops, Boosting Accuracy by 5.3 Points](https://alphasignal.ai/news/loom-pushes-moe-transformers-to-12-loops-boosting-accuracy-by-5-3-points.md) (2026-10-01)
- [Pangram Labs Warns AI Web Text Could Triple Pretraining Costs by 2028](https://alphasignal.ai/news/pangram-labs-warns-ai-web-text-could-triple-pretraining-costs-by-2028.md) (2026-09-30)
- [Synthetic Warm-Up Saves 21B Training Tokens by Boosting Long-Range Retrieval](https://alphasignal.ai/news/synthetic-warm-up-saves-21b-training-tokens-by-boosting-long-range-retrieval.md) (2026-09-30)
- [DataTalks.Club's Free ML Zoomcamp Hits 14,600 Stars and Opens 2026 Registration](https://alphasignal.ai/news/datatalks-club-s-free-ml-zoomcamp-hits-14-600-stars-and-opens-2026-registration.md) (2026-09-29)
- [A Silent BF16 Bug in FlashAttention-3 Can Derail Entire Pretraining Runs](https://alphasignal.ai/news/a-silent-bf16-bug-in-flashattention-3-can-derail-entire-pretraining-runs.md) (2026-09-28)
- [Researchers Train AI on Zero Human Data Using Self-Play Programs](https://alphasignal.ai/news/researchers-train-ai-on-zero-human-data-using-self-play-programs.md) (2026-09-24)
- [White Circle's Halo Open-Source Framework Trains 20B Models 2.8x Faster](https://alphasignal.ai/news/white-circle-s-halo-open-source-framework-trains-20b-models-2-8x-faster.md) (2026-09-22)
- [A 7.4B Model Matches GPT-3 13B Using 20x Less Compute](https://alphasignal.ai/news/a-7-4b-model-matches-gpt-3-13b-using-20x-less-compute.md) (2026-09-16)
- [Magic Matches DeepSeek V4 Pro Base Using 50x Less Compute](https://alphasignal.ai/news/magic-matches-deepseek-v4-pro-base-using-50x-less-compute.md) (2026-09-08)
- [Kyoto University Finds a Hard Speed Limit on What SGD Can Learn](https://alphasignal.ai/news/kyoto-university-finds-a-hard-speed-limit-on-what-sgd-can-learn.md) (2026-09-08)
- [UC Berkeley Team Cuts Diffusion Model Training Time by 40% With Frequency-Aware Loss](https://alphasignal.ai/news/uc-berkeley-team-cuts-diffusion-model-training-time-by-40-with-frequency-aware.md) (2026-09-02)
- [Tsinghua's Puro-2B Trains a Language Model for Less Than One GPU Rental](https://alphasignal.ai/news/tsinghua-s-puro-2b-trains-a-language-model-for-less-than-one-gpu-rental.md) (2026-09-01)
- [Figure AI's Brett Adcock Bets Gigawatt NVIDIA Deal on Hark Handoff](https://alphasignal.ai/news/figure-ai-s-brett-adcock-bets-gigawatt-nvidia-deal-on-hark-handoff.md) (2026-08-27)
- [One Hidden Ratio Controls How Fast AI Models Actually Learn](https://alphasignal.ai/news/one-hidden-ratio-controls-how-fast-ai-models-actually-learn.md) (2026-08-25)
- [Stanford's Marin Trains a 535B Open Model Anyone Can Watch Live](https://alphasignal.ai/news/stanford-s-marin-trains-a-535b-open-model-anyone-can-watch-live.md) (2026-08-21)
- [Kakao Trains a 155B MoE on 10T Tokens Without a Single Hyperparameter Sweep](https://alphasignal.ai/news/kakao-trains-a-155b-moe-on-10t-tokens-without-a-single-hyperparameter-sweep.md) (2026-08-20)
- [ReliquaryForge's qwen3-4b-base-dapo-v4 Brings 32B Reasoning Power to a Single GPU](https://alphasignal.ai/news/reliquaryforge-s-qwen3-4b-base-dapo-v4-brings-32b-reasoning-power-to-a-single.md) (2026-08-18)
- [Apple Open-Sources AXLearn, the System Training Siri Across 10,000 Experiments](https://alphasignal.ai/news/apple-open-sources-axlearn-the-system-training-siri-across-10-000-experiments.md) (2026-07-08)
- [Prime Intellect Raises $130M to Open RL Training Labs Once Locked Inside OpenAI](https://alphasignal.ai/news/prime-intellect-raises-130m-to-open-rl-training-labs-once-locked-inside-openai.md) (2026-07-08)
- [Sakana AI Unifies Two Rival AI Optimization Camps to Slash LLM Merging Costs](https://alphasignal.ai/news/sakana-ai-unifies-two-rival-ai-optimization-camps-to-slash-llm-merging-costs.md) (2026-07-03)
- [OpenAI Hunted a Rockset Bug and Found an 18-Year-Old Race Condition](https://alphasignal.ai/news/openai-hunted-a-rockset-bug-and-found-an-18-year-old-race-condition.md) (2026-06-30)
- [minimind-3 Lets Anyone Train a Real LLM for $0.42 on One GPU](https://alphasignal.ai/news/minimind-3-lets-anyone-train-a-real-llm-for-0-42-on-one-gpu.md) (2026-06-28)
- [Cursor Drops Composer 3, a 1.5T-Parameter Model Trained on SpaceX's Supercomputer](https://alphasignal.ai/news/cursor-drops-composer-3-a-1-5t-parameter-model-trained-on-spacex-s-supercomputer.md) (2026-06-22)
- [NVIDIA Opens Megatron-LM to the World, Hitting 1,233 TFLOPS on GB300](https://alphasignal.ai/news/nvidia-opens-megatron-lm-to-the-world-hitting-1-233-tflops-on-gb300.md) (2026-06-20)
- [Baidu's PaddlePaddle 3.3 Beats FlexAttention by 2.1x With FlashMaskV3](https://alphasignal.ai/news/baidu-s-paddlepaddle-3-3-beats-flexattention-by-2-1x-with-flashmaskv3.md) (2026-06-19)
- [DeepSpeed 0.19.2 Fixes a Silent Bug Corrupting Mixed-Precision Training Runs](https://alphasignal.ai/news/deepspeed-0-19-2-fixes-a-silent-bug-corrupting-mixed-precision-training-runs.md) (2026-06-16)
- [George Hotz's Tinygrad Crashes MLPerf With 25,000 Lines of Python](https://alphasignal.ai/news/george-hotz-s-tinygrad-crashes-mlperf-with-25-000-lines-of-python.md) (2026-06-16)
- [Fareed Khan's Open-Source Repo Teaches Anyone to Build a 2B LLM](https://alphasignal.ai/news/fareed-khan-s-open-source-repo-teaches-anyone-to-build-a-2b-llm.md) (2026-06-16)
- [Michigan Researchers Beat the Data Wall With MIR, Gaining 1.3x More Training Data](https://alphasignal.ai/news/michigan-researchers-beat-the-data-wall-with-mir-gaining-1-3x-more-training-data.md) (2026-06-05)
- [Researchers Finally Prove Deep ReLU Networks Generalize as Well as Shallow Ones](https://alphasignal.ai/news/researchers-finally-prove-deep-relu-networks-generalize-as-well-as-shallow-ones.md) (2026-06-04)
- [MIT's SMT Trains Powerful RNNs Without Unrolling, Matching Transformer Speed](https://alphasignal.ai/news/mit-s-smt-trains-powerful-rnns-without-unrolling-matching-transformer-speed.md) (2026-06-04)
- [PC Layer Cuts Llama-1B Training Tokens in Half With Zero Inference Cost](https://alphasignal.ai/news/pc-layer-cuts-llama-1b-training-tokens-in-half-with-zero-inference-cost.md) (2026-06-04)
- [LatentWave Beats Pixel Reconstruction for Wireless AI Across 5G Tasks](https://alphasignal.ai/news/latentwave-beats-pixel-reconstruction-for-wireless-ai-across-5g-tasks.md) (2026-06-04)
- [MResOpt Teaches Neural Networks to Respect Hard Physics Constraints First](https://alphasignal.ai/news/mresopt-teaches-neural-networks-to-respect-hard-physics-constraints-first.md) (2026-06-04)
- [Osaka's AdaKoop Beats Deep Learning on 71 Real-Time Forecasting Benchmarks](https://alphasignal.ai/news/osaka-s-adakoop-beats-deep-learning-on-71-real-time-forecasting-benchmarks.md) (2026-06-03)
- [MIT Warns Muon Optimizer Silently Breaks in Late Transformer Layers at Scale](https://alphasignal.ai/news/mit-warns-muon-optimizer-silently-breaks-in-late-transformer-layers-at-scale.md) (2026-06-02)
- [UC Berkeley Finds Rosetta Neurons Grow Sharper as AI Models Scale](https://alphasignal.ai/news/uc-berkeley-finds-rosetta-neurons-grow-sharper-as-ai-models-scale.md) (2026-06-02)
- [KAUST Proves PipeDream's Hidden Cost Grows 16x With Every Doubled Stage](https://alphasignal.ai/news/kaust-proves-pipedream-s-hidden-cost-grows-16x-with-every-doubled-stage.md) (2026-06-02)
- [KAIST's EqGINO Fixes the Rotation Blind Spot Breaking 3D Physics AI](https://alphasignal.ai/news/kaist-s-eqgino-fixes-the-rotation-blind-spot-breaking-3d-physics-ai.md) (2026-06-02)
- [Fudan's Plug-in Module Makes Protein AI Explore 37x Faster Without Retraining](https://alphasignal.ai/news/fudan-s-plug-in-module-makes-protein-ai-explore-37x-faster-without-retraining.md) (2026-06-01)
- [Sakana AI's DiffusionBlocks Trains Giant Neural Networks One Block at a Time](https://alphasignal.ai/news/sakana-ai-s-diffusionblocks-trains-giant-neural-networks-one-block-at-a-time.md) (2026-05-27)
- [How to Train Your GPT Teaches LLaMA's Architecture Without Math Prerequisites](https://alphasignal.ai/news/how-to-train-your-gpt-teaches-llama-s-architecture-without-math-prerequisites.md) (2026-05-24)
- [Nous Research Finally Explains Why Byte-Level Models Keep Losing to Tokenizers](https://alphasignal.ai/news/nous-research-finally-explains-why-byte-level-models-keep-losing-to-tokenizers.md) (2026-05-21)
- [Nous Research's Lighthouse Attention Trains 1.69x Faster on Million-Token Contexts](https://alphasignal.ai/news/nous-research-s-lighthouse-attention-trains-1-69x-faster-on-million-token.md) (2026-05-07)