Skip to main content
Back

ARC-AGI-3 Benchmark defeats all top models

Gemini API adds embeddings, Veo 3 available in Gemini API, Karpathy demos real-time video model, Mistral expands Le Chat

July 21, 2025 4 min read

Hey,

Welcome to AlphaSignal, the most read source of news by AI engineers and researchers. Every day, we identity and summarize the top 1% of news, papers, models, and repos, so you’re always up to date. Here’s today’s roundup:

./SUMMARY

Read time: 4 min 24 sec

Top News

▪ ARC unveils benchmark to test how quickly agents learn new tasks

Lambda

▪ Deploy NVIDIA GPU clusters in minutes with Lambda 1-Click

Trending Signals

▪ Google launches Gemini Embedding model, now available via Gemini API, Vertex AI

▪ Karpathy showcases how you can steer real-time video with AI-powered filters

▪ Google releases Veo 3 video generation model in Gemini API

▪ Mistral adds Deep Research, Voice, and multilingual reasoning to Le Chat

▪ OpenAI’s new model scores gold in Math Olympiad-style evaluation

Trending Repos

▪ NextChat: Self-host ChatGPT with API key and memory support

▪ Strapi: Build headless CMS with API-driven content management and plugins

▪ Gitleaks: Scan Git repos for API keys, tokens, credentials, and private keys

Python Tip

▪ Catch subtle type bugs instantly in VS Code with static analysis

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

./TOP_NEWS

ARC releases ARC-AGI-3, a benchmark measuring how agents adapt to unseen tasks, frontier models score 0%

4,836 Likes

ARC-AGI-3 is an interactive benchmark for evaluating general intelligence in AI agents. It extends the ARC series by introducing new game environments that test an agent’s ability to learn without any instructions.


How does it work at a high level?

The benchmark consists of games created by humans set in a grid world. Each task is fully interactive, with no prior instructions or rewards. Agents must explore, experiment, and figure out the goal through trial and error, just like a human would.


Key Features

  • Evaluates agent learning efficiency across ~100 unique games, 6 preview games now available

  • Games involve multi-step reasoning, theory-of-mind, memory, tool use, and long-horizon planning

  • Agents solve tasks without instructions or prompting, using built-in spatial, physical, and social priors

  • Measures intelligence by evaluating exploration, planning, skill acquisition, and reflection

  • All tested frontier models like, o3, Grok 4 currently score 0

  • Humans solve games in under 5 minutes with no prior training

  • Includes developer preview competition with $8,000 in total prizes

  • Runs under fixed compute: RTX 5090 GPU or $1K API cost, within 8 hours

How to use it?

Download the SDK and environments, available on GitHub or Hugging Face. Build an agent that can learn from scratch in multiple runs of each environment. Test on public games and submit your agent for the July 2025 developer preview. Submit your code and output by August 10. Open-source it by August 11.


Main Limitations

ARC-AGI-3 is in preview. Only the top ~10 submissions will be scored due to capacity. Only  focuses on grounded reasoning and learning aspects of human intelligence. Full benchmark will be launched in 2026.


Community Feedback

"looks like this one is intentionally EVEN MORE vision based, even harder to represent as text"

 - Cooper

"What makes it especially interesting: it measures not just success, but also how many steps you needed to learn. It’s a benchmark for real-time generalization, not pattern-matching."

- Vitaly Rodnenko

Try now

Provision NVIDIA GPU Clusters in Minutes, No Surprises

Lambda’s 1-Click Clusters™ move you from zero to a working GPU cluster in minutes. Skip the long setup times and cut out configuration headaches. ESG confirms these technical results:

  • 1-click deployment, no manual setup or waiting quarters

  • Clear, predictable pricing, no ingress or egress fees

  • Hardware-level security for isolation from day one

  • Pre-installed Lambda Stack with PyTorch, TensorFlow, NVIDIA drivers

  • Streamlined cluster management, avoid complex provisioning steps


See the full ESG validation report for more details and performance data.

Download the report

partner with us

./TOP_SIGNALS

Google announces Gemini Embedding model, optimized for multilingual tasks like classification and retrieval

4,538 Likes

Karpathy endorses MirageLSD, a real-time video model that transforms live feeds using text prompts

5,147 Likes

Google brings Veo 3 to Gemini API and AI Studio as paid preview, let you generate 8-second videos using text prompts

2,485 Likes

Mistral upgrades Le Chat with Deep Research, multilingual reasoning, voice mode, and advanced image editing

2,128 Likes

OpenAI’s new experimental reasoning LLM claims gold in math as LLM scores 35/42 on Olympiad-style evaluation

7,639 Likes

./TRENDING_REPOS

NextChat

84.395 Stars

Lets you a self-hosted ChatGPT web UI that supports GPT-4, Claude, Gemini, and Groq APIs. You can Import/export conversations, manage multiple accounts, use long context, and monitor token usage with real-time stats.

strapi

68,639 Stars

Build customizable, API-first CMS backends with REST and GraphQL support. Define schemas visually or in code to self-host with full control. The library lets you manage content and add functions with plugins

Gitleaks

22,359 Stars

Thsi repo helps you detect and prevent hardcoded secrets like API keys, tokens, and credentials in Git repos, files, or CI pipelines. Use it to check for leaks before pushing, during CI, or in old commits.

./PYTHON_TIP

Catch Subtle Type Bugs Instantly in VS Code with Static Analysis

Pyright is a fast static type checker designed for large Python codebases. It works inside VS Code and catches subtle bugs, even in untyped ML code using type inference.


Benefits

  • Detects shape/type issues in ML pipelines like NumPy, Pandas, Torch

  • Surfaces incorrect API usage before runtime

  • Supports incremental checking for large notebooks and scripts

Step 1

Install Pyright extension in VS Code


Step 2
Add a pyrightconfig.json




Step 3

Use inline # type: comments for complex types when needed

At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us