Skip to main content
Back

Study Reveals Claude 3.7 Hits 50% Success on Long Tasks—What’s Next?

READ MORE

March 20, 2025 6 min read

Hey,

Welcome to AlphaSignal – the most read newsletter by AI developers. 


We bring you the top 1% of news, papers, models, and repos, all summarized to keep you updated on the latest in AI.

IN TODAY'S SIGNAL

Read time: 6 min 16 sec

🎖️ Top News

📌  Speechmatics

⚡️ Trending Signals

💻  Top Lectures

🧠  Deep Dive

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

AI Performance

METR reveals AI task length doubling every 7 months, with Claude 3.7 Sonnet completing 1-hour tasks reliably

⇧ 4,836 Likes

What's New

Recent research from Model Evaluation & Threat Research (METR), shows a significant leap in AI’s ability to handle longer tasks. AI now completes tasks that require up to 59 minutes with 50% success, doubling performance from earlier systems like GPT-4. If this trend continues, AI will autonomously complete week-long tasks by 2027-2028 and month-long projects by 2029-2030.


Measuring AI Task Length

Researchers compared AI success rates to human task duration.

  • AI completed 170 software tasks, from two-second decisions to eight-hour engineering challenges.

  • They tracked human completion times and measured AI performance across different task lengths.

  • A logistic curve modeled AI success probability based on human task duration.

  • The 50% task completion time horizon increased exponentially, following a consistent historical trend.


Current AI Performance

AI handles short tasks well but struggles with extended workflows.

  • Claude 3.7 Sonnet completes one-hour tasks with 50% reliability.

  • GPT-4 handles tasks requiring 8-15 minutes of human effort.

  • 2019 models fail at tasks longer than a few seconds.

  • AI struggles with long-term coherence, dependency management, and failure recovery.


Tracking AI Progress

There are benchmarks that measure AI performance across real-world software tasks.

  • SWE-bench Verified, HCAST, and METR track AI success rates on different task durations.

  • These benchmarks use measured human completion times as reference points.

  • The study provides a quantifiable metric to evaluate AI capabilities over time.

Implications for Automation

The study implies that AI systems can now tackle more complex, longer-duration tasks.

  • AI-driven automation is expected to improve as task duration capabilities grow.

  • Task handling improvement could drive efficiency in extended automation.

  • Researchers are focusing on extending performance for even longer tasks.

Note

This research was presented by MTER, a nonprofit lab that has collaborated with companies like Anthropic and OpenAI on evaluating autonomous capabilities of frontier AI models.

Community Feedback

Dwarkesh Patel
"I'm not convinced - outside of coding tasks (think video editing, playing a brand new video game, coordinating logistics for a happy hour), AIs don't seem able to act as coherent agents for even short sprints.

But if I'm wrong, and this trend line is more general, then this is a very useful framing. If the length of time over which AI agents can act coherently is increasing exponentially, then it's reasonable to expect super discontinuous economic impacts."


Tamay Besiroglu
"We should be cautious interpreting the METR paper’s results—these ‘time horizons’ depend heavily on which tasks we pick. As a parallel, I ran a similar analysis on chess and found it can predict AI operating on decade‐long timescales."

Daniel Kokotajlo

"This is probably the most important single piece of evidence about AGI timelines right now. Well done! I think the trend should be superexponential, e.g. each doubling takes 10% less calendar time on average."

Read more

Voice tech should work in real-time, not fall behind

Speechmatics delivers real-time speech-to-text that’s instant, fluent, and 25% more accurate than the competition. This means:

  • AI assistants that respond naturally

  • Captions that stay in sync

  • Transcripts that capture every word

Supports 55+ languages with industry-leading accuracy—25% fewer errors than Microsoft, 50% fewer than Assembly AI, and 70% fewer than Deepgram.

Use it for healthcare, customer support, live events, and more. Build faster, more reliable voice AI without lag.

Try now

partner with us →

TRENDING SIGNALS

AI Model API

OpenAI announces o1-pro API, with 200K context, better reasoning, and vision support, price starting at $150/1M input tokens

⇧ 1,395 Likes

AI Feature

Google introduces Mind Maps in NotebookLM for exploring connections, and providing interactive, tailored learning tools

⇧ 4,594 Likes

Coding Assistance

Codeium unveils Windsurf Tab, an AI code prediction feature, integrating terminal and clipboard to capture fine-grained context

⇧ 1,295 Likes

AI Model

Hugging Face & IBM release SmolDocling, A 256M open-source Vision LM for full-page OCR, running in 0.35s on consumer GPUs

⇧ 3,693 Likes

Robotics

Google DeepMind unveils Gemini Robotics, Gemini 2.0-based multimodal models enabling robots to interact and act in real world

⇧ 1,493 Like

TOP LECTURES

Agent Architecture

Long-Term Agentic Memory with LangGraph

⇧ 1,359 Likes

Learn to build AI agents with long-term memory using LangGraph. Implement semantic, episodic, and procedural memory. Develop an email assistant that writes, schedules, and manages messages. Optimize system prompts for evolving behavior. Use hot-path and background updates. Store, retrieve, and update knowledge efficiently to improve AI assistants over time.

AI Trends

An unfiltered conversation with Dwarkesh Patel

⇧ 896 Likes

Product Lead for Google AI Studio Logan Kilpatrick explores the future of AI, covering AGI, fully automated firms, and the economic factors shaping AI development. Learn about AI’s impact on industries, authenticity in AI-generated content, and the risks of rapid AI evolution. Gain insights on trust in AI and robotics, and the role of government.

Multimodal AI

File prompting strategies for Gemini 

⇧ 651 Likes

Learn file prompting strategies for Gemini models. Use multimodal inputs like text, images, and audio. Apply best practices to improve responses: be specific, use few-shot examples, structure complex tasks, and define output formats. Troubleshoot issues by refining prompts and tuning sampling parameters like temperature, top-K, and top-P for better results.

DEEP DIVE

AI Development

Why Can't AI Make Its Own Discoveries? — With Yann LeCun

⇧ 498 Likes

Big Technology Podcast brings in-depth discussions on AI, machine learning, and the future of technology. Hosted by Alex Kantrowitz, this episode features Meta’s Chief AI Scientist, Yann LeCun, discussing why generative AI models memorize vast amounts of human knowledge but fail to make scientific discoveries.

He explains why large language models (LLMs) lack reasoning, planning, and an understanding of the physical world. The conversation covers AI’s limitations, the need for new architectures beyond LLMs, and the diminishing returns of scaling current models.

LeCun also introduces new approaches, including joint embedding predictive architectures (JEPAs) for building AI that can learn from real-world interactions. The episode also explores open-source AI development and its impact on innovation.

Watch now

At Alpha Signal, our mission is to build a sharp, engaged community focused on AI, machine learning, and cutting-edge language models, helping over 200,000 developers stay informed and ahead. We’re passionate about curating the best in AI, from top research and trending technical blogs to expert insights and tailored job opportunities. We keep you connected to the breakthroughs and discussions that matter, so you can stay in the loop without endless searching. We also work closely with partners who value the future of AI, including employers and advertisers who want to reach an audience as passionate about AI as we are.


Our partnerships are based on shared values of ethics, responsibility, and a commitment to building a better world through technology.Privacy is a priority at Alpha Signal. Our Privacy Policy clearly explains how we collect, store, and use your personal and non-personal information. By using our website, you accept these terms, which you can review on our website. This policy applies across all Alpha Signal pages, outlining your rights and how to contact us if you want to adjust the use of your information. We’re based in the United States. By using our site, you agree to be governed by U.S. laws.

Looking to promote your company, product, service, or event to 200,000+ AI developers? Let's work together.

Work with us