Skip to main content
Back

An Open Source Model Beats GPT4 in Coding and Math

TRY DEEPSEEK

June 18, 2024 4 min read

Hey,

Welcome to today's edition of AlphaSignal.

IN TODAY'S SIGNAL

🎖️ Top News

📌 Latitude

  • Deploy AI applications instantly with Launchpad – Purpose-built for speed and scalability on dedicated GPUs.

⚡️ Top 5 Signals

📄 Top Papers

🧠 Recommended Lecture

Read Time: 4 min 55 sec

Enjoying this newsletter?
Please forward it to a friend or colleague. It helps us keep this content free.

TOP NEWS

Open Source

DeepSeek-Coder-V2: First Open Source Model Beats GPT4-Turbo in Coding and Math

⇧ 1621 Likes

What's New

DeepSeek-Coder-V2, a new open-source code language model, demonstrates superior performance in coding and mathematics benchmarks compared to leading proprietary models such as GPT4-Turbo.

This model operates under a Mixture-of-Experts (MoE) framework and is available in two configurations: 16 billion and 236 billion parameters.

It has been trained with a vast multi-source dataset comprising 6 trillion additional tokens, enhancing its ability to process complex coding and mathematical problems.

Key Specifications and Model Features

  • Model sizes: 16 billion parameters (2.4 billion active) and 236 billion parameters (21 billion active).
  • Programming language support: 338 languages.
  • Context length: Up to 128,000 tokens.
  • The model is fully open-sourced, offering unrestricted commercial and research use under a permissive license.

Performance Metrics and Benchmarks

  • DeepSeek-Coder-V2 scored 90.2% on the HumanEval benchmark and 76.2% on the MBPP benchmark, setting new state-of-the-art results in these coding tests.

  • Maintains robust performance in general language tasks comparable to its predecessors and competing models.

Training and Development

  • The training dataset included 60% source code, 10% math corpus, and 30% natural language corpus, sourced predominantly from GitHub and popular coding forums like Stack Overflow.

  • The comprehensive training approach enables the model to handle a broader range of coding and computational tasks efficiently.

Access

Try deepseek

Deploy AI applications instantly with Launchpad – Purpose-built for speed and scalability on dedicated GPUs

Launchpad is Latitude.sh's purpose-built platform for AI applications, a fast solution that lets you run everything from small databases to the largest AI models.

The platform has been constantly evolving based on the feedback from the Machine-Learning community, and now has many new features available to improve your container experience:

  • Different GPUs: NVIDIA's L40S (48 GB vRAM) and H100 (80 GB vRAM)

  • SSH support: SSH access for debugging and development

  • Filesystem volumes: Add persistent storage to multiple containers

  • Per-minute billing: Pay only for what you use

  • Blueprint library: Instantly launch containers from a community-built library

Create your free account today and deploy container-based applications on dedicated GPUs and CPUs in just a few seconds!


Get started

partner with us

TRENDING SIGNALS

Video Generation

DeepMind’s new video-to-audio (V2A) model generates accurate soundtracks and dialogue for videos

⇧ 1321  Likes

Video Generation

Open-Sora 1.2 is out: an advanced video generation model that produces high-quality video the open-source way

⇧ 272 Likes

API

Context caching for the Gemini API is here, supports both 1.5 Flash and 1.5 Pro, is 2x cheaper than previously announced

⇧ 400 Likes

Report

Retool releases the State of AI report: get insights from over 700+ devs and tech leaders on AI use cases and best stacks

⇧ 23 Likes

Sound

ElevenLabs opens their Text-to-Sound Effects API

⇧ 751 Likes

TOP PAPERS

Language Models

Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B

⇧ 9021 Likes

Problem
LLMs struggle with complex mathematical reasoning, often producing incorrect or misleading answers due to limited accuracy and reliability.

Solution
The MCT Self-Refine (MCTSr) algorithm integrates LLMs with Monte Carlo Tree Search (MCTS) to enhance decision-making in mathematical tasks. This approach systematically explores and refines potential solutions using iterative selection, self-refine, self-evaluation, and backpropagation processes.

Results
The MCTSr significantly improves success rates across various datasets: GSM8K (from 74.07% to 96.66% with increased rollouts), MATH (from 24.36% to 58.24%), and mathematical Olympiad benchmarks like AIME (from 2.36% to 11.79%). This demonstrates its effectiveness in addressing the challenges of mathematical problem-solving with LLMs.

State Space Model

Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

⇧ 2010 Likes

Problem

Efficiently modeling sequences with infinite context length is challenging due to quadratic computation complexity and limited extrapolation ability.


Solution
A new model, Samba, introduces a hybrid architecture combining Mamba, a selective State Space Model (SSM), with Sliding Window Attention (SWA). This model compresses sequences into recurrent hidden states while enabling precise memory recall, maintaining linear-time complexity.


Results
Samba, with 3.8 billion parameters trained on 3.2 trillion tokens, achieves 71.9% on MMLU and 87.6% on GSM8K benchmarks, surpassing traditional models. It extrapolates efficiently to 256K context lengths and offers a 3.73x throughput increase over Transformers at 128K prompt length. 

Access the model here

Depth Estimation

Depth Anything V2

⇧ 601 Likes

Problem
Monocular depth estimation (MDE) models struggle with noisy real-world data and limited scenario coverage. Existing models lack precision and robustness in depth predictions for complex scenes.


Solution
Depth Anything V2 introduces a new MDE model using synthetic images for initial training to enhance depth precision, followed by pseudo-labeled real images to bridge domain gaps and improve scene diversity. The model scales from 25M to 1.3B parameters, targeting robust and fine-grained predictions.


Results
The V2 model achieves over 10 times faster processing and higher accuracy compared to previous models. It significantly outperforms Depth Anything V1 in robustness and fine details.


For detailed comparisons and access to models: Depth Anything V2.

MUST READ

Diffusion

Step-by-Step Diffusion: An Elementary Tutorial

"Step-by-Step Diffusion: An Elementary Tutorial" is a 50-page long course on diffusion models and flow matching for machine learning, aimed at a technical audience with no diffusion experience. 


It simplifies the mathematical details as much as possible (sometimes heuristically), while retaining enough precision to derive correct algorithms.

The content is divided into several focused sections:

  • Fundamentals of diffusion.

  • Stochastic sampling with DDPM.
  • Deterministic sampling with DDIM.
  • Flow matching techniques.
  • Practical applications in diffusion.

It provides intuitive explanations and step-by-step derivations.

Read the paper

Thank You.

Want to promote your company, product, job, or event to 190,000+ AI researchers and engineers?

Sponsor