Skip to main content
Back

New Karpathy Tutorial: How to Code GPT-2 in 4 Hours

WATCH THE LECTURE

June 10, 2024 4 min read

Hey,

Welcome to today's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI. 


Let's get into it,


Lior

IN TODAY'S SIGNAL

📰 Top News

  • Karpathy releases tutorial on coding GPT-2 in 4 hours, achieving close performance to GPT-3 model.

📌 AI Conference

⚡️ Top 5 Signals

🛠️ Top of Github

  • Marker: Converts PDFs to Markdown using deep learning, fast and accurate.

  • code2prompt: Transforms codebases into Markdown prompts for language models.

  • FunClip: Video speech recognition and clipping tool using AI.

🧠 Python Tip

  • Use subset operator (<=) on dictionary views to check if one dictionary is a subset of another.

Read Time: 4 min 18 sec

Enjoying this newsletter?
Please forward it to a friend or colleague. It helps us keep this content free.

TOP NEWS

Tutorials

Karpathy's New Tutorial: How to Code GPT-2 in 4 Hours

⇧ 11,081 Likes

What's New

Andrej Karpathy has launched a new YouTube tutorial, "Let’s reproduce GPT-2 (124M)," which guides you from an empty file to a fully functional GPT-2 model. 

This extensive four-hour video is structured into several critical phases:

  • Model Construction: Initially, Karpathy constructs the GPT-2 neural network. He incorporates learning from OpenAI’s original GPT-2 and GPT-3 papers, focusing on neural module implementation and parameter loading from Hugging Face.

  • Speed Optimization: The session covers training optimization techniques such as mixed precision training and kernel fusion, significantly reducing the training computation time from 1000 milliseconds to 93 milliseconds per training step.

  • Hyperparameter Setup: Detailed explanation of setting up training runs, including configuring hyperparameters like AdamW optimizer, gradient clipping, learning rate scheduling, and advanced training regimes such as distributed data parallel processing.

  • Model Evaluation and Training: After setting up, Karpathy runs the model overnight, evaluating its performance the next morning. The results showcase performance close to GPT-3 (124M), highlighting the effectiveness of the setup and optimizations.

Karpathy’s GitHub repository provides a complete commit history for this project, allowing developers to follow each code modification. The video ends by linking to the nanoGPT repository, which mirrors about 90% of the final codebase.


Access the Github repo here

Watch the lecture

The AI Conference: Share 2 Days with the Brightest Minds in AI

The AI Conference brings together OpenAI, Anthropic, Meta, DeepMind and many more. 

  • Engage with 60+ speakers leading the AI revolution

  • Network, collaborate, and co-create with industry pioneers

  • Explore topics including AGI, AI in enterprise, building with AI, and more!

Last chance to register for Early Bird pricing! 


Readers get 200$ off, code:

alpha24

Join the conference

partner with us

TRENDING SIGNALS

Industry

Apple will incorporate ChatGPT natively in Siri for free later this year

⇧ 2434 Likes

Open Source

A developer recreates an entire GPT architecture (85,000 parameters) in a spreadsheet - open source

⇧ 5012 Likes

Alignment

Anthropic releases new video on how they shaped and trained their LLM's (Claude) character and personality

⇧ 739 Likes

Leaderboard

Predibase releases "The Fine-tuning Index": visualize the performance of open models across 31 distinct tasks

⇧ 232 Likes

Robotics

Hugging Face and Pollen Robotics show off first project: an open source robot that does chores

⇧ 327 Likes

TOP REPOS

PDF Conversion

marker (fixed)

☆ 11,923 Stars

Marker converts PDFs to Markdown quickly and accurately using deep learning. It supports various documents, including textbooks and scientific papers, and works with all languages. You can run it on GPU, CPU, or MPS for increased speed. In benchmarks, Marker operates 4x faster than Nougat, achieving an accuracy score of 0.613721.

Coding

code2prompt

☆ 569 Stars

code2prompt converts your entire codebase into a structured Markdown prompt for use with language models. It details the source tree, counts tokens, and allows customization via templates. Features include respecting .gitignore, filtering with glob patterns, and options for output formatting, including JSON and direct clipboard copying.

Speech Recognition

FunClip

☆ 535 Stars

Open-source, accurate and easy-to-use video speech recognition & clipping tool, LLM based AI clipping integrated. Based on Alibaba's FunASR and Paraformer-Large models.

PYTHON TIP

Checking subset relationships between Dictionaries in Python

In Python, the subset operator (<=) can be applied to dictionary views to verify if all elements of one dictionary are contained in another.


This is particularly useful for checking subsets within dictionary key-value pairs using the items() view. It simplifies the process of ensuring that a smaller dictionary's items are entirely present within a larger one, which can be crucial for validating configurations or properties in data processing scenarios.


Example: Suppose we have two dictionaries:


a =
b =

# To check if b is a subset of a:
result = b.items() <= a.items()
print(result)  # Output: True

Thank You.

Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?

Sponsor