Skip to main content
Back

New Paper claims to reduce inter-GPU communication need by 1000x to 10,000x

READ THE PAPER

August 27, 2024 5 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 5 min 15 sec

🎖️ Top News

  • Nous Research introduced DisTrO an Architecture-agnostic distributed optimizers reducing inter-GPU communication by 1000x-10,000x.


📌 The AI Conference

  • The AI Conference is just 2 weeks away, with speakers from Stability AI, OpenAI, Meta, The Pentagon and many more.


⚡️ Trending Signals


🛠️ Trending Repos

  • Kotaemon: build and customize RAG-based document QA systems

  • courses: craft optimal prompts for Claude

  • LitServe: Enterprise-scale, Lightning-fast serving engine for AI models.


🧠 Tutorial

  • How to finetune Phi 3.5, 2x faster and with more accuracy

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Language Model

Nous Research introduced DisTrO an Architecture-agnostic distributed optimizers reducing inter-GPU communication by 1000x-10,000x.

⇧ 2,916 Likes

What's New

DisTrO (Distributed Training Over-the-Internet) Slashes Inter-GPU Communication for Large-Scale Model Training


Key features of DisTrO include

  • 1000x to 10,000x reduction in inter-GPU communication requirements for training large neural networks.

  • Compatibility with consumer-grade internet connections

  • Resilience to degraded or dropped nodes during training

  • Ability to accommodate new nodes joining mid-process

  • Potential for asymmetric data streaming in client-server setups

  • DisTrO enables efficient training over slow internet connections using heterogeneous hardware, potentially revolutionizing distributed machine learning.

Architectural Details

  • DisTrO-AdamW matched the convergence rate of standard AdamW with All-Reduce in pre-training a 1.2B parameter language model. 

  • It achieved this while cutting inter-GPU communication from 74.4 GB to 86.8 MB per step, an 857x reduction. The model was trained on 105B tokens over 25,000 steps, demonstrating DisTrO's effectiveness at scale.

  • Being architecture-agnostic and network-agnostic, making it adaptable to various model types and network topologies. 

  • DisTrO supports distributed data parallel training with minimal overhead, integrating seamlessly into existing training pipelines. 

Impact on Training Performance

  • DisTrO's performance raises questions about scaling laws in model training.

  • The researchers hypothesize that bandwidth reduction may increase with model size, suggesting DisTrO could be even more effective for larger models.

  • Could lead to new paradigms in GPU design, favoring larger VRAM and narrower interconnects.

While the initial tests focused on language models, the researchers suggest DisTrO could be applied to other types of large foundational models, such as diffusion models.

The team plans to release the code and complete methodology on GitHub, enabling further exploration and integration with existing distributed training frameworks.

Read the paper

The AI Conference is just 2 weeks away and almost SOLD OUT! 

With speakers from Stability AI, OpenAI, Meta, The Pentagon and many more. Discount codes expire this week!

  • 2 Days | 75+ Speakers | 3 Tracks

  • Top AI Companies Exhibiting in Expo

  • AI Impact in Healthcare, Enterprise, Defense + More

  • Live AI Startup Competition with VC Judges

  • Speaker Office Hours - Meaningful Networking

  • AI After Dark Networking Mixer + Expo Booth Crawl

Don't miss your chance to be at the forefront of AI innovation.

Register now to secure your spot!

partner with us

TRENDING SIGNALS

RAG

Langchain demonstrates a GitHub Repo containing a lot advanced RAG techniques

⇧430Likes

Anthropic

Anthropic publishes Claude’s system prompts for everyone to see. These only affect the Claude AI app on mobile and web, not the API.

⇧ 982 Likes

Google Gemini

Google allowed Gemini’s paid add-on for Google Workspace, so users can upload documents and perform data analysis. Sheets, Docs, CSVs, or PDFs are supported

⇧32 Likes

Coding Assistant

A Tweet went viral that showcases how to build financial dashboard in Cursor using only voice in less than 5min.

⇧ 2,900 Likes

Gen AI

Karpathy says Cloud+AI enables solo entrepreneurs to run multiple high-value companies leveraging AWS, GPT-4, and Stripe.

⇧ 4,200 Likes

TRENDING REPOS

RAG

Kotaemon

☆ 1,010 Stars

Helps you build and customize RAG-based document QA systems. Offers a clean UI for end-users and a framework for developers. Supports local LLMs and API providers like OpenAI. Includes hybrid retrieval, multi-modal QA, and complex reasoning methods. Configurable settings and extensible architecture allow fine-tuning of the RAG pipeline.

Prompt Engineering

courses

☆ 1,990 Stars

A collection of Prompt Engineering courses released from Anthropic for building with Anthropic API. You'll learn prompt structure, troubleshooting, and advanced techniques across 9 chapters. It covers Claude's capabilities, common pitfalls, and use cases. The course uses Claude 3 Haiku, Anthropic's fastest model.

LLM ServIng

LitServe

☆ 1,211 Stars

LitServe helps you serve AI models fast. It's built on FastAPI but 2x faster. You can deploy any model: LLMs, vision, audio, multi-modal. It handles batching, GPU scaling, and streaming. Works with PyTorch, JAX, TensorFlow. Self-host or use managed cloud.

TUTORIAL

LLM Finetuning

Notebooks: Phi 3.5 Finetuning 2x faster + Llamafied for more accuracy

Microsoft released Phi-3.5 mini last week with 128K context, and is distilled from GPT4 and trained on 3.4 trillion tokens. Unsloth library's maintainers uploaded a 4bit bitsandbytes quantized model and made some modifications to Phi-3.5 mini, with which you can eaziliy finetune it in Google Colab.


Here's how to use Unsloth to finetune Microsoft's new Phi-3.5 mini LLM:


Update Unsloth to the latest version.

  • Access the Unsloth Colab notebook for Phi-3.5 mini.

  • Load the 'Llamified' Phi-3.5 mini model from Hugging Face.

  • Use 4-bit quantization for reduced memory usage.

  • Apply LoRA for faster finetuning.

  • Train the model on your dataset.

  • Monitor training loss to gauge performance.

  • Save your finetuned model.


This process gives you

  • 2x faster finetuning

  • 50% less memory use

  • Improved accuracy due to unfused QKV matrices

Colab notebook to finetune phi-3.5

Thank You.

Want to promote your company, product, service, or event to 190,000+ AI researchers and engineers?

Sponsor