Skip to main content
Back

Microsoft's Model Turns Single Image to DeepFake

CHECK THE REPO

April 18, 2024 5 min read

Hey,

Welcome to today's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries ensure you're always up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

IN TODAY'S SIGNAL

  • Top Repo: Easily fine-tune LLMs using PyTorch

  • Trending News:

    • Boston Dynamics Robot Goes Electric

    • OpenAI Upgrades Their Assistants API

    • Reka Releases Reka Core

  • Top Paper: Microsoft's Hyper-Realistic Video from Image

Read Time: 5 min 32 sec

Enjoying this newsletter? Please forward it to a friend or colleague. It will help us keep this content free.

TOP REPO

Finetuning

torchtune: Easily fine-tune LLMs using PyTorch

⇧ 3847    ⇆ 423

What's New

PyTorch has released torchtune, a library designed for fine-tuning large language models (LLMs). This PyTorch-native tool prioritizes modularity and customization, offering straightforward, memory-efficient fine-tuning recipes that allow users to adapt LLMs for specific uses easily.


The library covers the full scope of the fine-tuning process, from preparing datasets and models to evaluating and running local inference.

Key aspects of the workflow include:

  • Preparing data and models: It automates the downloading and setup of datasets and model checkpoints.

  • Customizing training: Users can tailor the training process using adaptable building blocks for various architectures and tuning methods.

  • Logging and quantization: Torchtune logs progress and metrics and supports model quantization after tuning.

Torchtune integrates with well-known tools and platforms, enhancing its functionality within the LLM ecosystem. Important integrations feature:

  • Hugging Face Hub: Accesses and utilizes model weights.
  • PyTorch FSDP: Scales training across multiple GPUs.
  • Weights & Biases: Logs detailed metrics and tracks progress.
  • ExecuTorch: Facilitates efficient inference across diverse devices.

Torchtune stands out for its memory efficiency, making it suitable for users with limited hardware, such as a single 24GB GPU. This efficiency enables sophisticated fine-tuning on less powerful systems.


Designed with simplicity, extensibility, and correctness in mind, torchtune ensures ease of use and comprehensive understanding. All training recipes are concise, typically under 600 lines, without external frameworks.


Torchtune currently supports models such as Llama 2, Mistral, and Gemma 7B, with plans to expand support to include models up to 70 billion parameters and Mixture of Experts (MoEs) models.


!pip install torchtune

Community Feedback

Chris Levy: "Already loving the docs and that they are also educational. Thanks!"


Aniket Maurya: "Congrats PyTorch team on the launch but I think we could already finetune LLMs pretty easily with libraries like LitGPT, and Axolotl.  Genuinely asking what makes TorchTune different from the existing libraries?


FODUU: "Amazing, nothing could be better then having a builtin library with no third party dependency."

Check the repo

Don’t Miss The AI and ML Developer Conference

Join mlcon, a hybrid conference that has built a community of over 13,000+ AI & ML developers since 2021.


With the rapid emergence of generative AI technology, LLMs, and AI's continued maturity, mlcon is the central hub for professionals to learn from industry leaders. 


Break silos, share insights, and collaborate for collective progress in AI innovation.

Register

partner with us

TOP ANNOUNCEMENTS

Robotics

Boston Dynamics’ Atlas Humanoid Robot Goes Electric

Boston Dynamics has transitioned its Atlas robot from hydraulic to electric, enhancing its movement range and efficiency. The new Atlas features advanced actuators for improved mobility, AI-enhanced machine learning tools for adaptability, and is designed for integration with enterprise-level software for fleet management. Testing begins next year with Hyundai, targeting commercial production post-validation.

⇧ 70,938     ⇆ 26,133

 RAG 

OpenAI Upgrades Their Assistants API: Now Built-in RAG With Over 10,000 Files

OpenAI recently announced significant updates to its Assistants API, including built-in RAG over 10k files, additional controls, and support for fine-tuning, sparking excitement and curiosity among users and developers. The enhanced file retrieval speed and new controls have been praised, with discussions on the cost-effectiveness of using OpenAI's API versus developing one's own AI assistant.  

⇧ 2847     ⇆ 632

Language Models

Reka Releases Reka Core, Its Multimodal Language Model To Rival Gpt-4 And Claude 3 Opus

Reka AI, a startup with roots in DeepMind, Google, and Meta, has introduced Reka Core, a multimodal language model trained on thousands of GPUs and rivaling models like GPT-4 and Claude 3 Opus. Core understands multiple modalities (text, image, audio, video), supports 32 languages, has a 128K token context window, and offers API, on-premise, or device deployment.

⇧ 1810    ⇆ 471

Imagine an AI...  that can type anywhere you can on macOS with full context on what's on your screen 

Omnipilot brings AI to every Mac app, using the app's context to provide intelligent assistance. Invoke it with a shortcut to supercharge writing, email, and getting answers.

Download macOS app ↗️

TOP PAPER

Diffusion

Microsoft's Vasa-1: Hyper-Realistic Face Video And Lip-Sync From A Single Picture

⇧ 7921   ⇆ 2285

What's New

Microsoft recently introduced VASA-1, an AI model that produces realistic talking face videos from a single static image and an audio clip.


The model outputs videos at 512x512 resolution and up to 40 frames per second (FPS), with a latency of only 170 milliseconds on NVIDIA RTX 4090 GPU systems.


Diffusion-Based Model Architecture

  • Holistic Approach: Unlike traditional methods that may handle facial features separately, VASA-1 uses a diffusion-based model to generate holistic facial dynamics and head movements. This method considers all facial dynamics, including lip motion, expression, and eye movements, as parts of a single comprehensive model.

  • Face Latent Space: The model works within a disentangled and expressive face latent space, enabling it to control and modify facial dynamics and head movements independently from other facial attributes like identity or static appearance.

Data-Driven and Disentangled Training

  • Diverse Data: VASA-1 is trained on a large and diverse dataset, allowing it to handle a wide range of facial identities, expressions, and motion patterns. This training approach helps the model perform well even with input data that deviates from what it was trained on, such as non-standard audio inputs or artistic images.

  • Disentanglement Techniques: The model's training involves advanced disentanglement techniques, allowing for the separate manipulation of dynamic and static facial features. This is achieved through the use of distinct encoders for different attributes and a set of carefully designed loss functions to ensure effective separation of these features.


Extensive Benchmarks:  VASA-1 has been rigorously tested against various benchmarks and has shown to significantly outperform existing methods in terms of realism, synchronization of audio-visual elements, and expressiveness of the generated animations.


The research acknowledges existing limitations, such as the model's current inability to process full-body dynamics or to fully capture non-rigid elements like hair. Future work is planned to expand the model's capabilities and address these areas.

Community Feedback

Brian Roemmele: "Unfortunately this will assure that this technology gets a very brisk reactionary need to “act” by regulators almost like that was the point?"


hou.mon: "The eyes are always a dead giveaway, visible here as well. This is hands down the best demo yet."

Read more

Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.

Thank You.

Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?

Sponsor