Skip to main content
Back

Chinese AI that understand video: Qwen2-VL Breaks Barriers

READ MORE

August 30, 2024 5 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 5 min 46 sec

🎖️ Top News


📌 AssemblyAI

⚡️ Trending Signals

🧠 Top Papers


🧠 Recommended Lecture

  • Learn about PyTorch internals and architecture.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

Language Model

Qwen releases Qwen2-VL: advanced vision-language model with video understanding

⇧ 3,966 Likes

What's New

Qwen2-VL builds upon the Qwen2 language model family, offering enhanced visual understanding capabilities. 


The model comes in three sizes

  • 2B and 7B (open-sourced under Apache 2.0 license), and 

  • 72B (available via API).

Performance

  • Qwen2-VL demonstrates state-of-the-art performance on visual understanding benchmarks. The 72B model surpasses GPT-4o and Claude 3.5-Sonnet on most metrics. 

  • The 7B model achieves top performance in document understanding and multilingual text comprehension. Even the 2B model shows strong performance in video-related tasks and document understanding.

Key capabilities of Qwen2-VL include

  • Processing videos over 20 minutes long

  • Complex reasoning for device operation (e.g., mobile phones, robots)

  • Multilingual text understanding in images (European languages, Japanese, Korean, Arabic, Vietnamese)

  • Function calling for integration with external tools

  • Document understanding and general scenario question-answering

  • Integrates with Hugging Face Transformers and vLLM. 

  • Supports various tools for quantization, deployment, and fine-tuning, making it accessible for ML engineers and researchers to implement and customize.

The model introduces two key architectural innovations

  • Naive Dynamic Resolution support allows Qwen2-VL to handle arbitrary image resolutions by mapping them to a dynamic number of visual tokens. This ensures consistency between input and image information. 

  • Multimodal Rotary Position Embedding (M-ROPE) enables concurrent capture of 1D textual, 2D visual, and 3D video positional information.

Read more

Outpace the industry with speech-to-text models that evolve ahead of the rest

AssemblyAI is leading the Speech AI industry with unmatched accuracy, breakthrough speech understanding capabilities, developer-preferred API, and continual code-free updates to keep you on the cutting edge — without the lift.


The simple API offers:

  • High-accuracy transcribing and understanding accented speech, background noise, and natural conversations

  • Easy integration with models for speech recognition, speaker detection, PII redaction, and more

  • Low-latency processing for real-time applications

  • Access to Claude 3 models for precise call insights, summarizations, and more

Join over 200,000 developers building with AssemblyAI


Start building With $50 in free credits

Start building

partner with us

TRENDING SIGNALS

Vision Model

Google releases GameNGen, Diffusion models that predicts next frame of the shooter game DOOM, playable at 20 fps

⇧26,744Likes

LLM

Magic releases, LTM-2-Mini, the first model with a 100 million token context window, translating to 10mn lines of code, or 750 novels

⇧2,454 Likes

Voice AI

Bland.AI releases a conversational product, sounding close to human, and you can enter your number on their site to test it now

⇧3,171 Likes

LLM

Nous Research releases new open dataset, Hermes Function Calling V1 that includes Function Calling and Structured JSON Output

⇧ 573 Likes

Llama

Meta revealed the adoption stat of Llama models in the industry, with 350M downloads on HuggingFace, a 10x growth over last year

⇧ 1,990 Likes

TOP PAPERS

Computer Vision

Sapiens: Foundation for Human Vision Models

⇧ 3,100 Likes

Problem
Existing human-centric vision models lack generalization, broad applicability, and high fidelity. Current approaches often use general datasets and low-resolution models, limiting performance on specific human tasks.


Solution
Sapiens introduces a family of high-resolution (1024x1024) Vision Transformers pretrained on 300 million human images. It uses masked autoencoding and fine-tunes for pose estimation, segmentation, depth, and normal prediction using high-quality annotations.


Results
Sapiens-2B achieves state-of-the-art performance:

  • Pose estimation: 61.1 AP (+7.6 AP improvement)

  • Segmentation: 81.2 mIoU (+17.1 mIoU improvement)

  • Depth estimation: 22.4% relative RMSE reduction

  • Normal estimation: 53.5% relative angular error reduction

  • Models generalize well to in-the-wild scenarios despite limited or synthetic training data.

Foundational Models

The Mamba in the Llama: Distilling and Accelerating Hybrid Models

⇧ 589 Likes

Problem
Transformers dominate language modeling but are slow for long sequences. Linear RNNs like Mamba offer faster inference but lag in performance. The challenge is converting pretrained Transformers to linear RNNs for efficient deployment while maintaining quality.


Solution
The authors distill large Transformers into hybrid Mamba models by reusing attention weights. They propose a modified Mamba architecture, a multistage distillation approach, and a hardware-aware speculative decoding algorithm for Mamba and hybrid models.


Results

  • Distilled Llama3-8B-Instruct hybrid model (50% attention) achieves 29.61% length-controlled win rate on AlpacaEval 2 vs GPT-4 and 7.35 on MT-Bench.

  • It outperforms Mamba 7B trained from scratch with 1.2T tokens.

  • Hybrid models with 25% attention layers maintain comparable performance to original Transformers in chat benchmarks.

Transformers

Generative Verifiers: Reward Modeling as Next-Token Prediction

⇧ 574 Likes

Problem
LLMs often make logical errors in reasoning tasks. Best-of-N method uses verifiers to rank multiple solutions, but current discriminative verifiers don't utilize LLMs' text generation capabilities.


Solution
GenRM trains verifiers using next-token prediction, enabling chain-of-thought reasoning and majority voting. It unifies generation and verification, allowing knowledge transfer between tasks.


Results

  • GenRM outperforms discriminative RMs and LLM-as-a-Judge across tasks

  • On GSM8K, GenRM-CoT (Gemma-9B) improves problem-solving from 73% to 92.8%

  • Scales positively with model size, dataset size, and inference-time compute

  • Unified training boosts both verification and generation performance

LECTURE

PyTorch

A comprehensive overview of PyTorch's internal architecture and implementation details. 

⇧ 503 Likes

It covers the core concepts and mechanisms that power the PyTorch library. 

You'll gain insights into the tensor data structure, automatic differentiation system, and dispatch mechanisms. 

It helps you understand how PyTorch operates under the hood. This is valuable for optimizing your deep learning models and contributing to the PyTorch codebase.

Specific topics covered in this blog

  • Tensor data structure and storage

  • Stride-based memory layout and views

  • Automatic differentiation (autograd) implementation

  • Dispatch mechanisms for device types and dtypes

  • Extension points for custom tensor types

  • Kernel implementation techniques and tools

Read the blog

Thank You.

Want to promote your company, product, service, or event to 190,000+ AI researchers and engineers?

Sponsor