Hey, | Welcome to today's edition of AlphaSignal.
Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI.
Let's get into it!
Lior |
|
|
|
IN TODAY'S SIGNAL
| | Read Time: 5 min 39 sec |
|
|
|
Enjoying this newsletter? Please forward it to a friend or colleague. It helps us keep this content free.
|
|
|
|
BREAKTHROUGH
| ML Architecture | Disrupting Transformers with Parallelizable LSTMs |
⇧ 4102 Likes |
 |
What's New |
Sepp Hochreiter and his team introduced the xLSTM architecture, extending traditional LSTMs to compete with state-of-the-art Transformers.
The main innovation is a parallelizable LSTM, aiming to scale LSTMs effectively by leveraging techniques from modern LLMs and addressing known limitations of LSTMs.
Core Innovations The xLSTM architecture introduces exponential gating and modified memory structures: - Exponential Gating: Allows dynamic revision of storage decisions, overcoming traditional LSTM limitations.
- New Variants: sLSTM and mLSTM form the core of these innovations.
sLSTM and mLSTM Variants
sLSTM: - Features a scalar memory cell and scalar update with a new memory mixing mechanism.
- Stabilizes exponential gates through a normalizer state that sums the product of the input gate times all future forget gates. (This helps maintain stability by balancing inputs and forget signals.)
mLSTM:
- Replaces the scalar memory with a matrix memory, using a covariance update rule to store key-value pairs optimally. (mLSTM uses matrices for memory, improving storage capacity.)
- Fully parallelizable, addressing the sequential processing limitation of traditional LSTMs.
Integration and Architecture - xLSTM blocks integrate the new memory structures into residual modules with either:
- Post up-projection (sLSTM)
- Pre up-projection (mLSTM)
- These blocks are residually stacked into xLSTM architectures, enhancing memory capacity and computational efficiency.
Performance Metrics
- xLSTM demonstrates strong performance on large-scale language modeling tasks:
- Models trained on 300B tokens from SlimPajama.
- Ranging from 125M to 1.3B parameters.
- Maintain low perplexities for longer contexts.
- Outperform other models on various downstream tasks.
Efficiency in Long-Context Problems
- xLSTM shows linear computation and constant memory complexity with respect to sequence length. (xLSTM's performance remains efficient even with long inputs.)
- Synthetic tasks and the Long Range Arena benchmark confirm xLSTM's advantages in handling long sequences:
- Exponential gating and enhanced memory capacity contribute to performance.
- Results indicate effective scaling and promising potential for large-scale applications.
|
Access |
An implementation notebook by Andrew Gao is available here |
Community Feedback |
Sepp Hochreiter I am so excited that xLSTM is out. LSTM is close to my heart - for more than 30 years now. With xLSTM we close the gap to existing state-of-the-art LLMs. With NXAI we have started to build our own European LLMs. I am very proud of my team. |
Armen Aghajanyan
Did a quick look over the xLSTM paper, unfortunately there might be some problematic experiments. 1. When do scaling laws, experiments need to be FLOP controlled (either theoretical FLOP's or effective FLOP's). Unless LLaMA/Mamba/RWKV/xLSTM are all the same exact FLOP's for the exact same parameter count this is problematic. As an example imagine if you plotted MoE scaling laws with number of parameters instead of FLOP's. |
|
Read the paper
|
|
|
|
| Webinar: Stop Hallucinations in Real-time with a GenAI Firewall | Galileo just introduced "Protect", the first GenAI firewall that intercepts hallucinations, prompt attacks, and security threats.
Register now to the webinar to see Galileo Protect live in action and learn:
How Galileo Protect works, including our research-backed metrics
The key features of an enterprise GenAI firewall
What makes Galileo Protect unique |
|
Register now
| partner with us → |
|
|
|
TRENDING SIGNALS
| Industry |
|
⇧ 484 Likes | |
Biology |
|
⇧ 10,201 Likes | |
Prompt Engineering |
| ⇧ 51 Likes |
| Tutorial |
|
| Segmentation |
|
⇧ 388 Likes | | |
|
|
|
|
TOP OF HUGGINGFACE
| Benchmarks |
| ⇧ 110 Likes |
This Performance Leaderboard evaluates serverless models behind API endpoints using a Quality Index.
This index is a weighted average of normalized scores from MMLU, MT-Bench, and Chatbot Arena. Benchmark scores are sourced from model creators for MMLU, MT-Bench and HumanEval where reported, and from LMSYS Chatbot Arena Leaderboard for Chatbot Arena scores.
Not only does the leaderboard provide the overall score for each model, but also reports the latency, throughput, cost and context length. It gives a good overview of existing models and helps with choosing the right model for a specific use case. | |
Image Generation | |
⇧ 119 Likes | PuLID is a tuning-free ID customization method for text-to-image generation that is performant and highly editable.
Using one or multiple initial images and a prompt, the model is capable of generating between one and four new images that integrate both the face of in the image and the style of the prompt, without further training needed.
PuLID actually introduces both contrastive alignment loss and accurate ID loss, minimizing disruption to the original model and ensuring high ID fidelity, while integrating the prompt. |
| Datasets |
| ⇩ 359 Downloads |
The Youtube Commons Dataset is a collection of 30 billion words from 15,112,121 original and automatically translated transcripts from 2,063,066 videos on Youtube. All the videos where shared on YouTube with a CC-BY license, and most of them are in English (71%).
The text can be used for training a model and republished for reproducibility purposes.
This dataset is the result of an ongoing collaboration between state start-up LANGU:IA, the French Ministry of Culture and DINUM. It will be expanded in the coming months. |
| |
| |
|
|
PYTORCH TIP
| Industry | Write and Use Custom CUDA Extensions for Critical Operations |
Andrej Karpathy's work on llm.c using C and CUDA reminded us that PyTorch is not always the most optimized framework, although it is practical and easy to understand.
Luckily, PyTorch allows you to write custom extensions using CUDA, which can dramatically accelerate parts of your model that are either unique or not optimally supported by the existing PyTorch library.
You can start by using profiling tools to identify specific operations in your model that are potential bottlenecks and could benefit from a custom CUDA implementation. Then, you can write a custom CUDA kernel, like the one below: |
__global__ voidmy_kernel(float *x, float *y, int size) { int index = blockIdx.x * blockDim.x + threadIdx.x; if(index < size) { y[index] = 2 * x[index]; // Example operation } }
| Then, you can use PyTorch's cpp_extension module to create a bridge between your CUDA kernels and your PyTorch code.
Once compiled and loaded, the custom operation can be used directly in your PyTorch models like any other function.
Here's how you can use this custom CUDA kernel in Python in an easy example: |
from torch.utils.cpp_extension import load import torch
custom_op = load( name="custom_op", sources=["my_kernel.cu"], extra_cuda_cflags=["--expt-relaxed-constexpr"], )
tensor = torch.randn(1024, device='cuda') result = custom_op.my_kernel(tensor)
|
|
|
| Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.
| | | |
Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?
|
|
Sponsor
|
|
|
|
|