Skip to main content
Back

Apple Makes an Open-Source Comeback with OpenELM

DOWNLOAD OpenELM

April 26, 2024 5 min read

Hey,

Welcome to today's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

IN TODAY'S SIGNAL

  • Top News: Apple's New OpenELM

  • Top Announcements: Pytorch 2.3, Snowflake Arctic, Cohere Toolkit

  • Top Signals: OpenVoice V2, LLama 3 16k, FineWeb 15T

  • Top Lecture: Stanford's Transformers United V4

Read Time: 5 min 21 sec

Enjoying this newsletter?
Please forward it to a friend or colleague. It helps us keep this content free.

TOP NEWS

Language Models

Apple Releases OpenELM: Small, Open Source Language Models Designed To Run On-Device

⇧ 3829

What's New

Apple has unveiled OpenELM, a family of small yet efficient language models tailored for on-device applications. These models range from 270M to 3B parameters, making them suitable for deployment on mobile devices and computers.


Core Innovation: Layer-wise Scaling Architecture
The key innovation lies in OpenELM's "layer-wise scaling" architecture. It strategically allocates fewer parameters to the initial transformer layers near the input and gradually increases the parameter count towards the output layers. This approach optimizes compute resources based on the varying information complexity at each layer.


Model Performance and Benchmarks

  • OpenELM-1.1B outperforms AI2's OLMo-1B by 2.36% accuracy while using half as many pre-training tokens.

  • On ARC-C, MMLU, and HellaSwag benchmarks, the largest 3B model scored 42.24%, 26.76%, and 73.28% respectively.

Training Data and Availability

  • OpenELM models were trained on 1.8T tokens from datasets like RefinedWeb, deduplicated PILE, RedPajama subset, and Dolma v1.6 subset.

  • Both pre-trained and instruction-tuned checkpoints are available for all four model sizes (270M, 450M, 1.1B, 3B).

Open-Source and Licensing

  • Apple released OpenELM under a permissive "sample code" license, allowing commercial use and modification with retained license text on redistribution.

  • The CoreNet library used for pre-training and training recipes are also open-sourced to enable reproducibility.

Hardware Requirements
Benchmarks cited an Intel i9 workstation with RTX 4090 GPU and an M2 Max MacBook Pro as capable hardware for running OpenELM inference.

Why It Matters

As a vertically integrated hardware and software company, Apple's open-source OpenELM models pave the way for on-device AI assistants and language capabilities without privacy trade-offs, potentially setting the stage for more advanced device-centric AI experiences across Apple's ecosystem.

Community Feedback

Indira Negi: "Can’t wait for Apple to step into the LLM arena

They own the hardware in all our pockets. They have to be the one to do this 

Fingers crossed that they deliver the ability to run a decent model locally "


The AI Edge: "It seems like everyone is joining the trend of creating compact models, and this launch is another hint towards Apple’s possible advancements in on-device AI, which might be revealed at WWDC"


Vikash K Prasad: "Interesting that they have 270m and 460m odd parameters model finetuned that too instruction fine tuned. Parameter wise this is similar to Bert(non instruction) the first language model released by google in 2018."

Access

You also can run Apple's new OpenELM models in MLX LM:

pip install -U mlx-lm

DOWNLOAD OpenELM

Lightning AI Studio: Never set up a local environment again

Developed by the creators of PyTorch Lightning, this is where deep learning meets simplicity and speed. Prototype, train, serve, and so much more – all from one place. Zero setup.

  • Collaborative Deep Learning: Develop collaboratively in the cloud at any scale.

  • Massive Data Handling: Up to 12TB per studio and live distributed metrics.

  • Multi-node Jobs: Scalable and effortless, with no code changes required.

  • Advanced File Management: Visualize and manage over 100M+ files easily.

  • Flexible Connectivity: SSH, PyCharm, local VSCode integration, and AWS deployment..

Start today and get 22 free TP4 hours each month.

Try free

partner with us →

TOP ANNOUNCEMENTS

Pytorch

PyTorch 2.3 is out: Offers Support for User-Defined Triton Kernels in torch.compile

⇧ 751

PyTorch 2.3 introduces user-defined Triton kernels in torch.compile, enhancing performance without regressions or breaks in existing graphs. It supports tensor parallelism for training large-scale models and semi-structured sparsity, achieving up to 1.6x speedup in matrix operations.


Additional features include dynamic shape APIs in torch.export and asynchronous checkpoint generation.

Development

Snowflake Releases Arctic: An Open Source 480Bx128 MoE LLM

⇧ 1784

Snowflake has released Arctic, a 480B parameter Large Language Model utilizing a Dense-MoE Hybrid architecture with 128 experts, featuring 17B active parameters. This open-source model, licensed under Apache 2.0, combines a 10B dense transformer with 128x3.66B MoE MLPs. It also supports enterprise applications like SQL generation and complex instruction following.

Generative AI

Cohere Open-Sources 'Cohere Toolkit' to Accelerate Generative AI Application Development

⇧ 913

Cohere has released an open-source Cohere Toolkit that simplifies the integration of LLMs into enterprise systems by allowing single-click deployments on cloud platforms, including Azure. This toolkit packages models, prompts, user experience designs, and data connections into deployable applications, with the initial offering being a Knowledge Agent.

Easily Increase Your RAG Accuracy by 35%

Improve RAG retrieval accuracy in one line of code with Pongo's semantic filter. It uses a mixture of models and retrieval methods to score and order the top 100 results of your existing pipeline based on relevance to the query at runtime.


Sign up with the link below and get the first 500 queries for free

Try it free ↗️

TRENDING SIGNALS

Text-to-Speech

MIT CSAIL releases OpenVoice V2, a text-to-speech model that can clone any voice and speak in many languages.

⇧ 248

Startup

Eric Schmidt-backed Augment, a GitHub Copilot rival, launches out of stealth with $252M

⇧ 736

Datasets

HuggingFace releases FineWeb a 15 trillion tokens of high quality web data

⇧ 4352

Language Models

llamafile v0.8 release: 25x faster than ollama at running LLaMA3 70B on CPU

⇧ 1638

Language Models

Developers double Llama 3 context length to 16k tokens

⇧ 2422

TOP LECTURE

Transformers

Stanford's New Lecture on Transformers

⇧ 421

What's New

Stanford's first CS25 lecture, Transformers United V4, was just uploaded on Youtube. You can also attend it live on Zoom.


It features leading researchers discussing cutting-edge topics such as:

  • Large language model architectures (GPT, Gemini)
  • Creative applications (DALL-E for generating art, Sora)
  • Applications in biology, neuroscience, robotics, complex games

The series aims to provide an in-depth technical look at the state-of-the-art in transformer research. Scheduled speakers include experts who will dive into advanced concepts like:

  • Model alignment techniques (reinforcement learning, RLHF)
  • Sparse mixtures of experts architectures (Mixtral)
  • Developing precision language models for edge computing (U-LaMPS)
  • New training objectives for large language models
  • Code generation with language models (StarCoder)

The technical lectures offer insights into the latest transformer innovations from researchers at places like OpenAI, Google, NVIDIA, the Allen Institute, Mistral AI, Drexel University, Zhipu AI and more.

Watch now

Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.

Thank You.

Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?

Sponsor