Skip to main content
Back

Building Your Own Personal AI Assistants

TRY KHOJ

May 27, 2024 4 min read

Hey,

Welcome to today's edition of AlphaSignal. 


Whether you are a researcher, engineer, developer, or data scientist, our summaries are there to keep you up-to-date with the latest breakthroughs in AI. 


Let's get into it!


Lior

.

IN TODAY'S SIGNAL

Top Repo

Snorkel

Top Signals

Top Papers

  • SimPO streamlines language model training by eliminating the need for reference models.

  • Meta pioneers a clustering-based approach for automated, high-quality dataset curation.

  • Aaren blends RNN updating efficiencies with attention mechanisms for improved performance.

Tutorial

Read Time: 4 min 42 sec

Enjoying this newsletter?
Please forward it to a friend or colleague. It helps us keep this content free.

TOP REPO

AI Assistants

Khoj: An Agentic AI second brain

☆ 8931 Stars

What's New

Khoj is an application that creates always-available, personal AI agents for you to extend your capabilities.


Inputs


You can share your notes and documents with Khoj to extend your digital brain. It supports files in all formats, like pdf, markdown, org-mode, notion files and even GitHub repositories. But the AI agents deployed also have access to the internet, allowing you to incorporate realtime information to the responses.


Features

  • You'll get fast, accurate semantic search on top of your docs.
  • Your agents can create deeply personal images and understand your speech.
  • You will get incremental search for a fast, search-as-you-type experience
  • You have the option to keep your personal data local, meaning all search and indexing can done on your machine if you self-host. 

Access


Khoj is accessible on Desktop, Emacs, Obsidian, Web and Whatsapp. You can se use cloud to access Khoj anytime from anywhere, and self-host on consumer hardware for privacy. 


To self-host, the setup process involves installing Postgres, configuring the Khoj server (either via Docker or local installation), and optionally downloading desktop clients and plugins for integration with tools like Obsidian and Emacs.


This project is open-source, and self-hostable, and apparently intends to stay this way. 

Try khoj

Webinar: Learn How To Accelerate Data Labeling—go from months to days

Enterprise AI efforts are often blocked by the time and cost required to manually label data, whether by in-house SMEs or crowdsourcing. 


The fastest way to get unblocked is by creating data labeling functions which can capture SME knowledge and automatically apply it at scale. 


Join us and discover how to accelerate data labeling by: 

  • Capturing SME knowledge in reusable data labeling functions 

  • Leveraging LLM prompting to further accelerate data labeling at scale 

  • Measuring label accuracy and improving it with iteration 

June 6, 2024
10:00 AM PT / 1:00 PM ET


Can’t attend live? All registrants will receive an on-demand recording

Register now (free)

partner with us

TRENDING SIGNALS

Theory

New Model Architecture, Wav-KAN, overcomes MLPs interpretability, speed, and robustness limitations

⇧ 470 Likes

Safety

Stanford's new Model Transparency Index: evaluates 100 aspects of AI model transparency for 10 foundation models

⇧ 239 Likes

Object Detection

YOLOv10 is out, the new real-time object detection model has 46% less latency and 25% fewer parameters than YOLOv9

⇧ 1105 Likes

Inference

Mozilla Llamafile now allows you to run inference for Mixtral 8x22b at 20 tokens/sec on regular CPUs

⇧ 213 Likes

Industry

Musk's xAI Raises $6 Billion in Bid to Challenge OpenAI

⇧ 6829 Likes

TOP PAPERS

Fine-Tuning

SimPO: Simple Preference Optimization with a Reference-Free Reward

⇧ 987

Problem: Direct Preference Optimization (DPO), is a widely used optimization algorithm for fine-tuning language models. It reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability, but it is still computationally heavy and requires a reference model.


Solution: This paper propose SimPO, a simpler yet more effective preference optimization algorithm. The new reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient.


Results: SimPO outperforms existing preference optimization methods, including DPO and its variants, across various training setups and benchmarks. It achieves up to 6.4 point improvement on AlpacaEval 2 and 7.5 point improvement on the challenging Arena-Hard benchmark compared to DPO. 

Datasets

Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach

⇧ 501

Problem: The construction and curation of data collections for self-supervised pre-training typically require extensive human effort, which is costly, time-consuming, and limits the ability to scale the dataset size. This manual process has limitations similar to those encountered in supervised learning.


Solution: This paper by Meta proposes a clustering-based approach for automatic curation of high-quality datasets for self-supervised pre-training. The method involves successive and hierarchical applications of k-means clustering on a large and diverse data repository to obtain clusters that distribute uniformly among data concepts, followed by a hierarchical, balanced sampling step from these clusters. 


Results: Experiments on different data domains demonstrate that features trained on the automatically curated datasets outperform those trained on uncurated data. Plus, the performance of features trained on the automatically curated datasets is on par or better than those trained on manually curated data.

Language Models Architecture

Attention as an RNN

⇧ 1520

Problem: Transformers are computationally expensive at inference time, limiting their applications in low-resource settings such as mobile and embedded devices.


Solution: The authors introduce Aaren, an attention-based module that can be trained in parallel like Transformers and updated efficiently with new tokens like traditional RNNs. They show that attention can be viewed as a special RNN with the ability to compute its many-to-one RNN output efficiently, and that popular attention-based models such as Transformers can be viewed as RNN variants.


Results: Empirically, this paper demonstrates that Aarens achieve comparable performance to Transformers on 38 datasets across four popular sequential problem settings: reinforcement learning, event forecasting, time series classification, and time series forecasting tasks. This new architecture is also more time and memory-efficient than Transformers.

TUTORIALS

Implementation

All ML Algorithms Implemented in Python

If you want to learn the internals of machine learning algorithms or implement them from scratch, this allows you to do just that.


This repo offers minimal and clean implementations of a wide range of machine learning algorithms. It contains code that is easier to follow compared to optimized libraries, which makes it more accessible for those new to the field or looking to experiment.


The repo covers various algorithms including deep learning models (MLP, CNN, RNN, LSTM), linear regression, logistic regression, random forests, and SVMs with different kernels.


It uses Python libraries such as NumPy, SciPy, and autograd, making these complex algorithms approachable and easier to manipulate and understand.

Read more

Ghita is an editor at AlphaSignal and AI Engineer at Clear Ventures. Experienced in Computer Vision and Language models, she holds 2 masters in Applied Mathematics.

Thank You.

Want to promote your company, product, job, or event to 180,000+ AI researchers and engineers?

Sponsor