Skip to main content
Back

Now, unlock 90% Savings with Claude AI

READ MORE

August 16, 2024 4 min read

Hey,

Welcome to today's edition of AlphaSignal, a newsletter for developers by developers.

We identify and summarize the top 1% news, papers, models, and repos in the AI industry. 

IN TODAY'S SIGNAL

Read time: 4 min 54 sec

🎖️ Top News

  • Anthropic launches prompt-caching with 90% cost reduction, 85% lower latency.

📌 Ori

  • Ori Serverless Kubernetes: AI cloud service with affordable, scalable, simple management. Pay-per-use pricing. Autoscaling, powerful GPUs. $1000 signup credit.

⚡️ Trending Signals


🛠️ Top Repos

  • MetaGPT: Build entire software projects with one-line.

  • Deep-Live-Cam: Real time face swap and one-click video deepfake.

  • MiniCPM-V: GPT-4V Level MLLM for Single Image, Multi-Image and Video on Your Phone

🧠 Tutorial

  • Access the internal workings of a model during its forward pass with PyTorch.

If you're enjoying AlphaSignal please forward this email to a colleague. 

It helps us keep this content free.

TOP NEWS

LLM Inference Cost

Anthropic launched prompt-caching feature with 90% cost reduction, 85% lower latency

⇧ 14,329 Likes

What's New

Anthropic has introduced prompt caching for Claude AI models, allowing developers to reuse large portions of prompts between API calls. This slashes costs by 90% and reduces response time by 85%.


Key Features

  • Available for Claude 3.5 Sonnet and Claude 3 Haiku, with Claude 3 Opus support coming soon.

  • Ideal for Large Documents and repetitive instructions.

  • Supports Multi-Turn Conversations.

  • Pricing:
    Claude 3.5 Sonnet: $3/Mn tokens input, $3.75/Mn tokens cache write, $0.30/MTok cache read
    Claude 3 Haiku: $0.25/Mn tokens input, $0.30/Mn tokens cache write, $0.03/MTok cache read

  • Cache write tokens are 25% more expensive than base input tokens, while cache read tokens are 90% cheaper than base input tokens.


How It Works

Developers only need to add a cache control attribute ("cache_control": ) to content they want to cache and include a beta header ("anthropic-beta": "prompt-caching-2024-07-31") in API calls. 


The system checks for cached content on new calls and uses it if available. Caches have a 5-minute lifetime, refreshed with each use. Prompts are cached at 1024-token boundaries.


Cost and Performance Benefits

  • Up to 90% reduction in API input costs

  • Up to 85% reduction in latency for long prompts

  • Initial caching call: 25% more than base input token price

  • Subsequent cached content use: 10% of base input token price


Applications

Prompt caching enables efficient incorporation of entire codebases, use of hundreds of examples for improved task performance, and processing of large documents. It benefits conversational agents, coding assistants, and applications involving extensive document processing or detailed instruction sets.

Read more

Discover a new way to run AI inference

Meet Ori Serverless Kubernetes, an AI cloud service that brings you the best of Serverless and Kubernetes by blending affordability, powerful scalability, and simple management. Their transparent, pay-per-use pricing means you only pay for the resources you consume, helping you optimize GPU costs. 

  • Autoscaling: Adapt your AI infrastructure to inference demand, whilst optimizing costs.

  • Simplicity of Serverless: They take care of cluster management, load balancing and scaling.

  • Vanilla Kubernetes: No refactoring or learning curve for Kubernetes users.

  • Powerful GPUs: Choose between the NVIDIA H100, L40S, L4, or V100S.

Get started with $1000 in matched credit when you sign up.

Start integrating today

partner with us

TRENDING SIGNALS

Language Models

NVIDIA refined Llama 3.1 8B into smaller Llama-3.1-Minitron 4B using structured weight pruning and knowledge distillation

⇧ 5843 Likes

Language Models

Nous Research dropped new highly performant LLM Hermes 3, specialized in roleplaying, agentic tasks, function calling

⇧ 2003 Likes

Image Generation

Midjourney releases new web editor combining many separate image actions into a single unified interface

⇧ 2160 Likes

Video Generation

Runway released Gen-3 Alpha Turbo Image to Video, 7x faster and for half the price, while maintaining performance

⇧ 889 Likes

AI Software Engineer

Salesforce releases an open AI software engineering agent with a 55% resolve rate on SWE-Bench Lite

⇧ 425 Likes

TRENDING REPOS

Agent

MetaGPT

☆ 42,812 Stars

Helps you build software projects with one-line. It generates user stories, requirements, APIs, and code using a multi-agent system simulating a software company. Supports various LLMs and includes a Data Interpreter for analysis tasks and RAG support.

Deepfake

Deep-Live-Cam

☆ 24,016 Stars

Helps you perform real-time face swaps and create deepfake videos using a single image. It uses LLMs to swap faces in images, videos, and live webcam feeds with high-quality results while maintaining ethical safeguards. It supports both GPU and CPU processing.

Multimodal LLMs

MiniCPM-V

☆ 10,570 Stars

Helps you run a GPT-4V level multimodal LLM on your phone.  The models take image, video and text as inputs and provide high-quality text outputs. Outperforms GPT-4V, Gemini Pro in single image understanding. Supports real-time video understanding on iPad. Easy deployment via llama.cpp, ollama, vLLM.

PYTORCH TIP

Access the internal workings of a model during its forward pass with PyTorch

TorchLens is a Pytorch library that extracts intermediate outputs and metadata from PyTorch neural network models. 


What does it do and why it is helpful

  • Accesses model internals during forward pass without code modification.

  • Extracts layer activations and model structure metadata.

  • Aids in debugging, behavior analysis, and understanding information flow.

  • Works with any PyTorch model, supporting tasks like feature visualization and architecture comparison.

Working example with code
The example demonstrates TorchLens with a simple recurrent neural network. The model performs four iterations of a linear transformation followed by basic arithmetic operations. TorchLens logs the forward pass, saving all intermediate outputs. Users can then access specific layer outputs, such as the second pass of the first linear layer, as shown in the printed tensor.


# To install TorchLens, first install graphviz
# (required to generate the network visualizations)
sudo apt install graphviz

# and then install TorchLens using pip
pip install torchlens -q

class SimpleRecurrent(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(in_features=5, out_features=5)

    def forward(self, x):
        for r in range(4):
            x = self.fc(x)
            x = x + 1
            x = x * 2
        return x


simple_recurrent = SimpleRecurrent()
model_history = tl.log_forward_pass(simple_recurrent, x,
                                    layers_to_save='all',
                                    vis_opt='rolled')
print(model_history['linear_1_1:2'].tensor_contents)
# second pass of first linear layer

"""
tensor([[-2.9265, 1.3072, -0.4358, -0.9827, -3.0030],
        [-2.1188, 0.6454, 0.1414, -1.0781, -2.4770],
        [-1.8614, 1.0404, -0.7678, -0.7843, -2.4568],
        [-4.4318, 1.4188, -1.6984, -0.3384, -5.2524],
        [-1.8989, 0.8374, -0.7154, -0.6169, -2.7501],
        [-3.2618, 1.5131, 1.2455, -1.7886, -1.5585]],
       grad_fn= 0>)
"""



Code in google colab

Thank You.

Want to promote your company, product, service, or event to 190,000+ AI researchers and engineers?

Sponsor