Hugging Face's TEI Serves Embeddings at 450 Req/Sec Without Python Overhead
Hugging Face's Rust-based embeddings server delivers up to 3x throughput over ONNX with dynamic batching, Flash Attention, and support for Qwen3, ModernBERT, and Gemma3.
PRO- Hugging Face's Text Embeddings Inference is a Rust server for open embedding and reranker models
- Benchmarks show ~330 req/sec at batch size 1 on A10, beating ONNX O4 and Transformers
- Supports Qwen3, Gemma3, ModernBERT, Nomic, GTE, Jina, Snowflake arctic-embed, and more
- Exposes OpenAI-compatible
/v1/embeddingsendpoint for drop-in replacement of paid APIs - Ships Docker images for Turing through Blackwell GPUs, plus Homebrew binary for Apple Silicon
- Includes SPLADE sparse embeddings, gRPC, Prometheus metrics, and OpenTelemetry tracing
If you have ever tried to squeeze real throughput out of a sentence-transformers deployment behind a Python web server, you know the pain: the model itself takes milliseconds, but the wrapper eats the rest. Text Embeddings Inference (TEI) is Hugging Face's answer, a Rust-based server built specifically to serve embedding and reranker models at production speed.
The project has crossed 4,700 GitHub stars and continues to add support for the newest embedding architectures. It covers popular families including FlagEmbedding, Ember, GTE and E5. The pitch is straightforward: point it at a model ID on the Hub, get an HTTP or gRPC endpoint back, and skip the graph compilation dance that plagues most inference stacks.
The numbers behind the speed claim
Hugging Face benchmarks TEI against vanilla Transformers and ONNX with O4-level optimization on BAAI/bge-base-en-v1.5 running on an NVIDIA A10 with 512-token sequences. At batch size 1, TEI serves roughly 330 requests per second versus about 290 for ONNX O4 and roughly 100 for Transformers, with per-request latency dropping to around 2.5ms.
The gap widens under load. At batch size 32, throughput climbs to roughly 450 req/sec while latency stays around 70ms, meaningfully ahead of the ONNX baseline. Those gains come from a stack of specific choices: token-based dynamic batching, optimized transformer code using Flash Attention, Candle and cuBLASLt, Safetensors and ONNX weight loading, and no model graph compilation step.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
