Skip to main content
Back

LakeSail's Sail Replaces Spark With Rust, Running 4x Faster at 94% Lower Cost

Sail is an open source, Rust-native drop-in replacement for Apache Spark that runs existing PySpark code around 4x faster at roughly 94% lower cost.

Repo, Full Access
  • Sail is a Rust-native, drop-in Apache Spark replacement from LakeSail, Apache 2.0 licensed.
  • Runs unmodified PySpark code via Spark Connect, targeting Spark 3.5.x and 4.x compatibility.
  • Benchmarks show roughly 4x average speedup (up to 8x) and 94% lower compute cost on derived TPC-H.
  • Built on Apache Arrow and DataFusion, with zero-copy Python UDFs and Arrow-native shuffle.
  • Native Delta Lake and Iceberg support, plus Glue, Unity Catalog, Hive Metastore, and OneLake integration.
  • Install with pip install pysail; streaming support is on the roadmap, not yet shipped.

Spark has been the default engine for large-scale data processing for over a decade, but the JVM underneath it has become the tax nobody wants to keep paying: garbage collection pauses, slow startup, memory tuning, and shuffle spills that quietly balloon cloud bills. Sail, an open source project from LakeSail, tries to fix that without asking anyone to rewrite their pipelines.

Sail is a drop-in Apache Spark replacement written in Rust that unifies batch processing, stream processing, and compute-intensive AI workloads on a single distributed engine. It speaks the Spark Connect protocol, so existing PySpark SQL and DataFrame code runs unchanged. The project is Apache 2.0 licensed and available on PyPI as pysail.

Why rip out the JVM

If you run Spark in production, you already know the pain of tuning executors, chasing shuffle spills, handling OOM kills, and second-guessing your JVM GC config. Sail cuts all of it by dropping the JVM and Scala runtime entirely.

The engine leans on two well-known Rust projects to do the heavy lifting: Apache Arrow for columnar in-memory data and Apache DataFusion for SIMD-accelerated query execution. Workers are stateless and lightweight, with no GC pauses, no JVM startup overhead, and scale-to-zero support, which changes the economics of ephemeral clusters.