Skip to main content
Back

Meta's OpenZL Beats Zstd With 2x Compression by Learning Your Data's Structure

OpenZL v0.2.0 ships a rebuilt SDDL compiler, a native LZ codec, and automatic chunking for multi-GB streams

Repo, Full Access
  • OpenZL v0.2.0 released: Meta's format-aware compression framework ships its first major update since the v0.1.0 debut.
  • SDDL2 is a real compiler: The schema language was rebuilt from scratch — now parses at ~2.4 GB/s, ~300x faster than SDDL1.
  • Native LZ codec: New ZL_GRAPH_LZ codec is 10% faster to compress and 70% faster to decompress vs. zstd level 1.
  • Multi-GB stream support: Inputs are automatically chunked (~16 MB default), keeping memory bounded and enabling parallel processing.
  • Conditional fields in SDDL2: New when blocks let you describe optional, version-dependent, or flag-gated fields natively.
  • Free and open source: BSD-licensed, available on GitHub with C, C++, and Python APIs.

Meta's format-aware compression framework OpenZL just shipped its v0.2.0 release, announced by lead maintainer Yann Collet (Cyan4973 , the same engineer behind Zstandard). This release bundles several months of work since the original v0.1.0. The headline additions: a ground-up rewrite of the SDDL schema engine, a native LZ codec, and the ability to compress gigantic multi-GB input streams without blowing up memory.

If you haven't encountered OpenZL before: the core idea is that most data in production isn't random bytes , it follows a schema. In most datacenters, there are massive data flows, and this data typically follows a well-defined schema or format. If a compression algorithm can be taught about the structure of the data it is compressing, it can achieve significantly better compression performance. OpenZL lets you describe your data's layout, then builds a specialized compressor tuned to that exact format , while still using a single universal decompressor on the other end.

OpenZL is viewed as Meta's next leap in data compression beyond their work on Zstandard (Zstd). On the Silesia SAO corpus benchmark, it already achieves a 2.06x compression ratio at 203 MB/s , compared to zstd -3's 1.31x at 115 MB/s, and xz -9's 1.64x at a painful 3.1 MB/s.