Apache Spark 4.3 Ships Real-Time Stateful Streaming Under 150ms

Apache Spark 4.3 adds stateful processing to Real-Time Mode, Spark’s low-latency Structured Streaming execution path. Available through Databricks Runtime 19, the release also expands change data capture, accelerates small-data iteration, improves Spark Connect, and adds several SQL features.

The Apache Spark project is also shifting from semiannual to quarterly releases. The three-month cycle should reduce the time between merging a JIRA issue and shipping the resulting code in a release.

State enters the low-latency path

Earlier versions of Real-Time Mode could not run common stateful operations, so applications that maintained information across events often required a separate execution path or engine. Spark 4.3 initially supports three stateful capabilities:

• Streaming aggregations that update results as records arrive
• dropDuplicates for stateful deduplication
• transformWithState for custom keyed state in Scala and Java

These operators can retain information such as account activity, device status, or session history across events while using Structured Streaming’s existing programming model. Candidate workloads include fraud scoring, telemetry processing, and session-level personalization.

Benchmarks published in SPARK-58884 reported P99 end-to-end latency below 150 milliseconds for all three representative workloads. P99 means that 99 percent of measured results completed within the stated latency.

Actual latency will vary with cluster size, input rate, state size, key skew, and query structure. The published figures cover three specific workloads and do not provide a general service-level guarantee.

Pipelined shuffles run stages together

Traditional blocking shuffles require an upstream stage to finish producing its output before the downstream stage starts reading it. That scheduling model prevents continuous data flow between long-running streaming tasks.

Spark 4.3 introduces native concurrent stage scheduling and pipelined shuffles. A new PipelinedShuffleDependency allows consumers to read data while producers remain active, reducing the wait between connected stages.

The scheduler treats connected stages as one pipelined group, launches them together, and handles failures at the group level. Spark also checks whether the cluster has enough task slots for the entire group and fails early when capacity is insufficient, avoiding a stalled query. Because the capability sits in Spark’s scheduler and shuffle layers, other workloads can reuse it in future releases.

Auto CDC keeps row history

Change data capture feeds describe inserts, updates, and deletes from source systems. Slowly Changing Dimension Type 2 processing preserves those changes as historical row versions, allowing queries to reconstruct a record at an earlier point in time.

Spark 4.3 adds end-to-end SCD Type 2 support to open-source Auto CDC through Python, SQL, and Spark Connect. Auto CDC applies incoming changes and maintains __START_AT and __END_AT metadata columns for each version. The result supports historical reporting, audits, and point-in-time feature generation without hand-written merge logic.

Auto CDC still applies each micro-batch to its target with MERGE. The target must therefore use a catalog connector that implements Data Source V2 row-level operations. A standalone Parquet target lacks those operations; Iceberg provides a suitable open-source target for this workflow.

Feather trims tiny-query overhead

AI-assisted development often involves repeatedly running a transformation against a small sample, where distributed shuffle overhead can dominate execution time. Project Feather adds opt-in, shuffle-free, single-task execution for eligible single-partition query fragments.

Feather can optimize selected scans, aggregations, distinct operations, windows, sorts, and limits. Eligibility remains narrow: each scan may contain one nonempty small file, while local in-memory relations may contain up to 1,000 rows.

Feather also provides an opt-in cache serializer that stores cached batches in Apache Arrow’s columnar format. Enable it with the following setting:

spark.sql.cache.serializer=org.apache.spark.sql.execution.columnar.ArrowCachedBatchSerializer

Connect becomes traceable

Spark Connect, the gRPC-based remote execution layer, gains REST endpoints for listing and inspecting sessions and operations. Operation IDs can follow an individual request across the client and server, making failures and slow operations easier to trace.

Local development on POSIX systems can use the experimental SPARK_LOCAL_CONNECT_REUSE=1 option to keep a Connect server running across executions and avoid repeated startup. The JDBC driver also adds ARRAY, MAP, and STRUCT results, along with improved schema introspection.

Nanoseconds, ASOF joins, and JSON

Where Spark 4.3 ships

Open-source builds are available from Apache Spark downloads, and the migration guide covers compatibility considerations. Databricks customers can use Spark 4.3 through Runtime 19.