Databricks' Lakeflow Wants AI Agents to Replace Your Entire Data Stack

Data engineering has always been the unglamorous backbone of AI. Before any model runs, someone has to wire up dozens of connectors, babysit pipelines, and debug failures at 2am. Databricks just announced a sweeping upgrade to Lakeflow, its unified data engineering platform, that aims to hand most of that toil to AI agents , and in doing so, take direct aim at a sprawling ecosystem of specialized tools.

One platform to rule the modern data stack

Lakeflow is designed as a unified platform for all of data engineering , from ingestion to transformation and orchestration , with every capability fully integrated and centrally governed by Unity Catalog, giving AI agents a single, trusted source of real-time context. The announcement, made at the Data + AI Summit in San Francisco, covers five major product areas that collectively replace what most teams currently stitch together from Fivetran, Apache Airflow, Apache Kafka, and Apache Flink.

The default 2026 enterprise data stack centers on Snowflake or Databricks for storage, dbt for transformation, Apache Airflow for orchestration, and Airbyte or Fivetran for ingestion. Databricks is now betting it can collapse that entire stack into one platform , and that the agentic era makes the timing right.

What actually shipped

• Lakeflow Connect , 100+ connectors: Lakeflow Connect is expanding to support more than 100 native, managed connectors across enterprise applications, databases, file sources, and cloud storage , covering sources like Jira, GitHub, Confluence, SharePoint, Google Drive, and Outlook. A new free tier gives customers 100 free DBUs per day, supporting up to 100 million records daily across popular managed SaaS and database connectors.
• Zerobus Ingest , Kafka without Kafka: Zerobus Ingest handles high-volume event data with no message bus required, delivering near real-time writes in under 5 seconds and throughput up to 100MB/s (over 10GB/s per table). Critically, Kafka-Compatible APIs (Beta) let existing Kafka producers push data straight to Databricks with no code changes required. Meta is already using it in production: "We cut our end-to-end pipeline latency to under a minute with Zerobus Ingest and Spark Declarative Pipelines."