Real-Time ML Pipelines: Feature Chaining with Chronon
Modern ML applications demand features computed in near real-time with sub-100ms latencies. This talk dives into Chronon, an OSS feature platform bridging streaming data infrastructure and production ML. Using a two-tower search pipeline example, we’ll show how we can chain embeddings with tabular features while minimizing hot-path computation.
Turning the database inside out again
We rethink data systems by putting streams at the center. Expanding on Martin Kleppmann’s: Turning the Database Inside Out, this talk shows how Apache Kafka and Apache Iceberg together provide durable storage, indexing, and rich views that eliminate brittle ETL and unify real-time and historical analysis. A new way to see databases—and streams.
Streamling: Lightweight, Extensible Streaming on DataFusion
Apache DataFusion is moving beyond batch into streaming. We built Streamling, a Rust streaming engine that uses DataFusion planning and Arrow RecordBatch streams for real-time SQL/WASM transforms. This talk covers how we built it, highlights key features (FFI plugins, WASM transforms, and dynamic tables), and shares production lessons.
Beyond the Hype: When Apache Flink Solves Real Problems
When does Apache Flink solve real problems versus add complexity? Explore use cases where Flink becomes essential such as fraud detection, CDC, real-time analytics versus when batch or Kafka Streams suffice. Compare stream engines (Flink, Spark) with platforms (Kafka, Pulsar) to confidently decide when streaming delivers value.
Event-driven Agents with Complex Event Processing in Flink
Event-driven Agents calling LLMs can be combined with Pattern Recognition and Anomaly Detection in Apache Flink in smart ways to increase cost efficiency, avoid hallucinations and enforce predictable, deterministic behavior. Specifically in a business process context, this architecture provides opportunities for continuous real-time process mining.
The Agent Era: How AI Agents Are Reshaping Data Platforms
AI agents have quietly become some of the most demanding users of modern data platforms and most weren’t built with them in mind. In this panel, leaders from Snowflake, Elastic, MotherDuck, and Xata share what agentic workloads actually look like in production: what broke, what had to be rebuilt, and where the architecture is heading.
The Three-Body Problem of Inverse Hybrid Search
When users expect alerts for new products matching an uploaded image, the problem becomes inverse hybrid search. Unlike top-K search, alerting must guarantee fetch-all semantics: zero missed matches across all saved searches, combining vector similarity, boolean filters, and lexical signals. We show why this breaks traditional scaling intuition.
Apache Spark Declarative Pipelines in Action
Learn Spark 4.1’s brand-new Declarative Pipelines, a paradigm shift replacing imperative code with simple declarations. We’ll build a real-time data pipeline together, processing streaming ADS-B flight data from tens of thousands of aircraft overhead.
Keeping data private in real-time pipelines
Real-time data is awesome… until you realize it’s leaking names, emails, and locations. In this talk, you’ll learn how to keep streaming data private, from simple masking to tricks that beat re-identification. All with live demos and some juicy real-world stories.
What If We’ve Been Scaling Stream Processing Wrong All Along
We’ve normalised extraordinary inefficiency in stream processing. Thousands of events/sec don’t justify repartition storms, serialization overhead, state migration. This talk explores a different path: Kafka Streams DSL, adopt Flink-like exactly-once semantics, Project Loom, and challenging the assumption that stream processing must be distributed.