In this exclusive interview, Jay Kreps, CEO of Confluent and co-creator of Apache Kafka, shares his insights on the transformative power of data streaming for modern enterprises. As businesses strive to harness real-time data for better decision-making and competitive advantage, Kreps offers a deep dive into the current state of data streaming adoption, the challenges organizations face, and the pivotal role data streaming plays in enabling cutting-edge applications like real-time AI.
With his unique perspective as both a thought leader and a driving force behind the data streaming revolution, Kreps provides a glimpse into the future, where data streaming becomes the central nervous system for organizations, unlocking new levels of operational efficiency, customer experience, and revenue growth.
1. At what level are we on real time data streaming? How is it helping companies?
Data streaming fundamentally changed the way modern companies do business. With the emergence of Apache Kafka and the data streaming platform, companies can run in real time. Our 2023 Data Streaming Report shows that of 72% IT leaders are already using data streaming to power mission-critical systems.
With data streaming, teams can produce and consume data in real-time from completely disparate parts of the organization, allowing companies to realize real value from their data. As more parts of the business consume and contribute to data streams a virtuous cycle is created. As more data streams are produced, more data is available for different areas of a business to use, and the more value the organizations get out of it.
Data streaming is now a requirement for organizations across every industry, but we’re only just scratching the surface. I believe that the data streaming platform will be of similar importance and scale to databases, but acting as the central nervous system handling real time data.
2. As a developer of Apache Kafka, what are the critical challenges you see businesses facing in data integration and streaming processes?
Apache Kafka is the de facto standard for data streaming for good reason. Past solutions like ETL, messaging, and even databases forced point-to-point lossy connections that would break, lose data, and not work in real-time. These solutions led to a giant mess as companies tried to force these technologies to fit the real world. With Kafka, suddenly developers had access to a powerful system to streamline digital architectures in real time and break data silos.
While open-source Kafka offers powerful capabilities for data streaming, it isn’t a complete solution. And it can be costly and challenging to self-manage at scale.
Our goal at Confluent is to create a complete Data Streaming Platform with everything an organization needs to harness the full power of data. We’re taking on the hard parts of managing Kafka, so our customers can focus on business innovation and better products for their customers. We do that with a cloud-native service that is 10x better than Kafka and includes everything you need to connect every system and data source, process data in real-time and govern data as it flows across the business.
3. How close are enterprises to real time data access, and analytics, which will help them make decisions?
When I imagine a company that is fully operating with all data in real-time and able to harness the power of all of its data across the organization, it’s truly an incredible thing. But I can definitely say not all enterprises are there, yet.
At Confluent, we think about data streaming adoption by using the Data Streaming Maturity Curve. At the bottom of the curve are organizations that are just starting to experiment. At the very top are organizations that have achieved a central nervous system for their data. We continue to see a general shift up the curve as data streaming goes mainstream.
But the gap from production to a Central Nervous System for data is vast. It’s Confluent’s job to help organizations advance up the maturity curve, faster.
4. In recent years AI has taken the world by storm, but most of the data used to train AI models is historical rather than real time. This might be fine for predictive AI but what about real time AI use cases like fraud detection or personalized recommendations? How can data streaming help in real time AI applications?
One of the biggest barriers to AI innovation is access to clean and trustworthy data. The truth is, while you can have a great AI/ML model, without contextual, real-time
data from your business, the model is useless for most applications. Connecting AI models to enterprise data in real-time is one of the more challenging problems to solve.
RAG or retrieval augmented generation is the pattern that has become common for combining enterprise data with AI. Data streaming lets companies extract data from the many systems that run the business, transform it into the right format for their AI usage, and store it in a vector database for use with AI applications.
As data volumes grow and the demand for real-time responsiveness increases, the role of data streaming in powering real-time AI applications will become even more critical. Our Data Streaming for AI initiative includes new partnerships with vector databases, cloud providers and service partners, and in-product capabilities like a genAI-powered assistant, to make it easier for customers to build next-gen AI applications faster.
5. What impact is data streaming analytics having on enterprises' bottom lines and profitability? Can you share a few examples?
Data streaming enables real-time insights and decision-making, optimizing operations, enhancing customer experiences, and creating new revenue streams, all of which impact the bottom line. The ability to act on data in real-time is a significant competitive advantage, directly impacting profitability and driving growth and we’ve seen this with many of our customers at Confluent.
With Confluent, online retailers like Meesho can build real time recommendation engines that deliver great user experiences for customers and sellers. The Mobile Premier League (MPL) can keep its growing base of games secure with first rate personalization that increases user retention and engagement. And banks like Trust Bank can power real-time banking experiences for their customers with a lower cost of ownership than using open source Kafka