(Quardia/Shutterstock)
Confluent in the present day introduced the final availability of Tableflow, the Apache Iceberg-based performance that it first revealed a yr in the past. The corporate additionally used its convention this week in Bengaluru, India because the platform to launch Flink Native Inference, a brand new Apache Flink-based functionality designed to make it simpler to implement AI inference on streaming information.
It’s been virtually precisely a yr since Confluent introduced it was including a brand new characteristic referred to as Tableflow to Confluent Cloud, the corporate’s hosted model of Apache Kafka. Tableflow makes it straightforward for patrons to stream any information flowing in a Kafka matter immediately into an information lake as a desk within the Apache Iceberg format. Along with the info, Tableflow grabs related metadata, enabling the desk to get all the advantages of Iceberg administration, together with help for ACID transactions.
Many Confluent prospects tried to construct this Iceberg functionality themselves as they moved information from operational to analytical programs, stated Adi Polak, director of advocacy and developer expertise for Confluent.
“However it takes time, sources, and value to construct these further information pipelines,” she stated. “So what we did in Confluent is alleged, how about we’ll create this Tableflow for you, and with a click on of a, button you don’t even want to consider it.”
Along with Iceberg, Tableflow is now supporting Delta Lake, the desk format created by Databricks to be used with its information lake, or lakehouse platform, Polak stated. Whereas Databricks dedicated to supporting each Iceberg and Delta Lake following its acquisition of Tabular final yr (and finally merging them), the 2 codecs proceed for use independently. Since Confluent and Databricks solid a strategic partnership final month, it made sense for Confluent to help Delta Lake, too.
Along with creating Iceberg or Delta Lake tables out of Kafka matters, Confluent can also be creating the info essential for the tables to be found and managed by the metadata catalogs. The corporate is supporting AWS Glue and Snowflake’s Polaris catalogs out of the gate, Polak stated.
Help for Iceberg and Delta Lake is essential for Confluent prospects as a result of it makes it simpler to attach their transactional (or operational) and analytical programs. Confluent has been working with media corporations that might periodically dump information from their operational functions, together with Kafka, Confluent Cloud, and Flink streams, into an information warehouse for the aim of feeding dashboards and advert hoc queries. However the corporations wished so as to add real-time capabilities.
The corporate’s different large announcement is round Apache Flink, the favored information processing engine that works on streaming and static information. Confluent has been integrating the Flink stream processing engine into its Kafka streaming information pipelines for the previous yr. With the launch of Flink Native Inference, the mixing between Flink and Kafka will get even deeper.
In accordance with Polak, many Confluent prospects need to run machine studying or AI fashions in opposition to their streaming information. However taking the info out of Confluent Cloud to run ML or AI algorithms enhance latency and privateness considerations. The answer is to make use of Flink to run arbitrary machine studying fashions in opposition to streaming information, all hosted throughout the Confluent Cloud.
“We’re enabling them to have native inference on high of Confluent Cloud,” Polak stated. “That offers them flexibility and safety. We additionally assist them with value effectivity on the compute aspect in addition to latency, as a result of now they’re working their stream pipeline adjoining to the place their mannequin is being hosted. It is a recreation changer for lots of our prospects.”
Whether or not the AI is a homegrown mannequin developed in PyTorch or an open supply mannequin like DeepSeek or Llama, prospects can name it utilizing the Flink API and Flink SQL capabilities, and run it immediately inside their Confluent Cloud account.
The corporate introduced two different Flink capabilities, together with Flink Search, which provides prospects a approach to carry out vector searches throughout MongoDB, Elasticsearch, and Pinecone inside Confluent Cloud’s Flink SQL; and Constructed-in ML Features (early entry), which brings entry to Confluent-developed algorithms for information science duties, together with forecasting, anomaly detection, and real-time visualizations.
Flink Search will allow prospects to construct retrieval augmented technology (RAG) pipelines which are well-grounded, Polak stated.
“Numerous the challenges with AI fashions is round hallucinations,” she stated. “I’m taking an present mannequin, I’m deploying it, but it surely’s hallucinates as a result of it’s not grounded in latest context. And for that, we want a RAG sample or a RAG structure, and that is precisely what our Flink Search permits.”
All three Flink capabilities can be found as early entry on Confluent Cloud, that means the performance might change and it might not be totally steady. Confluent made these bulletins from Present Bengaluru 2025, its Kafka convention within the South Indian metropolis of 14 million (also referred to as Banglalore). Tickets for the present, which started Tuesday March 18, are offered out.
“We’re doing it in India as a result of there’s a number of pleasure in India for information streaming,” Polak stated. “We see an enormous development on this inhabitants round information streaming, and information streaming engineers as properly.”
Associated Gadgets:
Confluent and Databricks Be part of Forces to Bridge AI’s Knowledge Hole
Confluent Goes On Prem with Apache Flink Stream Processing
Confluent Provides Flink, Iceberg to Hosted Kafka Service