Datorios Announces New Apache Flink Observability Capabilities for Responsible Agentic AI

Correlated traces and data lineage make Flink-powered autonomous AI agents safer and more reliable

BERLIN, Oct. 23, 2024 /PRNewswire/ -- Datorios, the leading developer of data observability and data quality technology for real-time business operations, has announced the general release of new observability innovations for the open source Apache Flink stream processing system. Unveiled at Flink Forward 2024 underway in Berlin Germany, the new technology from Datorios equips businesses with the observability required to ensure that Flink-powered real-time AI applications drive accurate, rational business automation on a continuous basis.

Agentic AI is the next step in the evolution of AI applications. Agentic AI goes beyond generating answers to questions; it uses AI to create and execute plans to achieve specific goals. It automates decision making and operations based on those decisions. Gartner predicts that by 2028, 15% of day-to-day work decisions will be made autonomously via agentic AI.

Responsible Real-time AI Requires Apache Flink Observability
Apache Flink is a critically important part of the real-time agentic AI infrastructure. It is used to combine and shape data from across the enterprise, giving AI agents the data and understanding they need to take action. With businesses becoming increasingly dependent upon AI systems, it is essential to be able to observe the inner workings of Flink programs in order to understand what decisions agentic AI systems take.

"No responsible business would operate without cybersecurity in place today, and likewise, no responsible business should let AI drive operations without observability in place," said Ronen Korman, CEO and co-founder of Datorios. "Agentic AI holds tremendous positive potential for business, but you absolutely need visibility into how AI systems are automating your operations. Datorios gives you the visibility for the needed operational quality no business can leave aside."

New Correlated Traces and Data Lineage Analytics for Apache Flink
As real-time data streams into and out of AI models such as LLMs, it is processed by Apache Flink programs. From time to time, it becomes necessary to observe how these Flink programs are behaving–in response to a system crash, or a customer complaint, or a regulatory audit request. The new observability capabilities from Datorios provide this insight in interactive dashboards purpose-built for real-time stream data processing in Flink. This enables businesses to quickly pinpoint the root cause of an issue and resolve it.

New functionality includes:

Correlated traces - the cause of an AI system failure can nearly always be attributed to a problem with the data, or the Flink code that acts on the data, or the computing infrastructure on which Flink runs. Datorios provides a unified platform that correlates data traces across data, code, and infrastructure, allowing Flink teams to detect, isolate, and resolve issues quickly. With Datorios, operational complexities are unraveled, empowering Flink users to maintain the operational continuity they need to succeed.

Lineage analytics help Flink teams trace the journey of data within Flink jobs, which makes it clear when and why data changed, and who on the team is most familiar with the offending code or data source so that the issue can be fixed as quickly as possible. Lineage analytics capabilities include:

• Lineage visualization - View interactive Job Graphs down to the single operator within a task.
• Data flow detail - Drill down into Job Graphs to explore detailed views of downstream data flow and transformations.
• Data validation - Perform upstream investigation to validate and trace back breaks of your data and processing logics.
• Low performance overhead - Unlike other lineage tools, Linear Analyzer runs external to you Flink jobs, which gives you the visibility you need without affecting your real-time system performance.

To learn more, visit the Datorios blog or connect with community members on the company's Slack channel.

About Datorios
Rooted in the collective wisdom of the Apache Flink community and addressing the unique challenges faced by enterprises, Datorios is the first data observability platform for Apache Flink, empowering data engineers with the ability to supercharge real-time business innovation and make real-time data processing the organizational norm instead of the exception. Datorios is a remote-first employer with headquarters in Tel Aviv, Israel and Palo Alto, CA.

SOURCE Datorios