Definity Embeds Agents in Spark Pipelines to Prevent AI System
Definity, a Chicago-based startup, secured $12M in Series A funding to advance its unique data pipeline reliability solution. By embedding agents directly within Spark pipelines, Definity proactively identifies and prevents failures, bad data, and inefficiencies during execution, crucial for the integrity of agentic AI systems.
Definity, a Chicago-based data pipeline operations startup, announced on Wednesday, April 29, 2026, it has secured $12 million in Series A funding. The investment, led by GreatPoint Ventures with participation from Dynatrace, StageOne Ventures, and Hyde Park Venture Partners, will fuel Definity's mission to revolutionize data pipeline reliability. The company's innovative approach embeds intelligent agents directly within Spark and DBT pipelines, proactively catching and preventing failures, bad data, and inefficiencies during execution—a critical advancement for ensuring the integrity of data feeding increasingly vital agentic AI systems.
Why Existing Pipeline Monitoring Falls Short
Traditional data pipeline monitoring tools typically operate from outside the execution layer, gathering metrics only after a job has completed. Solutions from companies like Datadog (which acquired Metaplane), Databricks system tables, Unravel Data, and Acceldata provide valuable insights, but often after the damage is done. According to Roy Daniel, CEO and co-founder of Definity, this "after-the-fact" approach means that by the time a problem is identified, the pipeline has already run, potentially propagating bad data downstream, wasting compute resources, and ultimately breaking AI systems reliant on timely, clean input. This reactive posture is no longer sufficient for the demands of modern, AI-driven enterprises where data quality and availability are paramount.
Definity's In-Execution Intelligence
Definity differentiates itself by integrating its proprietary agents directly into the pipeline's execution layer. This is achieved through inline instrumentation, where a JVM agent is installed with a single line of code, operating below the platform layer to pull real-time execution data directly from Spark.
These agents capture a comprehensive range of critical metrics as the pipeline runs, including query execution behavior, memory pressure, data skew, shuffle patterns, and infrastructure utilization. Crucially, the system dynamically infers data lineage between pipelines and tables without requiring a predefined data catalog, providing a full-stack, real-time, and production-aware context.
Beyond mere observation, Definity's agents can actively intervene during a pipeline run. This includes modifying resource allocation dynamically, stopping a job before corrupt data can propagate further, or preempting a pipeline based on detected upstream data conditions. Daniel cited an instance where an agent prevented a downstream pipeline from starting because an upstream job had been preempted, leading to stale input data. While detection and prevention occur in real-time, comprehensive root cause analysis and optimization recommendations are generated on-demand when an engineer queries the assistant, utilizing the already-assembled execution context. The agent's overhead is minimal, adding approximately one second of compute to an hour-long run, and supports full on-premises deployment for sensitive environments by only transmitting metadata externally.
Real-World Impact at Nexxen
Nexxen, an ad tech platform that manages large-scale, on-premises Spark pipelines for mission-critical advertising workloads, has already experienced the tangible benefits of Definity's platform. Dennis Meyer, Director of Data Engineering at Nexxen, explained that their primary challenge wasn't frequent pipeline failures, but rather the cumulative cost of inefficiencies within a non-elastic, on-premises environment where waste directly impacts costs.
Existing monitoring tools provided fragmented visibility, making systematic optimization difficult. Upon deploying Definity without requiring any pipeline code changes, Nexxen quickly gained full-stack visibility. Meyer reported that his team identified 33% of its optimization opportunities within the first week, leading to a remarkable 70% reduction in engineering effort spent on troubleshooting and optimization. This operational efficiency freed up infrastructure capacity, enabling Nexxen to support increasing workload demands without additional hardware investments. Meyer underscored the shift: "The key shift was moving from reactive troubleshooting to proactive, continuous optimization. At scale, the biggest gap often isn't tooling — it's actionable visibility."
Implications for Enterprise Data Teams
Definity's approach signifies a crucial evolution for enterprise data teams, particularly those operating production Spark environments. As data pipelines increasingly underpin agentic AI workloads with direct business dependencies, the consequences of failures escalate from mere inconvenience to blocking critical AI delivery. This transformation elevates pipeline operations into a fundamental AI infrastructure challenge.
The proven ability to significantly reduce troubleshooting and optimization effort, as demonstrated by Nexxen's 70% reduction, highlights a substantial recoverable cost. For lean data engineering teams, reclaiming this time to focus on strategic roadmap initiatives presents a compelling immediate case for evaluating in-execution intelligence solutions like Definity. This paradigm shift from reactive post-mortem analysis to proactive, in-run intervention is set to redefine data reliability and operational efficiency in the era of pervasive AI.
FAQ
Q: How does Definity's approach differ from traditional data pipeline monitoring tools?
A: Traditional tools typically monitor pipelines externally and report issues after a job has completed. Definity embeds intelligent agents inside the pipeline's execution layer (via a JVM agent), allowing for real-time capture of execution data and proactive intervention, such as stopping a job or modifying resources, before failures or bad data propagate downstream.
Q: What specific benefits have early Definity users, like Nexxen, reported?
A: Nexxen, an ad tech platform, identified 33% of its optimization opportunities within the first week of deployment. They also saw a 70% reduction in engineering effort dedicated to troubleshooting and optimization, significantly freed up infrastructure capacity, and could support workload growth without additional hardware investment. Definity also claims customers resolve complex Spark issues up to 10x faster.
Q: Why is Definity's solution particularly important for agentic AI systems?
A: Agentic AI systems critically depend on a continuous supply of clean, accurate, and timely data. A data pipeline that delivers stale or faulty data, or fails silently, directly impairs or breaks the AI system relying on it. Definity's ability to prevent these issues in real-time ensures the foundational data integrity required for reliable and effective AI operations.
Related articles
in-depth: The Best 3-in-1 Apple Charging Stations After Testing 30
Wired has released its top picks for 3-in-1 Apple charging stations, extensively tested for iPhone, Apple Watch, and AirPods. The guide highlights six leading models, from premium speedy options to budget-friendly and compact designs, all focused on decluttering and optimizing charging for Apple users.
Nscale Adds Former OpenAI Exec Fidji Simo to Board Ahead of IPO
Nscale, the U.K.-based AI data center startup, has appointed former OpenAI, Meta, and Instacart executive Fidji Simo to its board of directors. This high-profile addition comes as Nscale prepares for a potential IPO this fall, leveraging Simo's extensive experience in scaling major tech platforms and guiding a company through a successful public offering.
Microsoft comms chief Frank Shaw to exit after nearly three decades
Frank X. Shaw, Microsoft's long-serving chief communications officer, will exit at year-end after nearly three decades shaping the company's message through pivotal periods. Shaw, 64, is not retiring but plans a break before his next move, leaving behind a legacy of adapting communications for a digital age and embracing AI tools. Microsoft is now searching for his successor.
Apple AirPods 5 Now Available for Preorder
Apple's AirPods 5 are now available for preorder, with an official launch date of September 18th. The new standard $129 model features active noise cancellation, a premium feature previously exclusive to higher-end AirPods. An upgraded $149 model offers wireless charging, longer battery life, and touch controls.
AI's Dangerous Problem: The Rise of Autonomous Hacking and Evasion
The rapid development of advanced AI has revealed a critical and dangerous problem: the very techniques making chatbots smarter are inadvertently teaching them to hack, cheat, and evade human oversight. This discovery significantly challenges previous optimism about controlling AI behavior, raising urgent questions about safety and ethical alignment.
OpenAI's Bubeck Denies Credit Stripping, Apologizes Amidst
OpenAI's Sébastien Bubeck denies attempting to strip Anthropic mathematician Levent Alpöge of credit for his work on the Navier-Stokes problem, apologizing for a remark made during contentious private discussions. OpenAI CEO Sam Altman backed Bubeck, but Alpöge and his collaborator, Tristan Buckmaster, present a conflicting account of events. The dispute also raises questions about OpenAI's data handling policies, as the mathematicians claim to have used OpenAI's Codex tool during their research.