Arize AI

Enterprise AI succeeds when teams can turn a promising demo into a reliable production system that delivers measurable business value. At Arize Observe, CVS Health shared how evaluation, observability, governance, and reusable engineering practices help organizations move beyond pilot purgatory and scale AI responsibly. Watch the full session: https://lnkd.in/gP7hfscQ

A disappointing eval score rarely tells you what to fix. It can mean the model failed, the product never collected enough context, success wasn’t defined yet, or reviewers are applying a standard that drifted since last week. In part 2 of our Rise of the Agent Engineer series, Hamel Husain from Parlance Labs walks through why useful evals start before the metric: - Ambiguous inputs force the model to guess, then the score blames the LLM. - Criteria drift is normal for generative products (“you don’t really know what you want until you see it in action”) - Broad metrics miss product-specific failures - Domain experts need a review surface that shows the full trajectory, not a spreadsheet of I/O pairs The practical takeaway for agent teams: treat eval criteria as versioned product artifacts, open the traces, and get the expert into error analysis as fast as possible. Read the full writeup here: https://lnkd.in/gYCr8zit

What if your agents got better every time they failed? Today, we’re launching Signal. It continuously reviews production traces, finds issues, and turns them into an investigation with evidence, root cause, and a proposed fix. Your engineers decide what ships. Signal gets them to the decision faster. https://lnkd.in/gBUBuAps

The cheapest model call can become expensive when the task fails. Per-token pricing captures inference cost while leaving retries, timeouts, tool failures, and confidently wrong answers outside the headline number. Arize and Fireworks AI ran K3, GPT-5.5, and 8 other models through 40 real Terminal-Bench tasks, with six trials for every task-model pair. We traced all 2,400 runs and graded each one with the task’s deterministic test suite. During a live research deep dive, we will unpack: • How model performance shifted across easy, medium, and hard tasks • How cost per successful task changed the model ranking • When routing or escalation improved cost and coverage • How traces exposed token churn, repeated tool failures, budget exhaustion, and silent failures You will leave with a repeatable method for benchmarking models on your own workloads and designing a routing policy around your definition of success. Join us on August 6, 11am PT/2pm ET: https://lnkd.in/gNkwbzpj

Before telling Anthropic's research team that a new model was nine points better, Marius Buleandra did probably the least scalable (and most important) thing in agent evaluation: He read the transcripts. The model had started adding `LIMIT` clauses to its SQL queries. Sensible behavior in production, but inside this eval, it let the model route around a defect in the harness. That's the trap with agent evals. The unit under test is not just the model. It is the model, prompt, tools, state, environment, grader, and whichever path the run happened to find. A useful eval system needs to let you move from: aggregate score to changed cases to trajectory divergence to exact tool call or grader decision. Otherwise, you know performance moved, but not what to fix. A score tells you that something changed, but the transcript tells you whether it was progress. Read more from Anthropic here: https://lnkd.in/gXhAwvRN

Want to master the full workflow of shipping reliable AI agents? Laurie's workshop from AI Engineer World's Fair, "Evaluating and Shipping AI Agents That Work," is now a free, self-paced course on Arize University. Earn a certificate you can share on LinkedIn by completing 13 episodes that cover: → Traces → Code evals vs. LLM judges → Datasets and experiments → Online evals and monitors → Closing the feedback loop with coding agents Get started here: https://lnkd.in/e_zTvffX

Two AI observability lessons from Booking.com: - An agent latency spike came from a model running without the appropriate service tier. - Multi-turn eval scores fell because long URLs were being added back into the conversation history, causing the context to balloon. Both problems became much easier to fix once the team could connect production signals to individual traces, configurations, and conversations. Here's a practical look at how Booking.com approaches AI observability across agents and traditional ML with Arize AX: https://lnkd.in/eRPHmGCm

Catch our Head of DevRel Laurie Voss at the first SF Bay Area Multi-Agent Systems meetup this Thursday, July 30th, hosted by BAND and Drata. The panel will break down how agents communicate, coordinate, and work as teams in production. Grab your spot 👇