Why automation should focus on confidence not coverage

If your AI rollout still forces people to double-check every answer, congratulations — you’ve automated nothing. The hidden tax here is verification cost. When trust isn’t engineered in, work shifts from creation to endless auditing, burning the very hours automation was meant to save. Verification cost lurks on every P&L, unbudgeted yet brutal. We watched an industrial predictive-maintenance tool collect dust even though its forecasts were spot-on. A redesign that surfaced “why,” exposed confidence levels, and invited feedback flipped usage from ignored to indispensable. In banking, an AI advisor’s stiff tone and opaque logic drove clients away — until we added explanations and a “skip” button. Engagement jumped 60 %. Before you chase higher model accuracy, audit your verification hours per user this quarter. Where is trust debt quietly killing ROI? https://lnkd.in/e4kQCS8g #AI #Technology #Transformation #Business #Trust

"100% test automation coverage" is killing our release velocity. . . . Last sprint, I spent 3 days fixing flaky tests. We didn't find a single new bug. Meanwhile, a junior tester did 2 hours of exploratory testing and caught a critical checkout issue that would've cost us customers. Here's what we're doing wrong: We have 380 automated tests. They take 6 hours to run. And honestly? 200 of them test stuff that NEVER breaks. Example: We have 15 tests for the "About Us" page. It's a static page. It hasn't changed in 2 years. Why are we running these tests every deployment? Meanwhile, what we're NOT testing: → How the app behaves on slow networks (our users are on 3G/4G) → What happens when users do weird things (they ALWAYS do weird things) → Whether the new feature actually makes sense to users The wake-up call: Last month: Our automation passed all 380 tests ✅ We released with confidence ✅ Got 50 support tickets in 2 days ❌ The issue? Payment button worked perfectly. But users couldn't FIND it on mobile. It was hidden below the fold. Automation said: "Button works!" ✅ Users said: "Where's the button?" ❌ What I'm changing: Instead of chasing 100% coverage, I'm asking: → Does this test catch issues users actually face? → Will this test still be valuable in 3 months? → Could I test this faster manually? Now we have: 180 automated tests (down from 380) 45-minute test runs (down from 6 hours) More time for actual testing The controversial truth: More automation ≠ Better quality Sometimes, 10 minutes of exploring the app like a confused user finds more bugs than 100 automated tests. Am I wrong? Tell me in comments 👇 #TestAutomation #SDET #QA #SoftwareTesting

I automated 500 test cases in 3 months. Almost every automation engineer says this in interviews. Impressive number. But I ask one question: “If production is down and you have 10 minutes… which 5 tests do you run?” That’s when confidence drops. Recently, I spoke with 12 automation engineers. Strong profiles: • 3–5 years of Selenium or Playwright • Custom frameworks • CI/CD integration • Thousands of automated tests But when I asked: “Why did you automate THESE specific tests?” Most of them couldn’t explain it clearly. They could explain: How the framework works. How they structured Page Objects. How coverage improved. But not: Why those tests mattered. What risk they were reducing. What would actually hurt the business if it failed. One candidate showed 47 automated login tests. I asked: “If login breaks in production, what happens?” He said: “Users can’t log in.” I asked: “And what does that mean for the business?” Silence. Here’s the real issue in QA right now: We are building automation engineers who can code. But not testers who can think in risk. You can learn Selenium in weeks. You can learn Playwright quickly. AI can even generate test scripts for you now. But deciding what to test? That’s the hard part. That’s strategy. The automation engineers getting hired faster today start differently. Before writing a single test, they ask: • What’s the user impact? • What’s the business risk? • What’s the probability of failure? • If I could only run 3 tests, which ones protect revenue? Only then do they decide: Automate it. Test it manually. Or don’t test it at all. Because automation is not about quantity. It’s about protecting what matters. Companies don’t need more people who can write test scripts. They need people who can answer: “Are we testing the right things?” That’s the difference between an automation engineer… And a tester who automates with judgment. Tools will change. Frameworks will change. AI will write your test code. But testing judgment? That’s what makes you irreplaceable. Let me ask you: If production is down right now, which 3 tests would you run first?

Too many teams treat testing as a metric rather than an opportunity. A developer is told to write tests, so they do the bare minimum to hit the required coverage percentage. A function runs inside a unit test, the coverage tool marks it as covered, and the developer moves on. The percentage goes up, leadership is satisfied, and the codebase is left with the illusion of quality. But what was actually tested? Too often, the answer is: almost nothing. The logic was executed, but its behavior was never challenged. The function was called, but its failure modes were ignored. The edge cases, error handling, and real-world complexity were never explored. The opportunity to truly exercise the code and ensure it works in every scenario was completely missed. This is a systemic failure in how organizations think about testing. Instead of seeing unit, integration, and end-to-end (E2E) testing as distinct silos, they should recognize that all testing is just exercising the same code. The farther you get from the code, the harder and more expensive it becomes to test. If logic is effectively tested at the unit and integration level, it does not suddenly behave differently at the E2E level. Software is a rational system. A well-tested function does not magically start failing in production unless something external—such as infrastructure or dependencies—introduces instability. When developers treat unit and integration testing as a checkbox exercise, they push the real burden of testing downstream. Bugs that should have been caught in milliseconds by a unit test are now caught minutes or hours later in an integration test, or even days later during E2E testing. Some are not caught at all until they reach production. Organizations then spend exponentially more time and money debugging issues that should never have existed in the first place. The best engineering teams do not chase code coverage numbers. They see testing as an opportunity to build confidence in their software at the lowest possible level. They write tests that ask hard questions of the code, not just ones that execute it. They recognize that when testing is done well at the unit and integration level, their E2E tests become simpler and more reliable—not a desperate last line of defense against failures that should have been prevented. But the very best testers go even further. They recognize the system for what it truly is—a beautiful, interconnected mosaic of logic, data, and dependencies. They do not just react to failures at the UX/UI layer, desperately trying to stop an avalanche of possible combinations. They seek to understand and control the system itself, shaping it in a way that prevents those avalanches from happening in the first place. Organizations that embrace this mindset build more stable systems, ship with more confidence, and spend less time firefighting production issues. #SoftwareTesting #QualityEngineering

100% test coverage. 0% confidence. I have seen it happen. More than once. Here is how it works: A team gets told "we need higher coverage." So they write tests. Lots of tests. Tests that check if a button exists. Tests that verify a function returns something. Tests that assert true equals true. Coverage goes up. The dashboard turns green. Everyone celebrates. Then a critical bug ships to production. "But we had 100% coverage!" Yes. Coverage of lines executed. Not coverage of behavior that matters. The problem is what we measure. Line coverage tells you code was run during a test. It does not tell you the test would catch a bug. I have seen tests with zero assertions pass because they ran the code and exited. That counts as coverage. It catches nothing. What actually builds confidence: 1. Critical path coverage over total coverage Map your most important user journeys. Cover those completely. A login flow with 20 edge cases tested beats 100 trivial getters with basic checks. 2. Mutation testing Tools like Stryker or PIT change your code on purpose and check if tests fail. If your tests pass when the code is broken, they are not protecting you. 3. Bug escape tracking When a bug reaches production, ask: "Could a test have caught this?" Then write that test. Real bugs teach you what to cover. 100% coverage is a vanity metric. Confidence is the goal. What coverage metric does your team actually care about?

I see this dilemma more and more across teams today during my conversations. Many decision-makers feel confident because automated regression tests run after every deployment. On paper, everything looks under control. In reality, companies experience defect leakage, client complaints, and ongoing frustration with functional, integration and usability issues. But here is my question: How do we know the automation is actually covering what truly matters? Automation only checks what we told it to check. It does NOT question assumptions, notice new patterns, or understand how real users behave. If the original scenarios were incomplete, outdated, or based on the wrong understanding of the product, the tests will still pass and provide a false sense of safety. That’s where the danger lies. Your 100% passing rate on automated regression testing: edge cases, negative scenarios, integrations, data issues, usability problems, or real-world workflows that were never turned into scripts in the first place. Real confidence does NOT come from having automation. It comes from knowing: a. Why each test exists b. Which risks are covered and which ones we are accepting after evaluation c. What assumptions the automation is built on d. Who is continuously questioning and evolving that coverage Automation is incredibly powerful, but it needs human thinking behind it. Without that, it does NOT build confidence, it just makes us feel comfortable and it is a HUGE Risk by itself. And that comfort is NOT the same as quality. HIST Human Intelligence Software Testing guides teams toward intelligent automation, not automation done simply to look impressive without a clear purpose. Thoughts?

Test Metrics Never Lie (But They Don’t Tell the Truth Either) Thanks to Robert Fey for getting me thinking... More software testing doesn’t always mean better software. Someone, somewhere decided 90% test coverage was a great target. It’s now folklore. A nice round number - not too low to look careless, not too high to be infeasible. But like any metric that morphs into a goal, it’s become a checkbox, not a measure of confidence. Teams chase test coverage like racing dogs chasing a rabbit. We automate relentlessly. We track pass rates and mean time to detect bugs. But none of these, on their own, guarantees we’re actually testing effectively. You can have thousands of tests and still miss the one bug that matters. You can have 95% code coverage, but if the missing 5% touches payment systems, critical APIs, or complex logic - you’re celebrating when you should be panicking. Test coverage is the map, not the terrain. It shows where tests exist, not whether those tests are useful. Your map might lead you into a lake. A test that hasn't failed since 2023 may be a sign of quality, or may be dead weight - code you're maintaining long after it's lost its value. Same for automation. Teams can automate fragile tests that fail for reasons that have nothing to do with software problems. That sends people scrambling, but shouldn't increase anyone's confidence. And test metrics provide an illusion of control, but often reflect effort, not effectiveness. A 100% pass rate looks great but may hide the fact that nobody tested the failure paths, edge cases, or user journeys that break in production. Percent complete only has meaning if you measure everything that matters. Here’s my theory: Most bugs that matter don’t come from logic errors in isolated functions. They come from misunderstanding the problem, misusing the system, or miscommunicating expectations. No tests will catch a feature nobody wanted in the first place. So how do we test better? Treat tests as investments, not artifacts. Question their ongoing value. Pay attention to test quality, not just quantity. Ask what each test is guarding against - and whether that risk still matters. Remember, there's an ever-expanding cost to maintaining an ever-expanding test library. Every test should have an ROI. Use test coverage to assess what you’ve missed, not celebrate what you’ve covered. Treat unreliable tests as problems to eliminate, not nuisances to ignore. Pair automated tests with exploratory testing to challenge your assumptions. Trace production defects back to gaps in your testing. That’s where the real insights are. And maybe (just maybe) instead of bragging about coverage ratios, ask, "What are we not testing - and why are we not testing that?" Effective testing isn’t about checking boxes. It’s about asking better questions, uncovering blind spots, and continuously refining what confidence looks like. And I'll add that all pre-prod testing is just a precursor to the real tests.

AI support is growing 40% a year. But only 1 in 4 vendors show you if the answer is even trustworthy. That’s the disconnect. According to Gartner: - GenAI-enabled customer support is growing at ~40% CAGR - But <25% of vendors offer confidence scores or transparency in decisions - And 30%+ of rollouts are delayed due to trust, data, and governance gaps Meanwhile, in production: A cancer patient’s claim was denied. The AI was only 52% confident. No one saw the risk. The decision went through. This isn’t a hypothetical. It’s what happens when AI is deployed in regulated CX environments without oversight. Everyone’s excited about GenAI. But most are automating without a safety net. At Zingtree, we built a framework to fix that. We score every LLM output in real-time and route based on confidence. Here’s how we evaluate AI decisions in high-stakes workflows: - 95–100% confidence → Let automation proceed - 85–94% → Route to human agent - 70–84% → Ask the customer or agent for clarification - 50–69% → Trigger fallback (policy-based logic, form flow, etc.) - Below 50% → Escalate immediately and flag Not just “smart automation.” Controlled automation. That’s the difference between scale and exposure. If your AI can’t measure its own uncertainty, it shouldn’t be making decisions. Curious what factors we analyze before delivering any AI response? #DeterministicAI #ConfidenceScore #CustomerSupport

Automation validates what you tell it to validate. If you haven't written down what the system must not get wrong, the AI has nothing real to work from. It generates coverage, not confidence. The missing piece is the intent layer: the explicit list of what your system must never get wrong. Pricing logic must never round to $0. Authorization rules must never let user A see user B’s data. Checkout flows must handle currency exactly the same way as the billing system. These aren't edge cases. They're the constraints that define whether the software works. Most teams haven't written them down. Not because they don’t know them, but because documenting invariants feels slower than writing tests. So they skip it. The test suite grows, the real risks stay undocumented, and the gap never closes. Before your next AI-assisted test run: what invariants did your team commit to paper this month?

Reliability, evaluation, and “hallucination anxiety” are where most AI programmes quietly stall. Not because the model is weak. Because the system around it is not built to scale trust. When companies move beyond demos, three hard questions appear: →Can we rely on this output? →Do we know what “good” actually looks like? →How much human oversight is enough? The fix is not better prompting. It is a strategy and operating discipline. 𝐅𝐢𝐫𝐬𝐭: ⁣Define reliability like a product, not a vibe. Every serious AI use case should have a one-page SLO sheet with measurable targets across: →Task success ↳Right-first-time rate and rubric-based acceptance →Factual grounding ↳Evidence coverage and unsupported-claim tracking →Safety and compliance ↳Policy violations and PII leakage →Operational quality ↳Latency, cost per task, escalation to humans Now “good” is no longer opinion. It is observable. 𝐒𝐞𝐜𝐨𝐧𝐝:  evaluation must be continuous, not a one-off demo test. Use a simple loop: 𝐏lan: Define rubrics, datasets, and risk tiers 𝐃⁣o: Run offline evaluations and limited pilots 𝐂heck: Monitor drift and regressions weekly 𝐀ct: Update prompts, data, guardrails, and workflows Support this with an AI test pyramid: →Unit checks for prompts and tool behaviour →Scenario tests for real edge failures →Regression benchmarks to prevent backsliding →Live monitoring in production Add statistical control charts, and you can detect silent degradation before users do. 𝐓𝐡𝐢𝐫𝐝: reduce hallucinations by design. →Run a short failure-mode workshop and engineer controls: →Require retrieval or evidence before answering →Allow safe abstention instead of confident guessing →Add claim checking and tool validation →Use structured intake and clarifying flows You are not asking the model to behave. You are designing a system that expects failure and contains it. 𝐅𝐨𝐮𝐫𝐭𝐡: make human-in-the-loop affordable. Tier risk: →Low risk: Light sampling →Medium risk: Triggered review →High risk: Mandatory approval Escalate only when signals demand it: low confidence, missing evidence, policy flags, or novelty spikes. Review becomes targeted, fast, and a source of improvement data. 𝐅𝐢𝐧𝐚𝐥𝐥𝐲: Operate it like a capability. Track outcomes, risk, delivery speed, and cost on a single dashboard. Hold a short weekly reliability stand-up focused on regressions, failure modes, and ownership. What you end up with is simple: ↳Use case catalogue with risk tiers ↳Clear SLOs and error budgets ↳Continuous evaluation harness ↳Built-in controls ↳Targeted human review ↳Reliability cadence AI does not scale on intelligence alone. It scales on measurable trust. ♻️ Share if you found thisuseful. ➕ Follow (Jyothish Nair) for reflections on AI, change, and human-centred AI #AI #AIReliability #TrustAtScale #OperationalExcellence