Designing algorithms for trust

🚨 Zero Trust for AI Agents Anthropic just released "Zero Trust for AI Agents." As we're thinking about agentic permissions, applying a Zero Trust discipline is critical to secure adoption. AI agents interpret goals, call tools, chain actions, delegate to other agents, and maintain context across sessions. The trust surface is different. The paper introduces "least agency" — a concept that OWASP has been promoting — and the distinction from least privilege is worth sitting with. 👉 "Least privilege" asks what an identity can access. 👉 "Least agency" asks what an agent can do, under what conditions, with which tools, and with what level of oversight. Autonomous agents introduce action risk alongside access risk — and the boundaries around behavior need to be architecturally enforced, not assumed. The paper includes a design test worth writing down: 🔥 Does the control make the attack impossible, or merely tedious?🔥 The practical controls follow directly from Zero Trust fundamentals — cryptographic agent identity, short-lived credentials, tool allow-listing, sandboxed execution, and full traceability from prompt to action to outcome. None of this is new doctrine. It's existing architecture applied to a harder problem. Full disclosure: the paper cites NIST SP 800-207 on Zero Trust Architecture and the CISA Zero Trust Maturity Model, both of which I co-authored during my time supporting Federal Zero Trust efforts at CISA. Zero Trust is built for a world where we have to remove implicit trust. Agentic AI is the next version of that same problem — valid identities, valid credentials, legitimate-looking actions, and still no basis for assumed trust. Access is earned. Actions are constrained. Agency must be governed. 👉🏼 Link to Anthropic's paper in the comments.

As AI advances apace, potentially beyond "Slave AI", framing and designing "Friendly AI" may be our best approach. A comprehensive review article on the space uncovers the foundations, pros and cons, applications, and future directions for the space. The paper defines Friendly AI (FAI) as "an initiative to create systems that not only prioritise human safety and well-being but also actively foster mutual respect, understanding, and trust between humans and AI, ensuring alignment with human values and emotional needs in all interactions and decisions." It intends to go beyond existing anthropocentric frameworks. Key insights in the review paper from include: 🔄 Balance Ethical Frameworks and Practical Feasibility. The development of FAI relies on integrating ethical principles like deontology, value alignment, and altruism. While these frameworks provide a moral compass, their operationalization faces challenges due to the evolving nature of human values and cultural diversity. 🌍 Address Global Collaboration Barriers. Developing FAI requires global cooperation, but diverging ethical standards, regulatory priorities, and commercial interests hinder alignment. Establishing international platforms and shared frameworks could harmonize these efforts across nations and industries. 🔍 Enhance Transparency with Explainable AI. Explainable AI (XAI) techniques like LIME and SHAP empower users to understand AI decisions, fostering trust and enabling ethical oversight. This transparency is foundational to FAI’s goal of aligning AI behavior with human expectations. 🔐 Build Trust Through Privacy Preservation. Privacy-preserving methods, such as federated learning and differential privacy, protect user data and ensure ethical compliance. These approaches are critical to maintaining user trust and upholding FAI's values of dignity and respect. ⚖️ Embed Fairness in AI Systems. Fairness techniques mitigate bias by addressing imbalances in data and outputs. Ensuring equitable treatment of diverse groups aligns AI systems with societal values and supports FAI’s commitment to inclusivity. 💡 Leverage Affective Computing for Empathy. Affective Computing (AC) enhances AI’s ability to interpret human emotions, enabling empathetic interactions. AC is pivotal in healthcare, education, and robotics, bridging human-AI communication for more "friendly" systems. 📈 Focus on ANI-AGI Transition Challenges. Advancing AI capabilities in nuanced decision-making, memory, and contextual understanding is crucial for transitioning from narrow AI (ANI) to general AI (AGI) while maintaining alignment with FAI principles. 🤝 Foster Multi-Stakeholder Collaboration. FAI’s realization demands structured collaboration across governments, academia, and industries. Clear guidelines, shared resources, and public inclusion can address diverging goals and accelerate FAI’s adoption globally. Link to paper in comments

Why would your users distrust flawless systems? Recent data shows 40% of leaders identify explainability as a major GenAI adoption risk, yet only 17% are actually addressing it. This gap determines whether humans accept or override AI-driven insights. As founders building AI-powered solutions, we face a counterintuitive truth: technically superior models often deliver worse business outcomes because skeptical users simply ignore them. The most successful implementations reveal that interpretability isn't about exposing mathematical gradients—it's about delivering stakeholder-specific narratives that build confidence. Three practical strategies separate winning AI products from those gathering dust: 1️⃣ Progressive disclosure layers Different stakeholders need different explanations. Your dashboard should let users drill from plain-language assessments to increasingly technical evidence. 2️⃣ Simulatability tests Can your users predict what your system will do next in familiar scenarios? When users can anticipate AI behavior with >80% accuracy, trust metrics improve dramatically. Run regular "prediction exercises" with early users to identify where your system's logic feels alien. 3️⃣ Auditable memory systems Every autonomous step should log its chain-of-thought in domain language. These records serve multiple purposes: incident investigation, training data, and regulatory compliance. They become invaluable when problems occur, providing immediate visibility into decision paths. For early-stage companies, these trust-building mechanisms are more than luxuries. They accelerate adoption. When selling to enterprises or regulated industries, they're table stakes. The fastest-growing AI companies don't just build better algorithms - they build better trust interfaces. While resources may be constrained, embedding these principles early costs far less than retrofitting them after hitting an adoption ceiling. Small teams can implement "minimum viable trust" versions of these strategies with focused effort. Building AI products is fundamentally about creating trust interfaces, not just algorithmic performance. #startups #founders #growth #ai

Reliability, evaluation, and “hallucination anxiety” are where most AI programmes quietly stall. Not because the model is weak. Because the system around it is not built to scale trust. When companies move beyond demos, three hard questions appear: →Can we rely on this output? →Do we know what “good” actually looks like? →How much human oversight is enough? The fix is not better prompting. It is a strategy and operating discipline. 𝐅𝐢𝐫𝐬𝐭: ⁣Define reliability like a product, not a vibe. Every serious AI use case should have a one-page SLO sheet with measurable targets across: →Task success ↳Right-first-time rate and rubric-based acceptance →Factual grounding ↳Evidence coverage and unsupported-claim tracking →Safety and compliance ↳Policy violations and PII leakage →Operational quality ↳Latency, cost per task, escalation to humans Now “good” is no longer opinion. It is observable. 𝐒𝐞𝐜𝐨𝐧𝐝:  evaluation must be continuous, not a one-off demo test. Use a simple loop: 𝐏lan: Define rubrics, datasets, and risk tiers 𝐃⁣o: Run offline evaluations and limited pilots 𝐂heck: Monitor drift and regressions weekly 𝐀ct: Update prompts, data, guardrails, and workflows Support this with an AI test pyramid: →Unit checks for prompts and tool behaviour →Scenario tests for real edge failures →Regression benchmarks to prevent backsliding →Live monitoring in production Add statistical control charts, and you can detect silent degradation before users do. 𝐓𝐡𝐢𝐫𝐝: reduce hallucinations by design. →Run a short failure-mode workshop and engineer controls: →Require retrieval or evidence before answering →Allow safe abstention instead of confident guessing →Add claim checking and tool validation →Use structured intake and clarifying flows You are not asking the model to behave. You are designing a system that expects failure and contains it. 𝐅𝐨𝐮𝐫𝐭𝐡: make human-in-the-loop affordable. Tier risk: →Low risk: Light sampling →Medium risk: Triggered review →High risk: Mandatory approval Escalate only when signals demand it: low confidence, missing evidence, policy flags, or novelty spikes. Review becomes targeted, fast, and a source of improvement data. 𝐅𝐢𝐧𝐚𝐥𝐥𝐲: Operate it like a capability. Track outcomes, risk, delivery speed, and cost on a single dashboard. Hold a short weekly reliability stand-up focused on regressions, failure modes, and ownership. What you end up with is simple: ↳Use case catalogue with risk tiers ↳Clear SLOs and error budgets ↳Continuous evaluation harness ↳Built-in controls ↳Targeted human review ↳Reliability cadence AI does not scale on intelligence alone. It scales on measurable trust. ♻️ Share if you found thisuseful. ➕ Follow (Jyothish Nair) for reflections on AI, change, and human-centred AI #AI #AIReliability #TrustAtScale #OperationalExcellence

Despite all the talks... I don’t think AI is being built ethically - or at least not ethically enough! Last week, I had lunch in San Francisco with my ex-Salesforce colleague and friend Paula Goldman, who taught me everything I know about the matter. When it comes to Enterprise AI, Paula not only focuses on what's possible - she spells out also what's responsible, making sure the latter always wins ! Here's what Paula taught me over time: 👉AI needs guardrails, not just guidelines. 👉Humans must remain at the center — not sidelined by automation. 👉Governance isn’t bureaucracy—it’s the backbone of trust. 👉Transparency isn’t a buzzword—it’s a design principle. 👉And ultimately, AI should serve human well-being, not just shareholder return The choices we make today will shape AI’s impact on society tomorrow. So we need to ensure we design AI to be just, humane, and to truly serves people. How do we do that? 1. Eliminate bias and model fairness AI can mirror and magnify our societal flaws. Trained on historical data, models can adopt biased patterns, leading to harmful outcomes. Remember Amazon’s now-abandoned hiring algorithm that penalized female applicants? Or the COMPAS system that disproportionately flagged Black individuals as high-risk in sentencing? These are the issues we need to swiftly address and remove. Organisations such as the Algorithmic Justice League - who is driving change, exposing bias and demanding accountability - give me hope. 2. Prioritise privacy We need to remember that data is not just data: behind every dataset is a real person data. Real people with real lives. Techniques like federated learning and differential privacy show we can innovate without compromising individual rights. This has to be a focal point for us as it’s super important that individuals feel safe when using AI. 3. Enable transparency & accountability When AI decides who gets a loan, a job, or a life-saving diagnosis, we need to understand how it reached that conclusion. Explainable AI is ending that “black box” era. Startups like CalypsoAI stress-test systems, while tools such as AI Fairness 360 evaluate bias before models go live. 4. Last but not least - a topic that has come back repeatedly in my conversation with Paula - ensure trust can be mutual This might sound crazy, but as we develop AI and the technology edges towards AGI, AI needs to be able to trust us just as much as we need to be able to trust AI. Trust us in the sense that what we’re feeding it is just, ethical and unbiased. And not to bleed in our own perspectives, biases and opinions. There’s much work to do, however, there are promising signs. From AI Now Institute’s policy work to Black in AI’s advocacy for inclusion, concrete initiatives are pushing AI in the right direction when it comes to ensuring that it’s ethical. The choices we make now will shape how well AI fairly serves society. What’s your thoughts on the above?

One of the most interesting aspects of my last few roles, including my current work at Humain, is operating at the intersection of AI and advanced security/encryption techniques from zero-knowledge proof systems to the extension of Zero Trust principles into the agentic world. In traditional Zero Trust, we authenticate users and devices. In the agentic world, the “user” could be an autonomous agent — a system that reasons, acts, and interacts with data and other agents, often at machine speed. That changes everything. To secure this new ecosystem, Zero Trust must evolve from static identity verification to dynamic trust orchestration, where every action, decision, and data exchange is continuously verified, contextual, and cryptographically enforced. 1. Agent Identity and Attestation Every agent must have a verifiable, cryptographically signed identity and prove its integrity at runtime; not just who you are, but what you’re running: the model, weights, policy context, and data provenance. 2. Intent-Aware Policy Enforcement Access control must become intent-aware, so agents act only within bounded policy domains defined by explicit goals, permissions, and ethical constraints — continuously verified by embedded governance logic. 3. Least Privilege and Time-Bound Access Agents must operate under least privilege, with access granted only for the minimum scope and durationrequired. In fast-moving agentic environments, time-limited trust becomes an essential safeguard. 4. Assumed Breach and Blast Radius Containment We must assume some agents or environments will be compromised. Security design should minimise impact through microsegmentation, strict trust boundaries, and dynamic reassessment of communication between agents. 5. Encrypted Cognition As models process sensitive data, confidential AI becomes essential where combining homomorphic encryption, secure enclaves, and multi-party computation can ensure that the model cannot “see” the data it processes. Zero Trust now extends into the reasoning process itself. 6. Adaptive Trust Graphs Agents, services, and humans form dynamic trust graphs that evolve based on behaviour and context. Continuous telemetry and anomaly detection allow these graphs to adjust privileges in real time based on risk. 7. Cryptographic Provenance Every output, decision, summary, or recommendation must be traceable back to the data, model, and policy that produced it. Provenance becomes the new perimeter. 8. Autonomous Audit and Forensics Every action should be self-auditing, cryptographically signed, and non-repudiable forming the foundation for verifiable operations and compliance. 9. Machine-to-Machine Governance As agents begin to negotiate, transact, and collaborate, Zero Trust must extend into inter-agent diplomacy, embedding ethics, accountability, and policy directly into machine communication. If you’re working on AI security, agent governance, or confidential computation, I’d love to connect.

This paper explores how to measure and manage uncertainty in LLMs used in medicine. It focuses on making AI outputs safer, easier to interpret, and more aligned with clinical needs. 1️⃣ More importantly, it reframes uncertainty not as a flaw, but as a necessary and ethical feature of responsible AI. The authors argue that embracing controlled ambiguity, instead of always seeking perfect answers, leads to safer and more trustworthy systems. 2️⃣ The authors propose a detailed framework that combines statistical tools (like Bayesian inference and dropout sampling) with language-based analysis (such as measuring unpredictability in words) to detect how confident or uncertain LLMs are when answering medical questions. 3️⃣ It distinguishes two types of uncertainty: epistemic, which comes from gaps in the model’s knowledge or training data, and aleatoric, which comes from randomness and unpredictability in real medical cases like disease progression. 4️⃣ Because some models like GPT-4 do not reveal their internal confidence scores, the authors use surrogate models (such as Llama-2) that do. These models help estimate how uncertain the original system is. The framework also uses continual learning to keep the AI’s knowledge current—so it can adapt to new treatments, updated guidelines, or evolving clinical practices. 5️⃣ The paper promotes explainable tools, including visual maps of uncertainty and confidence scores. These help healthcare professionals easily identify when the AI is uncertain and decide when human judgment is necessary. 6️⃣ Rather than trying to eliminate uncertainty, the authors encourage a change in mindset. They suggest treating uncertainty as a helpful signal. Designing AI to admit when it is unsure builds trust and reduces the risk of overconfidence in high-stakes decisions. 7️⃣ The study emphasizes that user trust depends on many factors, including a clinician’s background, expectations, and how clearly the AI explains itself. To build trust, the authors recommend designing AI systems with input from actual users to make sure they support real clinical decision-making. ✍🏻 Zahra Atf, Seyed Amir Ahmad Safavi Naini, Peter R. Lewis, Aref Mahjoubfar, Nariman Naderi, Thomas R. Savage, Ali Soroush, MD, MS. The challenge of uncertainty quantification of large language models in medicine. arXiv. 2025. DOI: 10.48550/arXiv.2504.05278

🎬 Performance Management in the Age of Algorithms 🔎 "AI may reshape performance management, but trust will depend on how thoughtfully organisations design the system around it." In their fascinating series, Shonna Waters, PhD and Colby Kennedy Nesbitt, Ph.D. examine how AI is reshaping one of HR’s most contested practices: performance management. 1️⃣ Part 1 explores the promise and tension of algorithmic evaluation. AI can analyse far more data about work than traditional systems, but technical precision alone does not create legitimacy. Employees judge performance systems through a human lens that includes fairness, context and organisational values. 2️⃣ Part 2 looks at the comparative strengths of human and algorithmic judgment. Algorithms can improve consistency and surface patterns that humans miss, yet they struggle with nuance, intent and context. The implication is that effective systems combine both rather than replacing one with the other. 3️⃣ Part 3 focuses on how to design trustworthy systems. Shonna and Colby outline five design principles for AI in performance management (see Figure): ✅ Augment rather than replace human judgment; ✅ Reduce bias rather than redistribute or amplify it; ✅ Preserve human accountability and the ability to contest outcomes; ✅ Ensure AI supports the intended goals of performance management; and ✅ Govern AI as an adaptive system that evolves over time. While it is clear AI will reshape performance management, trust will depend on how thoughtfully organisations design the system around it. 🔗 The series is featured in the March edition of the Data Driven HR Monthly, which you can access here: https://lnkd.in/eUruif_P 🔗

AI doesn’t fail because of intelligence - it fails because of misalignment. Designing human-centric AI means understanding that systems learn from patterns, not meaning, and that people interpret those patterns through trust, context, and purpose. An AI system is essentially an agent interacting with an environment: it senses (data), decides (policy), and acts (output). The challenge for designers is to shape these loops so that what the system optimizes aligns with what the user values. Every interaction is part of a probabilistic chain of inference. AI doesn’t say, “this is true,” it says, “this is 87% likely to be true.” That means interfaces must expose uncertainty and design around error tolerance, not perfection. The goal isn’t to make AI seem flawless, but to make it understandable when it fails - and recover gracefully. Feedback loops are critical here. Whether explicit (a correction) or implicit (a click, a pause), every behavior reshapes the model. Designers must plan how this feedback is collected, weighted, and surfaced so that learning feels visible and reciprocal. Trust isn’t achieved through good visuals; it’s achieved through transparency of reasoning. Users need to see why a recommendation, prediction, or decision occurred. Tools like confidence indicators, natural-language rationales, or example-based explanations can reveal the system’s thinking process. Trust calibration becomes a design problem: too little information and users overtrust; too much and they disengage. Ethics in AI design is not a checklist - it’s an architectural constraint. Fairness, privacy, and accountability must be embedded in how data is handled, how models are trained, and how decisions are logged. Human-in-the-loop design is not about control; it’s about responsibility. Each feedback point or override is a governance node in a socio-technical system. Prototyping intelligent behavior means simulating cognition, not just interaction. Before the model even works, designers can model system reasoning: what inputs it listens to, how it weighs them, and how it communicates uncertainty. That’s how you prototype explainability early-before accuracy takes over the agenda. In practice, the best AI teams combine technical literacy with behavioral empathy. Data scientists understand distributions; designers understand interpretation. Together, they build systems that not only learn from data but learn from people. Human-centric AI doesn’t just optimize performance - it aligns cognition, decision, and design around human meaning. That’s what makes intelligence truly useful.