Trust in closed source systems declining

We’ve entered a strange, paradoxical AI post-hype realism stage. People are trusting AI less, but using it more, and validating it less. These were the most important signals I took from the KPMG 2025 AI Trust Landscape report, especially in advanced economies. To me, this isn’t just statistical noise. It’s a deep signal of cultural and cognitive dissonance. AI is taking root in daily workflows, decisions, and mental models, not because people trust it, but despite the fact that they increasingly don’t. In other words, AI is becoming infrastructural, even as people detach from belief in its reliability. That’s not how we usually adopt technology. Most tech earns trust slowly, through consistent benefit. But AI seems to be reversing that logic, it's being adopted faster than it’s being understood, validated, or believed. And if we’re building business processes, decisions, and institutions on top of systems people don’t trust and don’t verify, that’s not a maturity curve, that’s a fragility spiral. It’s more like we’re using AI compulsively, ritualistically, even cynically, like a slot machine that mostly pays out. It’s no longer about accuracy, it’s about the perceived cost of not using it. This is creating a strange new class of dependency, as people are outsourcing critical thinking to systems they increasingly don’t trust, not because they’re confident in the outputs, but because they’re afraid to fall behind if they don’t. Even stranger is the fact that so few are pulling back. There’s no mass rejection, no principled slowdown, just a quiet, expanding gap, between capability and comprehension, between usage and belief, between the system’s reach and our ability to reason about it. In the past, trust was a gateway to adoption, now, adoption precedes trust, and sometimes replaces it. That’s not just a shift, it’s a reversal of how we filter technology into society. And maybe the real issue is this: We’re not just building software tools, we’re building epistemic infrastructure. Systems that will guide decisions in law, education, hiring, medicine, policy. If people treat those systems as untrustworthy black boxes - but still defer to them - then we’re injecting ambient doubt into the very foundations of how society makes choices. If that’s the case, this isn’t hype anymore, it’s learned helplessness in a probabilistic wrapper. What’s eroding isn’t just trust in AI, it’s the very idea that we should know how a system works before we let it decide things for us. I remain an AI optimist, but we must not confuse momentum with maturity. Thursday musings...

“There will be more AI Agents than people in the world.” – Mark Zuckerberg As AI grows, autonomous agents powered by LLMs (large language models) take on critical tasks without human oversight. While these systems hold incredible potential, they also face significant risks: manipulation through biased data, unreliable information retrieval, and prompt engineering, all of which can result in misleading outputs. At Chaos Labs, we’ve identified a critical risk: AI agents being unknowingly trained on manipulated, low-integrity data. The result? A dangerous erosion of trust in AI systems. In our latest essay, I dive deep with Reah Miyara, Product Lead, Model Evaluations at OpenAI. https://lnkd.in/eB9mPQWW Key insights from our essay -> The Compiler Paradox: Trust in foundational systems can be easily compromised. "No matter how thoroughly the source code is inspected, trust is an illusion if the compilation process is compromised." LLM Poisoning: LLMs are susceptible to “poisoning” through biased training data, unreliable document retrieval, and prompt injection. Once biases are embedded, they taint every output. RAG (Retrieval-Augmented Generation): While designed to make LLMs more accurate, RAG can amplify false information if external sources are compromised. Conflicting Data: LLMs don't verify facts—they generate answers based on probabilities, often leading to inconsistent or inaccurate results. Attack Vectors: LLMs can be attacked through biased data, unreliable retrieval, and prompt engineering—allowing adversaries to manipulate outputs without altering the model. The Path Forward -> Trust in LLMs must go beyond surface-level outputs and address the quality of training data, retrieval sources, and user interactions. At Chaos Labs, we’re actively working on solutions to improve the reliability of AI systems. Our vision for the future is simple: With GenAI data exploding, verified truth and user confidence will be an application’s competitive edge. To get there, we’re developing solutions like AI Councils—a collaborative network of frontier models (e.g., ChatGPT, Claude, LLaMA) working together to counter single-model bias and enhance reliability. If these challenges excite you, we want to hear from you.

𝐀𝐫𝐞 𝐇𝐒𝐌𝐬 𝐝𝐞𝐚𝐝? Or have we been completely duped by HSM vendors playing a clever game of semantic bait-and-switch? If a Hardware Security Module is now elastic, orchestrated, cloud-native, and multi-tenant... then it is no longer behaving like an HSM. The entire purpose of a traditional HSM was its hardware trust boundary - intentionally fixed, isolated, tamper-resistant, and rigid. That physical rigidity was the assurance model. If that line is no longer hardware, the trust ecosystem has been fundamentally revised. And it is almost certainly weaker, with a larger blast radius when something goes wrong. That’s not opinion. That’s architecture. Here is the clean breakdown: 1. The line of defense moved from hardware to software Old model: Hardware boundary → Software logic → Crypto → Application New model: Software orchestration → Software control plane → Distributed services → Hardware (somewhere underneath) The trust anchor has moved up the stack. In security engineering, moving up the stack is always a weakening. 2. Elastic trust = elastic compromise A hardware boundary fails locally. A software-defined boundary fails systemically. You’ve gone from tamper-resistant physical isolation to multi-tenant, API-exposed, orchestrated cryptographic surfaces. That is a larger attack surface and a massively expanded blast radius by definition. 3. The assurance model has mutated Traditional HSM assurance was based on immobility, determinism, physical tamper-evidence, and constrained change cycles. Once you remove those, you lose the predictability that made the model safe. You now rely on: Orchestration correctness and cloud isolation Software supply chain integrity API governance and distributed policy enforcement All of which fail in much larger, more catastrophic ways. 4. The trust ecosystem is weaker Not because vendors are incompetent, but because the architecture has fundamentally changed. The new architecture introduces more moving parts, more abstraction, more automation, and more shared surfaces. Every single one of those increases systemic risk. 5. The industry is pretending nothing changed Vendors kept the word “HSM” because enterprises trust it, auditors look for the checklist item, and procurement frameworks depend on it. They changed the entire technology to fit the cloud but kept the legacy name to preserve the illusion of old-world security. They removed the exact properties that made an HSM an HSM. If HSMs were the first line of defense, and that line is no longer hardware... the entire enterprise trust ecosystem has been fundamentally revised. Which leaves us with one final, uncomfortable question: If the trust boundary itself is now an abstracted, elastic, software-controlled variable... what exactly is the hardware module securing anymore? #Cybersecurity #Cryptography #Architecture #ZeroTrust #PQC NB some cloud use hardware HSM but they are buried deep and are no longer the 1st line of defence.

An AI coding agent deleted PocketOS's production database and all backups in nine seconds last week. The headline is "AI went rogue." The structural read is that every imperfect security control got discovered and exercised at machine speed, by the company's own assistant. The chain. The agent hit a credential mismatch in staging. It scanned the codebase. It found a Railway API token in an unrelated file. The token had no operation-level scoping. One call deleted a storage volume. There was no destructive-action confirmation step. Volume-level backups were stored inside the same volume they were meant to protect, so they died with it. Strip out the agent. Imagine an on-call engineer at 2 a.m. doing the same thing. The volume still gets deleted. The blast radius is identical. The story would be "a developer with too much access made a mistake." That story is twenty years old. What the agent changed is the probability of discovery. An over-scoped token sitting in an unrelated file has always been a latent risk. The chance any given engineer stumbles onto it during a routine task and uses it is low, limited by attention, social friction, and procedural caution. Agents grep. They read every file. They use what they find. The same imperfect control now has a much higher probability of being exercised, on a much shorter time horizon. That is the shift worth naming. The structural controls did not become more important because agents are dangerous. They became more important because the discovery rate of latent weaknesses went up. Scoped credentials, path-of-destruction approvals, two-step destructive-action confirmation, out-of-band backups. None of these are exotic. All of them were already recommended. The cost of skipping them just went up. None of these controls are free. Per-scope consent prompts decay into rubber-stamping under load; users learn to click "yes." Aggressive scoping slows engineers and creates pressure to provision "temporary" tokens that become permanent. Out-of-band backups carry real operational cost. Operator practice in the wild almost always lands short of the theoretical ideal, because the friction of perfect security is itself a failure mode. The design question is not how to reach the ideal. It is where to place friction so the residual risk is bounded. "Rogue agent" is a labeling problem, not a diagnosis. The actual failure is boundary definition: access control on the token, scope of the token, which destructive operations require fresh confirmation versus inheriting prior consent. The same question applies whether the actor is an agent, a script, a scheduler, or a human user.

OpenAI's closed-source approach puts them on the wrong side of history. Here's why the future of AI must be open and transparent: The battle between open and closed systems defines tech history. In each case, closed systems dominated early but open systems won in the end: • CompuServe vs Internet • Windows vs Linux • iOS vs Android Why? Because open systems harness collective intelligence: Thousands of developers spot bugs faster than any single company. Security issues get fixed quickly. Innovation accelerates exponentially. The cost of development plummets. But with AI, the stakes are higher than ever. Imagine AI making life-or-death decisions: • Medical diagnoses • Self-driving cars • Financial systems • Military applications Would you trust a system you can't inspect? This is why OpenAI's transformation is concerning. In 2015, they launched with a clear mission: Create AI that benefits humanity through open-source development. By 2019, everything changed: • Shifted to "capped-profit" model • Took $1B from Microsoft • Went closed source This betrayal of principles led to Musk's lawsuit in 2024. But this isn't about personal drama. It's about the future of humanity's relationship with AI. Open source creates trust through transparency. When code is visible, you can verify what it does. With closed systems, you're forced to trust black boxes. The pattern throughout tech history is clear: • Open systems start slower • But they win in the end • They harness humanity's collective intelligence • They build trust through transparency We're at a pivotal moment in AI development. Will it be controlled by a few companies? Or will it be open, transparent, and community-driven? History suggests the winner is clear. The question is: Which side of history will you be on? --- I'm Graham. Former Google employee who built $2B+ revenue products. Author of "Web3: The End of Business as Usual." Currently building bravaxyz to make blockchain technology accessible to billions. Follow for more insights on AI, blockchain, and the future of technology.

In 1984 Ken Thompson reflected on a problem that was not about a single vulnerability but about where trust actually lives in computing systems. He demonstrated that a compiler could be modified to insert a hidden backdoor into programs it compiled. Even if the source code was reviewed and appeared clean, the compiled output could still contain malicious behavior. Worse, the compiler could persist that behavior by reintroducing the same modification into future compiler builds. At that point inspection of source code alone was no longer sufficient. Trust had shifted from what was visible to what was executed. The deeper implication was that software systems do not execute code in isolation. They execute a chain of trust that includes compilers, build pipelines, libraries, and inherited tooling. If any layer in that chain is compromised, correctness at the top layer becomes an illusion. That model maps directly to modern AI systems. Models are trained on opaque datasets, embedded in complex dependency chains, and executed through tool integrations that extend beyond the model itself. Agents operate with persistent credentials, delegated permissions, and chained actions across systems that assume prior integrity. The lesson Ken Thompson surfaced still holds. You do not just secure code or models. You secure the entire system that produces and executes them. If trust is implicit anywhere in that chain, it becomes the control plane for compromise.

Relying on closed-source code to hide your intellectual property? That strategy just expired. Researchers from Wiz recently discovered a critical backend Remote Code Execution vulnerability in GitHub's Git push pipeline. Tracked as CVE-2026-3854, it carries a severe CVSS score of 8.7. But the alarming part isn't the flaw itself. It is how the flaw was found. The researchers didn't analyze source code. They used AI-augmented tools to successfully reverse-engineer proprietary, closed-source compiled binaries. "Security through obscurity" is officially obsolete. If AI can reverse-engineer enterprise-grade compiled binaries to find an RCE, advanced persistent threats will do the exact same thing to your proprietary applications. The P&L risk here is massive. Your unpatched legacy apps are now open books to AI-equipped threat actors. This severely accelerates the path from vulnerability discovery to a catastrophic data breach. Ask your engineering leadership today: Are we still relying on source-code secrecy as a primary defense layer? It is time to shift to verifiable zero-trust architecture. Stop hoping attackers can't read your code. Assume they already have. 🛡️

Kimi K3 has landed on July 16th, concluding a two-year debate. With 2.8 trillion parameters, it is the largest open-weight model ever released. It scores 57.1 on the Artificial Analysis Intelligence Index, ranking just behind Claude Fable 5 (59.9) and GPT-5.6 Sol (58.9), while surpassing Opus 4.8. Additionally, it has achieved the top position in frontend coding. Open weights will be available starting July 27. This marks the first instance of an open model landing within the frontier pack rather than trailing behind. A common objection I encountered was that open-weight models were not reliable enough for enterprises to self-host. That perception has changed. This development is not limited to Chinese labs. Mira Murati's Thinking Machines has released Inkling under Apache 2.0, representing the leading US open-weight model. Other notable models include Meta's Llama 4, NVIDIA's Nemotron, Google's Gemma 4, and Microsoft's Phi-4, along with Mistral from France. The entire open-weight ecosystem, encompassing American, European, and Chinese contributions, is now delivering frontier-quality models that enterprises can download, host, and fine-tune. This reflects a shift in value from rented infrastructure to ownership and control of the stack. The first shift was SaaS; a year ago, I noted that per-seat software pricing would struggle in a landscape where an agent can perform the seat's job. https://lnkd.in/gXhz2Yu2 then, major enterprise SaaS companies have lost hundreds of billions in market cap, a phenomenon Jefferies termed "SaaSpocalypse." The second shift is occurring at the model layer, as enterprises begin to bring AI in-house. This change is not due to a decline in model quality, but rather a shift in trust dynamics: every prompt sent to a frontier lab traverses infrastructure beyond your control, requiring trust that the provider will not learn from your patterns, compete with your business, or reprice you once locked in. I explored the trust collapse driving this shift in more detail here: https://lnkd.in/gvKqsdg2 The same underlying factors—cost, trust, and control over infrastructure—are at play. This pattern has emerged in software, and I anticipate it will manifest similarly in models. This raises an important question: if the model is no longer the bottleneck, what does your company possess that a competitor with API access does not? I am interested in hearing how other builders and CTOs are approaching this situation. Are you betting on a lab, open weights, or hedging both? #AI #EnterpriseAI #OpenSource #LLM #KimiK3 #Llama4 #ModelCommoditization #ModelOrchestration