Best Practices for AI Experimentation

If you are building AI agents or learning about them, then you should keep these best practices in mind 👇 Building agentic systems isn’t just about chaining prompts anymore, it’s about designing robust, interpretable, and production-grade systems that interact with tools, humans, and other agents in complex environments. Here are 10 essential design principles you need to know: ➡️ Modular Architectures Separate planning, reasoning, perception, and actuation. This makes your agents more interpretable and easier to debug. Think planner-executor separation in LangGraph or CogAgent-style designs. ➡️ Tool-Use APIs via MCP or Open Function Calling Adopt the Model Context Protocol (MCP) or OpenAI’s Function Calling to interface safely with external tools. These standard interfaces provide strong typing, parameter validation, and consistent execution behavior. ➡️ Long-Term & Working Memory Memory is non-optional for non-trivial agents. Use hybrid memory stacks, vector search tools like MemGPT or Marqo for retrieval, combined with structured memory systems like LlamaIndex agents for factual consistency. ➡️ Reflection & Self-Critique Loops Implement agent self-evaluation using ReAct, Reflexion, or emerging techniques like Voyager-style curriculum refinement. Reflection improves reasoning and helps correct hallucinated chains of thought. ➡️ Planning with Hierarchies Use hierarchical planning: a high-level planner for task decomposition and a low-level executor to interact with tools. This improves reusability and modularity, especially in multi-step or multi-modal workflows. ➡️ Multi-Agent Collaboration Use protocols like AutoGen, A2A, or ChatDev to support agent-to-agent negotiation, subtask allocation, and cooperative planning. This is foundational for open-ended workflows and enterprise-scale orchestration. ➡️ Simulation + Eval Harnesses Always test in simulation. Use benchmarks like ToolBench, SWE-agent, or AgentBoard to validate agent performance before production. This minimizes surprises and surfaces regressions early. ➡️ Safety & Alignment Layers Don’t ship agents without guardrails. Use tools like Llama Guard v4, Prompt Shield, and role-based access controls. Add structured rate-limiting to prevent overuse or sensitive tool invocation. ➡️ Cost-Aware Agent Execution Implement token budgeting, step count tracking, and execution metrics. Especially in multi-agent settings, costs can grow exponentially if unbounded. ➡️ Human-in-the-Loop Orchestration Always have an escalation path. Add override triggers, fallback LLMs, or route to human-in-the-loop for edge cases and critical decision points. This protects quality and trust. PS: If you are interested to learn more about AI Agents and MCP, join the hands-on workshop, I am hosting on 31st May: https://lnkd.in/dWyiN89z If you found this insightful, share this with your network ♻️ Follow me (Aishwarya Srinivasan) for more AI insights and educational content.

The difference between AI experimentation and AI performance is operational design. MIT’s NANDA Institute found that 95% of generative AI pilots fail to deliver measurable P&L impact. Not because of the models. Because of how companies build and deploy AI. The ones breaking out of pilot purgatory are running a different system. Mid-market companies have a unique edge: fewer layers of bureaucracy & faster decision cycles let them move from idea to impact quicker. Here’s the 7-point framework winners are using: 1/ Hybrid Org Design, Not a Centralized AI Team → One AI leader (Ops, RevOps, or IT) owns direction and governance → Embedded power users in Marketing, Sales, and CX drive execution → External partners support setup and early use cases Takeaway: Centralized in direction, distributed in execution is what scales. 2/ Use-Case First, Not Platform First → Start with: “Where can we save time or increase revenue within 90 days?” → Not: “Let’s build an AI platform” → Common entry points: content, sales personalization, support automation Takeaway: If you start with a platform, you end up with a science project. 3/ A 3-Layer Stack, Consumed, Not Custom-Built → Layer 1: Models (OpenAI, Anthropic, Google via API) with no custom training, no hosting → Layer 2: Orchestration (Zapier, Make) is the control layer that connects AI to your CRM, CMS, and workflows. → Layer 3: Data (CRM, warehouse, lightweight RAG systems) is where most mid-market companies struggle due to messy or inaccessible data Takeaway: Mid-market advantage = speed. Heavy infrastructure kills it. 4/ Prompt Playbooks, Not Pilots → Winners don’t run 6-month pilots hoping to scale → They build prompt libraries, workflows, and repeatable use cases → Playbooks spread. Pilots stall → Example: A sales personalization playbook used by every SDR vs one isolated AI pilot in RevOps Takeaway: Designed for adoption, not experimentation. 5/ Governance That Enables Speed, Not Friction → Define a small set of approved tools/models → Establish clear guardrails: what data can and cannot be used → Require lightweight vendor review (security, data handling, compliance) → Assign ownership (who is accountable for AI usage across teams) → Skip heavy frameworks early, as they slow momentum before value is proven Takeaway: Most AI initiatives fail because governance either doesn’t exist or shows up too heavy, too early. 6/ Small Bets → Proven ROI → Then Scale → Initial budgets: $10K–$100K → Spend on tools, not headcount → Expand only after measurable returns Takeaway: AI budgets grow on proof. 7/ Measure Efficiency First, Then Tie to Revenue → Early wins: time saved (content, research, reporting) → Then: pipeline, conversion, CX metrics → If AI doesn’t connect to revenue within ~6 months, it stalls Takeaway: The gap between “we use AI” and “AI is a capability” is execution + measurement. Most companies will adopt AI. Very few will operationalize it. That’s where the advantage is now.

AI field note: Reducing the 'mean time to ah-ha' (MTtAh) is critical for driving AI adoption—and unlocking the value. When it comes to AI adoption, there's a crucial milestone: the "ah-ha moment." It's that instant of realization when someone stops seeing AI as just a smarter search tool and starts recognizing it as a reasoning and integration engine—a fundamentally new way of solving problems, driving innovation, and collaborating with technology. For me, that moment came when I saw an AI system not just write code but also deploy it, identify errors, and fix them automatically. In that instant, I realized AI wasn’t just about automation or insights—it was about partnership. A dynamic, reasoning collaborator capable of understanding, iterating, and executing alongside us. But these "ah-ha moments" don’t happen by accident. Systems like ChatGPT or Claude excel at enabling breakthroughs, but it really requires us to ask the right questions. That creates a chicken-and-egg problem: until users see what’s possible, they struggle to imagine what else is possible. So how do we help people get hands-on with AI, especially in enterprise organizations, without relying on traditional training? Here are some approaches we have tried at PwC: 🤖 AI "Hackathons" or Challenges: Host short, low-stakes events where employees can experiment with AI on real problems. For example, marketing teams could test AI for campaign ideas, while operations teams explore process automation. ⚙️ Sandbox Environments: Provide low-friction, risk-aware access to AI tools within a dedicated environment. Let users explore capabilities like text generation, workflow automation, or analytics without worrying about “messing something up.” 🚀 Pre-built Use Cases: Offer ready-to-use templates for specific challenges, such as drafting a client email, summarizing documents, or automating routine reports. Seeing results in action builds confidence and sparks creativity. At PwC we have a community prompt library available to everyone, making it easier to get started. 🧩 Embedded AI Mentors: Assign "AI champions" who can guide teams on applying AI in their work. This informal mentorship encourages experimentation without formal, structured training. We do this at PwC and it's been huge. ⚡️ Integrate AI into Existing Tools: Embed AI into everyday platforms (like email, collaboration tools, or CRM systems) so users can naturally interact with it during routine workflows. Familiarity leads to discovery. Reducing the mean time to ah-ha—the time it takes someone to have that transformative realization—is critical. While starting with familiar use cases lowers the barrier to entry, the real shift happens when users experience AI’s deeper capabilities firsthand.

In the world of Generative AI, 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹-𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 (𝗥𝗔𝗚) is a game-changer. By combining the capabilities of LLMs with domain-specific knowledge retrieval, RAG enables smarter, more relevant AI-driven solutions. But to truly leverage its potential, we must follow some essential 𝗯𝗲𝘀𝘁 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀: 1️⃣ 𝗦𝘁𝗮𝗿𝘁 𝘄𝗶𝘁𝗵 𝗮 𝗖𝗹𝗲𝗮𝗿 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲 Define your problem statement. Whether it’s building intelligent chatbots, document summarization, or customer support systems, clarity on the goal ensures efficient implementation. 2️⃣ 𝗖𝗵𝗼𝗼𝘀𝗲 𝘁𝗵𝗲 𝗥𝗶𝗴𝗵𝘁 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗕𝗮𝘀𝗲 - Ensure your knowledge base is 𝗵𝗶𝗴𝗵-𝗾𝘂𝗮𝗹𝗶𝘁𝘆, 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱, 𝗮𝗻𝗱 𝘂𝗽-𝘁𝗼-𝗱𝗮𝘁𝗲. - Use vector embeddings (e.g., pgvector in PostgreSQL) to represent your data for efficient similarity search. 3️⃣ 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗠𝗲𝗰𝗵𝗮𝗻𝗶𝘀𝗺𝘀 - Use hybrid search techniques (semantic + keyword search) for better precision. - Tools like 𝗽𝗴𝗔𝗜, 𝗪𝗲𝗮𝘃𝗶𝗮𝘁𝗲, or 𝗣𝗶𝗻𝗲𝗰𝗼𝗻𝗲 can enhance retrieval speed and accuracy. 4️⃣ 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗲 𝗬𝗼𝘂𝗿 𝗟𝗟𝗠 (𝗢𝗽𝘁𝗶𝗼𝗻𝗮𝗹) - If your use case demands it, fine-tune the LLM on your domain-specific data for improved contextual understanding. 5️⃣ 𝗘𝗻𝘀𝘂𝗿𝗲 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 - Architect your solution to scale. Use caching, indexing, and distributed architectures to handle growing data and user demands. 6️⃣ 𝗠𝗼𝗻𝗶𝘁𝗼𝗿 𝗮𝗻𝗱 𝗜𝘁𝗲𝗿𝗮𝘁𝗲 - Continuously monitor performance using metrics like retrieval accuracy, response time, and user satisfaction. - Incorporate feedback loops to refine your knowledge base and model performance. 7️⃣ 𝗦𝘁𝗮𝘆 𝗦𝗲𝗰𝘂𝗿𝗲 𝗮𝗻𝗱 𝗖𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝘁 - Handle sensitive data responsibly with encryption and access controls. - Ensure compliance with industry standards (e.g., GDPR, HIPAA). With the right practices, you can unlock its full potential to build powerful, domain-specific AI applications. What are your top tips or challenges?

Tired of AI projects that don't deliver? Try this human-centred approach. From my research over the past couple of years, I’ve noticed a recurring pattern. We often treat AI as a technology experiment rather than an upgrade to how people actually work. That mindset can quietly limit a project’s success. To support better decisions, I’ve developed a human-centred AI readiness checklist based on that research. I hope it’s useful for your next initiative. 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝘆 𝗮𝗻𝗱 𝗢𝘂𝘁𝗰𝗼𝗺𝗲 𝗖𝗵𝗲𝗰𝗸 (𝗖𝗥𝗜𝗦𝗣-𝗗𝗠 𝗺𝗶𝗻𝗱𝘀𝗲𝘁) →Are we clear on the operational outcome and metric we are improving? ↳If we cannot say “this reduces X by Y%”, we are chasing tools, not performance. 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻 𝗠𝗮𝗽𝗽𝗶𝗻𝗴 𝗖𝗵𝗲𝗰𝗸 (𝗟𝗲𝗮𝗻 𝘄𝗼𝗿𝗸𝗳𝗹𝗼𝘄 𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴) →Which real human decisions are we supporting? ↳AI should strengthen judgment points like prioritisation or scheduling, not automate activity without purpose. 𝗣𝗿𝗼𝗰𝗲𝘀𝘀 𝗦𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗖𝗵𝗲𝗰𝗸 (𝗟𝗲𝗮𝗻 𝗽𝗿𝗶𝗻𝗰𝗶𝗽𝗹𝗲) → Is the workflow stable enough to augment? ↳Automating instability scales, defects and frustrates the people doing the work. 𝗩𝗮𝗹𝘂𝗲 𝘃𝘀 𝗗𝗶𝘀𝗿𝘂𝗽𝘁𝗶𝗼𝗻 𝗖𝗵𝗲𝗰𝗸 (𝗣𝗼𝗿𝘁𝗳𝗼𝗹𝗶𝗼 𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴) →Does the benefit outweigh frontline disruption? ↳Operational AI should improve flow, not create friction for teams. 𝗗𝗮𝘁𝗮 𝗥𝗲𝗮𝗹𝗶𝘁𝘆 𝗖𝗵𝗲𝗰𝗸 (𝗖𝗥𝗜𝗦𝗣-𝗗𝗠 𝗱𝗮𝘁𝗮 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴) →Does our data reflect lived operational reality? ↳Human trust collapses when AI runs on distorted inputs. 𝗛𝘂𝗺𝗮𝗻 𝗖𝗼𝗻𝘁𝗿𝗼𝗹 𝗖𝗵𝗲𝗰𝗸 (𝗛𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗲𝗿𝗲𝗱 𝗔𝗜 𝗱𝗲𝘀𝗶𝗴𝗻) →Where does AI advise, where do humans review, and where does automation act? ↳Clear boundaries protect autonomy and accountability. 𝗥𝗶𝘀𝗸 𝗮𝗻𝗱 𝗥𝗲𝘀𝗶𝗹𝗶𝗲𝗻𝗰𝗲 𝗖𝗵𝗲𝗰𝗸 (𝗡𝗜𝗦𝗧 𝗔𝗜 𝗿𝗶𝘀𝗸 𝗺𝗼𝗱𝗲𝗹) →Have we planned for failure, overrides, and fallback workflows? ↳Operations must remain safe and continuous when systems misfire. 𝗢𝘄𝗻𝗲𝗿𝘀𝗵𝗶𝗽 𝗖𝗵𝗲𝗰𝗸 (𝗢𝗽𝗲𝗿𝗮𝘁𝗶𝗻𝗴 𝗺𝗼𝗱𝗲𝗹 𝗰𝗹𝗮𝗿𝗶𝘁𝘆) →Who owns outcomes, model behaviour, and data quality? ↳Human accountability must remain visible after launch. 𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗥𝗲𝗮𝗹𝗶𝘁𝘆 𝗖𝗵𝗲𝗰𝗸 (𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝘁𝗵𝗶𝗻𝗸𝗶𝗻𝗴) →Will this support how people actually work? ↳Tools that slow teams are quietly abandoned. 𝗔𝗱𝗼𝗽𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗧𝗿𝘂𝘀𝘁 𝗖𝗵𝗲𝗰𝗸 (𝗖𝗵𝗮𝗻𝗴𝗲 𝗱𝗶𝘀𝗰𝗶𝗽𝗹𝗶𝗻𝗲) →Are we designing for understanding, transparency, and behavioural adoption? ↳Trust grows when teams see AI improving their work, not replacing it. AI is an amplifier. It scales what we already have: good or bad ↳𝐆𝐚𝐫𝐛𝐚𝐠𝐞 𝐢𝐧. 𝐀𝐦𝐩𝐥𝐢𝐟𝐢𝐞𝐝 𝐠𝐚𝐫𝐛𝐚𝐠𝐞 𝐨𝐮𝐭.⁣ ⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣ ⁣⁣⁣⁣⁣⁣⁣⁣The strongest AI initiatives aren’t just technology deployments. They are human-centred operating upgrades that happen to use AI. ♻️ Share if you found this useful. #AIinBusiness #HumanCenteredAI #Operations #Leadership #AIStrategy

We did an experiment: 36 leaders. Same data. Different use of AI. Completely different outcomes. Last week, I was a guest lecturer in Dr. H /Hila Lifshitz (Hán, X也)’s Artificial Intelligence for Managers program, where we conducted a live experiment with 36 senior leaders, including CEOs, VPs, and CIOs from manufacturing, healthcare, public sector, and tech. The goal: to explore how different ways of working with AI change the quality of decisions. Each group received the same set of startup pitch decks, but with different AI access: Group A used AI from the start. Group B used AI only in the last 15 minutes. Group C used no AI until the end. The results were eye-opening. Here’s what we learned fast: → Prompts = process Give AI a role (“Act as a seed-stage angel with 100 investments”), set step-by-step criteria (team → market → moat → risk), and finish with a devil’s advocate challenge. → Stay in control Use AI as analyst and coach, but you make the final call. → Match the mode to the moment: ↳ Sentry (guardrails first) for high-risk or regulated work ↳ Cyborg (human + AI intertwined) for complex decision-making ↳ Autopilot (delegate and verify) only for low-risk, repeatable tasks In the second part of the lecture, I shared a practical framework I call “your new AI toolbox”: 1. NotebookLM for board prep Upload materials, ask for blind spots and key questions. 2. AI Studio for difficult conversations Draft, role-play, and refine your language. 3. Gemini for talent Recruit your best advisory board with personalized outreach. 4. AI Studio Live Share your screen, co-prompt, and capture real-time decisions. 5. Feedback loop Let AI critique your work, then decide what to keep or drop. Why it matters: same people, same data. But leaders who know how to use AI thoughtfully make faster, clearer, and more confident decisions. If your team is navigating how to integrate AI into daily decisions, feel free to DM me and consult with me. ♻️ Repost if you found this helpful. 🔔 Follow me, Yuval Passov, for weekly insights on startup growth, founder wellness, and leadership in the age of AI.

Don’t let your AI project die in a notebook. You don’t need more features. You need structure. This is the folder setup that actually ships from day one. 📁 𝗧𝗵𝗲 𝗳𝗼𝗹𝗱𝗲𝗿 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝘁𝗵𝗮𝘁 𝘄𝗼𝗿𝗸𝘀 Forget monolithic scripts. You need this: /config 🔹YAML files for models, prompts, logs 🔹Config lives outside the code, always /src 🔹Modular logic: llm/, utils/, handlers/ 🔹Clean, testable, scalable from day one /data 🔹Cached outputs, embeddings, prompt responses 🔹Cut latency + save on API costs instantly /notebooks 🔹For testing, analysis, and iteration 🔹Never pollute your main codebase again 𝗪𝗵𝗮𝘁 𝘁𝗵𝗶𝘀 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲 𝘀𝗼𝗹𝘃𝗲𝘀 ▪️Prompt versioning is built in ▪️Rate limiting and caching come standard ▪️Error handling is modular ▪️Experiments stay reproducible ▪️Deployment is one Dockerfile away 𝗕𝗲𝘀𝘁 𝗽𝗿𝗮𝗰𝘁𝗶𝗰𝗲𝘀 𝗯𝗮𝗸𝗲𝗱 𝗶𝗻 1. Prompts are versioned by default ▪️Stored in prompt_templates.yaml + templates⋅py ▪️Track, test, roll back 2. Rate limiting is pre-integrated ▪️rate_limiter⋅py stops API overloads and surprise bills 3. Caching is plug-and-play ▪️Duplicate calls get stored in /data/cache ▪️Cut costs by 70% on day one 4. Each module does one thing only ▪️Models in llm/, logs in utils/, errors in handlers/ ▪️No sprawl 5. Notebooks are safely isolated ▪️Run tests and explorations in prompt_testing.ipynb ▪️Nothing leaks into production logic ⚙️ Clone the github template below - in first comment This structure ships faster, costs less and scales without rewrites. ------------ ⚡I’m Nina. I build with AI and share how it’s done weekly. #aiagents #softwaredevelopment #MCP #genai #promptengineering

85% of AI projects fail. Not because the model is wrong. But because the thinking was fuzzy from day one. Your AI initiative doesn’t need more tech. It needs sharper questions. Here are 4 frameworks to clarify your AI project, from start to scale: 🧠 1. The "Why AI?" Test ↳Know exactly what you're solving. 🔸 If this AI vanished tomorrow, would anyone notice? 🔸 Can you quantify exactly how success looks in reality? 🔸 Are ethical & business risks listed & understood? 🔸 Has every simpler solution already been tested? 🔸 Can stakeholders explain why this AI must exist? 📊 2. The Garbage Test ↳Your AI is only as good as its data. 🔸 Would you vouch for the data you're using? 🔸 Can you defend privacy and compliance decisions? 🔸 What hidden biases could sabotage your model? 🔸 What is your data not telling you? 🔸 When was the last deep data quality audit? 🧪 3. The Demo vs Life Test ↳Make sure your model actually works. 🔸 Can you justify why you chose this specific model? 🔸 Has your model survived real-world test conditions? 🔸 Could you explain the model's decisions to your CEO? 🔸 Can you pinpoint which features matter & why? 🔸 Can you break your model and find when it fails? 🚨 4. The Murphy's Law Test ↳Deploy once, succeed continuously. 🔸 Could you safely roll back deployment in 10 minutes? 🔸 Will you be alerted exactly when the model slips? 🔸 Have you automated triggers to indicate retraining? 🔸 Who is accountable if compliance breaks? 🔸 Can the system scale without late-night emergencies? Don’t just ask: “Does it work?” Ask: “Will it keep working when it matters most?” And don’t ship AI until it passes all four tests. ♻️ Save this and repost to help someone build better AI. ➕ Follow me, Sairam, for more practical AI breakdowns.

🧵10 lessons from building AI Agents to fight crime at TRM. Last week we co-hosted a meetup with Alexander Moks (MASc), and the Vercel team on what it takes to run AI agents in production. The talk was called "AI Agents for Crime Fighters", where I shared 10 hard-earned lessons building mission critical production agents. 1. Thin harness, fat skills. In early 2025 we wrapped a model in supervisor agents, an LLM-judge, a ReACT loop, and hundreds of lines of glue. It gave up after three turns. The newer models let us delete most of that and beat our own evals. 2. Plan to rebuild your harness every 3 months. Models and harnesses are co-designed now. The right harness for last quarter's model is the wrong one for this quarter's. Treat it like the ship of Theseus. Don't get attached. 3. Treat sandboxes like cattle, not pets. Don't SSH into a long-lived stateful sandbox to debug. Expose the sandbox as a tool, keep the harness in the trusted boundary, and make the runtime disposable. 4. Invest in gateways and proxies earlier than you think. LLM providers run at about two nines. A gateway gives you failover; a forward proxy blocks prompt-injection exfil. Bolt these on at the start. 5. Don't paste human APIs into agent tools. Agents choke on deeply nested filters. Flatten the surface, and make your error messages teach. A 422 that names the bad field lets the agent self-heal on the next call. 6. Everything is a file. RAG is over-rated. Persist tool results as files and let the agent grep, jq, and pipe through them. The right primitive is usually a filesystem and a shell, not a vector database. 7. Use auto-research to hill-climb your skill files. Mine past investigations for patterns, hill-climb against best-practice docs, then root-cause failed sessions back into skill diffs. One skill went from 42% to 88% eval pass-rate, every diff agent-authored, and human validated. 8. Invest in observability and experimentation early. Full session traces: every tool call, every parameter, every reasoning step. Without traces, you cannot close the loop. 9. Decouple the agent loop from the transport layer. The loop should run headless, in CI, a notebook, or a cron job, exactly the way it runs behind a websocket. Evals run in CI, not by a developer running scripts. 10. Implicit feedback beats explicit, almost always. Fewer than 10% of sessions leave a thumbs up or down. The other 90% tell you everything through shares, copies, dwell time, and abandons. They never raised their hand. They voted with their behavior. Thanks to all the folks for attended, and the incredible Q&A...we ran an a couple hours over, and got kicked out 🫣 If you want the full talk, or want to dive deeper into any of these topics -- shoot me a DM!

AI tool anarchy is a leadership challenge you can’t ignore. Here are 5 best practices to end it. Across different industries and company sizes, the same pattern keeps surfacing. A senior leader recently told me during a meeting: "we're doing great with AI, everyone here is already using it. I use ChatGPT for writing, Gemini for research, and Copilot in Outlook". Others in the room nodded. Different tools, different habits, no shared rules. This isn't AI strategy, it's AI tool anarchy. Experimentation is healthy. But without structure, it quickly turns into chaos. Most organizations fail to educate teams on: - what data can be shared with AI assistants or agents, - what must remain confidential, - and the risks when sensitive data leaves the organization. Users rarely have bad intentions. But without clear guardrails, security and confidentiality risks are inevitable. Hybrid work amplifies the problem. Personal and corporate tools blend, and boundaries vanish faster than policies can adapt. How should organizations respond to AI tool anarchy? Banning tools is not the answer. The solution is to change how the organization operates with AI. Here are five best practices that consistently deliver results: 1. Educate first (and continuously) AI literacy is non-negotiable for organizations determined to stay competitive and avoid misuse. AI evolves daily.  At CloudX, we certified 100% of our +300 employees in AI, becoming the first company to achieve this. But certification is just the beginning. Our R&D team continuously upskills the entire organization to keep pace. 2. Define clear AI usage principles Establish clear, practical rules for what is allowed, restricted, and forbidden. A ‘traffic light’ framework works: Green — public data and generic text are safe. Yellow — internal non-sensitive data requires human review and sound judgment. Red — PII (Personally Identifiable Information), client secrets, and confidential data are strictly off-limits. 3. Standardize your AI stack Approved enterprise tools, such as Gemini for Google Workspace or Copilot for Microsoft, protect company data from being used to train public models. Standardization also prevents wasted budget on redundant, disconnected subscriptions. 4. Redesign workflows AI assistants are just the starting point. The real value comes from re-engineering processes, not just automating tasks. Identify bottlenecks and redesign those workflows with AI at the core. 5. Align leadership on outcomes, not pilots Adopting AI demands a shift in your operating model. KPIs must move from usage to measurable value. If a tool does not deliver business outcomes within six months, it’s time to reconsider its place in your stack. How is your organization addressing AI tool anarchy? What changes have proven effective in restoring order? Please feel free to share if you want. #AIGovernance