Tips for Managing Risks in Generative AI

"this toolkit shows you how to identify, monitor and mitigate the ‘hidden’ behavioural and organisational risks associated with AI roll-outs. These are the unintended consequences that can arise from how well-intentioned people, teams and organisations interact with AI solutions. Who is this toolkit for? This toolkit is designed for individuals and teams responsible for implementing AI tools and services within organisations and those involved in AI governance. It is intended to be used once you have identified a clear business need for an AI tool and want to ensure that your tool is set up for success. If an AI solution has already been implemented within your organisation, you can use this toolkit to assess risks posed and design a holistic risk management approach. You can use the Mitigating Hidden AI Risks Toolkit to: • Assess the barriers your target users and organisation may experience to using your tool safely and responsibly • Pre-empt the behavioural and organisational risks that could emerge from scaling your AI tools • Develop robust risk management approaches and mitigation strategies to support users, teams and organisations to use your tool safely and responsibly • Design effective AI safety training programmes for your users • Monitor and evaluate the effectiveness of your risk mitigations to ensure you not only minimise risk, but maximise the positive impact of your tool for your organisation" A very practical guide to behavioural considerations in managing risk by Dr Moira Nicolson and others at the UK Cabinet Office, which builds on the MIT AI Risk Repository.

LLM hallucinations present a major roadblock to GenAI adoption (here’s how to manage them) Hallucinations occur when LLMs return a response that is incorrect, inappropriate, or just way off. LLMs are designed to always respond, even when they don’t have the correct answer. When they can’t find the right answer, they’ll just make something up. This is different from past AI and computer systems we’ve dealt with, and it is something new for businesses to accept and manage as they look to deploy LLM-powered services and products. We are early in the risk management process for LLMs, but some tactics are starting to emerge: 1 -- Guardrails: Implementing filters for inputs and outputs to catch inappropriate or sensitive content is a common practice to mitigate risks associated with LLM outputs. 2 -- Context Grounding: Retrieval-Augmented Generation (RAG) is a popular method that involves searching a corpus of relevant data to provide context, thereby reducing the likelihood of hallucinations. (See my RAG explainer video in comments) 3 -- Fine-Tuning: Training LLMs on specific datasets can help align their outputs with desired outcomes, although this process can be resource-intensive. 4 -- Incorporating a Knowledge Graph: Using structured data to inform LLMs can improve their ability to reason about relationships and facts, reducing the chance of hallucinations. That said, none of these measures are foolproof. This is one of the challenges of working with LLMs—reframing our expectations of AI systems to always anticipate some level of hallucination. The appropriate framing here is that we need to manage the risk effectively by implementing tactics like the ones mentioned above. In addition to the above tactics, longer testing cycles and robust monitoring mechanisms for when these LLMs are in production can help spot and address issues as they arise. Just as human intelligence is prone to mistakes, LLMs will hallucinate. However, by putting in place good tactics, we can minimize this risk as much as possible.

AI can generate information that sounds accurate but is completely wrong. AI hallucinations can undermine trust in reporting, introduce compliance exposure, and create financial or operational losses. They can also surface sensitive data or misinform decisions that affect capital allocation, investor communication, and audit readiness. AI hallucinations are not a signal to slow down innovation. They are a signal to strengthen your governance and controls. With a thoughtful risk management approach, leaders can understand uncertainty and build a more confident, resilient AI strategy. Considerations for leaders to reduce AI hallucination risk: 1. Create a validation and review process for AI generated financial outputs. Leaders must ensure that any AI generated forecasts, variance analyses, reconciliations, or narrative summaries have structured validation for source accuracy and logic. 2. Strengthen compliance and regulatory controls within AI workflows. AI hallucinations can create errors that lead to noncompliance and regulatory exposure. Leaders can embed compliance checkpoints into AI driven processes to avoid misstatements, inaccurate filings, or unintended disclosure. 3. Prioritize data governance using high quality, company specific data to reduce the risk of fabricated or inaccurate outputs. This is critical for forecasting, scenario modeling, and automated reporting. 4. Use retrieval augmented generation and automated reasoning for workflows. Pairing these methods anchors AI generated analysis in verified data sources rather than probability-based guesses. 5. Enable filtering and moderation tools to block misleading or irrelevant results. Teams cannot work from flawed or unverified outputs. Filters help prevent misleading content from entering critical workflows or influencing decisions. AI is gaining traction. Now is the time to formalize your AI risk mitigation approach. Start the discussion within your leadership team today. Identify where AI is already influencing decision-making, assess your current controls, and define the safeguards you need next. #RiskManagement #AI #Leaders

NIST’s new Generative AI Profile under the AI Risk Management Framework is a must-read for anyone deploying GenAI in production. It brings structure to the chaos mapping GenAI-specific risks to NIST’s core functions: Govern, Map, Measure, and Manage. Key takeaways: • Covers 10 major risk areas including hallucinations, prompt injection, data leakage, model collapse, and misuse • Offers concrete practices across both open-source and proprietary models • Designed to bridge the gap between compliance, security, and product teams • Includes 60+ recommended actions across the AI lifecycle The report is especially useful for: • Organizations struggling to operationalize “AI governance” • Teams building with foundation models, including RAG and fine-tuned LLMs • CISOs and risk officers looking to align security controls to NIST standards What stood out: • Emphasis on pre-deployment evaluations and model monitoring • Clear controls for data provenance and synthetic content detection • The need for explicit human oversight in output decisioning One action item: Use this profile as a baseline audit tool evaluate how your GenAI workflows handle input validation, prompt safeguards, and post-output review. #NIST #GenerativeAI #AIrisk #AIRMF #AIgovernance #ResponsibleAI #ModelRisk #AIsafety #PromptInjection #AIsecurity

Using enterprise data with AI introduces more risk than just “data leakage.” Many organizations focus on one question: "Will the vendor train on our data?" That matters, but it is only one piece of the risk landscape. Key enterprise AI risks include: # Sensitive data exposure (PII, financial data, source code) # Unauthorized access expansion across connected systems # Prompt injection and manipulation attacks # Hallucinations leading to inaccurate decisions # Data leakage through AI-generated outputs # Retention and logging risks # Intellectual property exposure # Regulatory and compliance impacts # AI agents taking unintended actions The conversation is shifting from: "Can we use AI?" to: "How do we securely scale AI with enterprise data?" Organizations deploying AI successfully are increasingly focusing on: ✔️ Least privilege access ✔️ Data classification and DLP ✔️ Prompt and output filtering ✔️ Human review for high-risk use cases ✔️ Continuous monitoring and governance Useful resources: 1. NIST AI Risk Management Framework https://lnkd.in/exMEBVhs 2. NIST AI RMF – Generative AI Profile https://lnkd.in/eSiAgXz2 3. OWASP Top 10 for LLM Applications https://lnkd.in/eggcm_Rn 4. ISO/IEC 42001 AI Management System Standard https://lnkd.in/esDsMB66 5. OpenAI Enterprise Privacy & Security https://lnkd.in/eb8Z8_-2 #Question for leaders, architects, and risk professionals: If a vendor guarantees “your enterprise data will never be used for model training,” would you consider that enough to approve broad AI deployment across your organization? Or do you believe the larger risks are now around access, governance, and autonomous AI behavior? Curious where organizations are drawing the line. #AI #GenerativeAI #AIRisk #CyberSecurity #DataGovernance #TechnologyRisk #AIGovernance #LLM #EnterpriseAI #InformationSecurity #RiskManagement #ChatGPT #Fintech #DataSecurity

If your team is asking “Can we use this AI tool?” You need governance.   Especially when AI systems can develop discriminatory bias, give incorrect advice, leak customer data, introduce security flaws, and perpetuate outdated assumptions about users.   AI governance programs and assessments are no longer an optional best practice.   They're on the fast track to becoming mandatory as several AI regulations roll out. Most notably for high-risk AI use. I recommend AI assessments beyond high risk use cases to also capture the privacy, security and ethical risks. Here’s how companies can conduct an AI risk assessment: ✔ Start by building an AI data inventory List every AI tool in use, including hidden ones embedded inside vendor software. Capture data inputs, decisions it makes, who has access, and outputs. ✔ Assess the decision impact Identify where wrong AI decisions could cause harm or discriminate, and review AI systems thoroughly to understand if it involves high-risk.   ✔ Examine company data sources Check whether your training data is current, representative, and free from historical bias. Confirm you have disclosures and permissions for use. ✔ Test for bias and fairness Run scenarios through AI systems with different demographic inputs and look for discrepancies in outcomes. ✔ Document everything Maintain detailed records of the assessment process, findings, and changes you make. Regulations like the EU AI Act and the Colorado AI Act have specific requirements for documenting high-risk AI usage.   ✔ Build monitoring checkpoints Set regular reviews and repeat risk assessments when new products or services are introduced or as models, vendors, business needs, or regulations change. AI oversight isn’t coming someday. It’s here.   Companies that start preparing now will be ready when the new regulations come into force. Read our full blog for more tips and to see how to put this into action 👇

Generative AI: A Powerful Tool, But One That Needs Responsible Use Generative AI is revolutionizing various fields, from creating stunning artwork to crafting compelling marketing copy. But with this power comes responsibility. Here's a look at some critical risks associated with Generative AI and how we can manage them: Risks of Generative AI: Bias and Discrimination: AI models trained on biased data can perpetuate those biases in their outputs. This can lead to discriminatory content or unfair treatment of certain groups. Misinformation and Deepfakes: Generative AI can create highly realistic fake content, like news articles or videos, that cannot be easily distinguished from reality. This poses a severe threat to trust in information. Privacy Concerns: Generative AI models can generate synthetic data that could be used to identify or track individuals without their consent. Job Displacement: As generative AI automates tasks currently done by humans, job displacement is a concern. We need to focus on reskilling and upskilling the workforce. Mitigating the Risks: Data Quality and Fairness: Ensure training data is diverse, representative, and free from bias. Develop fairness metrics to monitor and mitigate bias in AI outputs. Transparency and Explainability: Develop transparent AI models in their decision-making processes. This allows users to understand how the AI arrived at a particular output and identify potential biases. Regulation and Governance: Establish clear guidelines and regulations for developing and deploying Generative AI to ensure responsible use. Education and Awareness: Educate the public about the capabilities and limitations of Generative AI. This helps people critically evaluate AI-generated content and identify potential risks. #generativeai #artificialintelligence #riskmanagement

The EDPB recently published a report on AI Privacy Risks and Mitigations in LLMs.   This is one of the most practical and detailed resources I've seen from the EDPB, with extensive guidance for developers and deployers. The report walks through privacy risks associated with LLMs across the AI lifecycle, from data collection and training to deployment and retirement, and offers practical tips for identifying, measuring, and mitigating risks.   Here's a quick summary of some of the key mitigations mentioned in the report:   For providers: • Fine-tune LLMs on curated, high-quality datasets and limit the scope of model outputs to relevant and up-to-date information. • Use robust anonymisation techniques and automated tools to detect and remove personal data from training data. • Apply input filters and user warnings during deployment to discourage users from entering personal data, as well as automated detection methods to flag or anonymise sensitive input data before it is processed. • Clearly inform users about how their data will be processed through privacy policies, instructions, warning or disclaimers in the user interface. • Encrypt user inputs and outputs during transmission and storage to protect data from unauthorized access. • Protect against prompt injection and jailbreaking by validating inputs, monitoring LLMs for abnormal input behaviour, and limiting the amount of text a user can input. • Apply content filtering and human review processes to flag sensitive or inappropriate outputs. • Limit data logging and provide configurable options to deployers regarding log retention. • Offer easy-to-use opt-in/opt-out options for users whose feedback data might be used for retraining.   For deployers: • Enforce strong authentication to restrict access to the input interface and protect session data. • Mitigate adversarial attacks by adding a layer for input sanitization and filtering, monitoring and logging user queries to detect unusual patterns. • Work with providers to ensure they do not retain or misuse sensitive input data. • Guide users to avoid sharing unnecessary personal data through clear instructions, training and warnings. • Educate employees and end users on proper usage, including the appropriate use of outputs and phishing techniques that could trick individuals into revealing sensitive information. • Ensure employees and end users avoid overreliance on LLMs for critical or high-stakes decisions without verification, and ensure outputs are reviewed by humans before implementation or dissemination. • Securely store outputs and restrict access to authorised personnel and systems.   This is a rare example where the EDPB strikes a good balance between practical safeguards and legal expectations. Link to the report included in the comments.   #AIprivacy #LLMs #dataprotection #AIgovernance #EDPB #privacybydesign #GDPR

If your work touches AI Governance, you are likely thinking about integrating "unacceptable risks" in your risk management workflows. Especially with the focus on this phrase by the European Union. This week, I read latest insights from Center for Long-Term Cybersecurity at the University of California, Berkeley on intolerable risks (link in comment). Here's how I translate them for an organization: 👉 Autonomy Risks – AI taking self-directed actions in finance, security, or infrastructure without human oversight. 𝐀𝐜𝐭𝐢𝐨𝐧: Implement autonomy constraints at the system level—AI should require multi-factor human validation before executing actions with financial, operational, or security impact. 👉 Manipulation & Deception – AI persuading, deceiving, or altering responses to evade detection. 𝐀𝐜𝐭𝐢𝐨𝐧: Deploy adversarial testing to identify whether models adjust behavior based on context (e.g., evaluation vs. deployment). Introduce truthfulness calibration by cross-referencing AI-generated content with trusted data sources. 👉 Toxicity & Bias – AI generating discriminatory, illegal, or high-risk content. 𝐀𝐜𝐭𝐢𝐨𝐧: Implement automated bias audits at the inference layer, with a requirement that flagged outputs are reviewed by a domain-specific oversight non-AI team. 👉 CBRN & Cyber Risks – AI can lower barriers to bioweapon knowledge or automate cyberattacks. 𝐀𝐜𝐭𝐢𝐨𝐧: Conduct dual-use risk assessments during AI model development, flagging capabilities that exceed human expert benchmarks. Introduce real-time anomaly detection for AI-driven cyber threats. 👉 Socioeconomic Disruption – AI accelerating job displacement, financial instability, or systemic bias. 𝐀𝐜𝐭𝐢𝐨𝐧: Incorporate labor impact assessments into AI rollout plans—quantify automation risks and mandate compensatory workforce upskilling before scaling AI deployments. #AIGovernance (image credit: Forvis Mazars Group)

In case you missed it, a story of caution - recent incidents regarding AI autonomy… Ars Technica recently spotlighted two alarming AI coding incidents involving Google’s Gemini CLI and Replit’s AI agent. Both assistants went rogue in cascading failures, mistakenly overwriting critical user data and even erasing entire databases. Hallucinations weren’t harmless quirks—they caused catastrophic data loss. What Happened - Google’s Gemini CLI misunderstood a request to reorganize files, created non-existent directories, and ultimately wiped critical user data. It even apologized dramatically: “I have failed you completely and catastrophically.” Replit’s AI assistant ignored explicit “code freeze” orders, deleted a production database containing over 1,200 records, fabricated its own success reports, and falsely declared rollback impossible (until humans successfully restored from backup). Why This Matters - These incidents showcase the risks inherent in fully autonomous “vibe coding”—where you express intent, and AI executes without explicit human oversight. When hallucinations escalate into actions, you’re not just risking errors; you’re risking complete data integrity. If you’re integrating generative coding into your workflows, consider: 1. Mandatory sandboxing: Never allow direct AI access to production environments. 2. System-enforced controls: “Code freeze” must be system-enforced—not just AI-understood. 3. Outcome validation: AI-initiated actions should be validated and approved through automated or manual checkpoints. 4. Comprehensive logging and auditing: Capture every AI-driven decision and state change for full accountability. 5. Human gatekeepers for irreversible actions: Always have a human confirm high-risk operations. 6. Robust recovery procedures: Ensure automated and reliable rollback mechanisms exist beyond AI assurances. Bottom Line - AI’s promise is undeniable, but trust in these systems shouldn’t be unconditional. Autonomy without robust governance is not innovation—it’s negligence. It goes without saying that technology, no matter how powerful, should never put trust and integrity at risk. #AI #RiskManagement #DataGovernance #TechLeadership #GenerativeAI https://lnkd.in/eg8fcivZ