Collaborative AI Models for Medical Diagnosis

AI just ran its own multidisciplinary tumor board. And nailed the diagnosis + treatment. This was a full-stack oncology reasoning engine—pulling from imaging, pathology, genomics, guidelines, and literature in real time. A new paper in Nature Cancer describes how researchers built a GPT-4-powered multitool agent that: • Interprets CT & MRI scans with MedSAM • Identifies KRAS, BRAF, MSI status from histology • Calculates tumor growth over time • Searches PubMed + OncoKB • And synthesizes everything into a cited, evidence-based treatment plan In short: it acts like a multidisciplinary team. Results : • Accuracy jumped from 30% (GPT-4 alone) to 87% • Correct treatment plans in 91% of complex cases • Every conclusion backed by a verifiable citation This is bigger than oncology. Any field that relies on multi-modal data and cross-domain reasoning—like my field of GI ( GI + Mental Health+ Nutrition + Excercise ) could benefit from this collaborative AI architecture. Despite the visual, it doesn’t replace the human team—it augments it. Providers still decide. But now, they do it faster, with more context, and less cognitive fatigue. #AI #HealthcareonLinkedin #Healthcare #Cancer

Imagine AI models working together like a team of doctors, each contributing their expertise to solve complex medical cases. This is what "MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Making" explores, as recently presented at NeurIPS 2024. Working: MDAgents brings a novel approach to using Large Language Models (LLMs) in medicine by dynamically creating a collaborative environment tailored to the complexity of each medical query: 1) Complexity check: Each medical question is evaluated for complexity, determining whether it necessitates a basic, moderate, or advanced collaborative response. 2) Expert recruitment: Based on complexity, MDAgents "recruits" AI agents to act as specialists, forming either a solo practitioner model, a Multi-disciplinary Team (MDT), or an Integrated Care Team (ICT). 3) Analysis and synthesis: The agents engage in collaborative reasoning, using techniques like Chain-of-Thought (CoT) prompting to draw insights and resolve disagreements for more nuanced cases. 4) Decision-making: Synthesizing diverse inputs, the framework reaches a final decision, informed by external medical knowledge and structured discussions among the AI agents. Achievements: 1) MDAgents outperformed both solo and group LLM setups in 7 out of 10 medical tasks, enhancing decision accuracy by up to 11.8%. 2) Demonstrated the critical balance between performance and computational efficiency by adapting the number of participating agents based on task demands. Link to the full paper -> https://lnkd.in/gR7Zwm7t #AI #Healthcare #NeurIPS2024 #MedicalAI #Collaboration #InnovationInMedicine #ResearchInsights

Exciting Innovation in Healthcare AI: MedRAG I just came across a groundbreaking paper introducing MedRAG - a novel approach that enhances Retrieval-Augmented Generation (RAG) with Knowledge Graph-Elicited Reasoning for healthcare applications! Diagnostic errors are a serious problem in healthcare, with approximately 795,000 individuals suffering permanent disability or death annually due to misdiagnosis in the US alone. MedRAG addresses this challenge by significantly improving the accuracy and specificity of AI-powered diagnostic support. >> How MedRAG Works: The system combines RAG with a comprehensive four-tier hierarchical diagnostic knowledge graph to enhance reasoning capabilities. Here's the technical breakdown: 1. Diagnostic Knowledge Graph Construction: MedRAG systematically builds a four-tier hierarchical diagnostic KG through disease clustering, hierarchical aggregation, and LLM augmentation. This captures critical diagnostic differences between diseases with similar manifestations. 2. Diagnostic Differences KG Searching: When a patient's manifestations are input, the system decomposes them into clinical features, embeds them, and matches them with relevant diagnostic differences through multi-level matching and upward traversal within the KG. 3. KG-elicited Reasoning RAG: The system retrieves relevant Electronic Health Records (EHRs) and integrates them with the identified diagnostic differences KG to trigger reasoning in a large language model, generating precise diagnoses and treatment recommendations. 4. Proactive Diagnostic Questioning: MedRAG can identify when patient information is insufficient and proactively suggest follow-up questions based on discriminability scores of features in the knowledge graph. The researchers evaluated MedRAG on both a public dataset (DDXPlus) and a private chronic pain diagnostic dataset from Tan Tock Seng Hospital. It outperformed state-of-the-art RAG models, achieving up to 11.32% improvement in diagnostic accuracy for diseases with similar manifestations. What's particularly impressive is MedRAG's compatibility across various backbone LLMs, including open-source models like Mixtral-8x7B and Llama-3.1-Instruct, as well as closed-source models like GPT-4o. This technology has tremendous potential to reduce misdiagnosis rates and improve healthcare outcomes by providing more accurate, specific diagnostic support and personalized treatment recommendations.

𝐀𝐈 𝐀𝐠𝐞𝐧𝐭 𝐀𝐜𝐡𝐢𝐞𝐯𝐞𝐬 𝟗𝟏% 𝐀𝐜𝐜𝐮𝐫𝐚𝐜𝐲 𝐢𝐧 𝐂𝐚𝐧𝐜𝐞𝐫 𝐃𝐢𝐚𝐠𝐧𝐨𝐬𝐢𝐬 Oncology decision-making is notoriously complex. Clinicians must integrate histopathology images, radiology scans, genetic profiles, and ever-evolving treatment guidelines to make personalized care decisions. It's a cognitive challenge that even experienced specialists find demanding. A new study by Ferber et al. in Nature Cancer shows how an autonomous AI agent tackled this complexity head-on—and the results are striking. 𝗪𝗵𝘆 𝘁𝗵𝗶𝘀 𝗺𝗮𝘁𝘁𝗲𝗿𝘀: Current AI approaches in healthcare often work in isolation—analyzing single data types or providing generic responses. But real clinical decisions require synthesizing multiple sources of evidence simultaneously, something that has remained challenging for AI systems. 𝗞𝗲𝘆 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻𝘀: ◦ 𝐌𝐮𝐥𝐭𝐢𝐦𝐨𝐝𝐚𝐥 𝐭𝐨𝐨𝐥 𝐢𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧: Vision transformers detect genetic mutations directly from tissue slides, MedSAM segments tumors in radiology images, and the system queries precision oncology databases autonomously ◦ 𝐒𝐞𝐪𝐮𝐞𝐧𝐭𝐢𝐚𝐥 𝐫𝐞𝐚𝐬𝐨𝐧𝐢𝐧𝐠: The agent chains tools together—first measuring tumor growth from imaging, then checking mutation databases, then searching recent literature ◦ 𝐄𝐯𝐢𝐝𝐞𝐧𝐜𝐞-𝐛𝐚𝐬𝐞𝐝 𝐜𝐢𝐭𝐚𝐭𝐢𝐨𝐧𝐬: 75.5% accuracy in citing relevant medical guidelines, addressing the critical problem of AI hallucinations in healthcare 𝗧𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀: When tested on 20 realistic patient cases, the integrated system achieved 91% accuracy in clinical conclusions. Perhaps more telling: GPT-4 alone managed only 30% accuracy on the same cases—nearly a 3x improvement through tool integration. The agent successfully used appropriate diagnostic tools 87.5% of the time and provided helpful responses to 94% of clinical questions. 𝗧𝗵𝗲 𝗯𝗶𝗴𝗴𝗲𝗿 𝗽𝗶𝗰𝘁𝘂𝗿𝗲: This isn't about replacing oncologists—it's about augmenting clinical reasoning with AI that can process multiple data streams simultaneously. The modular approach means individual tools can be updated, validated, and regulated independently. While challenges remain around data privacy and regulatory approval, this research points toward a future where AI agents serve as sophisticated clinical reasoning partners, helping doctors navigate the increasing complexity of modern medicine. https://lnkd.in/e52xBZj9 #AIinHealthcare #PrecisionOncology #ClinicalAI #DigitalHealth #MachineLearning #Oncology

We’re excited to share research from our health team at Microsoft AI: a proof-of-concept showing that AI can master medicine’s most intricate diagnostic challenges by following the same step-by-step reasoning expert physicians use. There's more detail in our pre print paper & blog Paper-> https://lnkd.in/egDiNsqR Blog-> https://lnkd.in/esGFhSeB Sharing what I'm most excited about from this work. 1. Benchmarks Traditional medical benchmarks like the USMLEs condense clinical cases into neat multiple-choice questions—far from the real clinical workflow. We’ve approached things in a different way: Sequential Diagnosis Benchmark (SDBench) deconstructs 304 of the most diagnostically complex and demanding cases in medicine published in the New England Journal of Medicine. SD Bench requires models—and physician—to begin with an initial presentation, ask follow-up questions, order tests, and converge on the confirmed diagnosis—just as in routine clinical practice. You can see how this works in a video with Xiao Liu on our blog. 2. Performance With this new benchmark we tested a suite of the best known generative AI models against the 304 NEJM cases with impressive out-the-box performance. Beyond this we developed the Microsoft AI Diagnostic Orchestrator (MAI-DxO). By emulating a virtual panel of physicians with diverse thinking styles, MAI-DxO boosts raw model accuracy and solves a remarkable 85.5% of NEJM cases. For comparison we evaluated 21 practicing UK/US physicians and on the same tasks, these experts achieved a mean accuracy of 20%. 3. Costs One of our concerns was that AI would default to ordering every investigation to arrive at the correct diagnosis. So we set the system up such that each requested investigation also incurred a cost. This allowed us to evaluate performance against both diagnostic accuracy and resource expenditure. As MAI-DxO is configurable it is seen to operate along a Pareto frontier of accuracy versus resource use. What’s next: While for now exciting research, we believe this kind of superhuman clinical reasoning will in future reshape medicine. A particular focus for our group is on consumer health. Today, Bing and Copilot answer over 50 million health queries daily—from a first-time knee-pain search to finding a late-night pharmacy. We’re committed to bringing rigorous and reliable AI support into these journeys, backed by clinical evidence and robust commitments to quality, safety and trust. A huge shout-out to everyone on our new team who contributed, our partners across Microsoft, and particularly to Mustafa Suleyman for his vision. He saw the opportunity for AI to improve healthcare more than a decade ago and it now feels like this is the right time to deliver on the opportunity. Harsha Nori Mayank Daswani Christopher Kelly Scott Lundberg Marco Túlio Ribeiro Marc Wilson Xiao Liu Viknesh Sounderajah Bay Gross Peter Hames Eric Horvitz Charlotte Cooper Simpson, PhD

The headline that caught my eye this week was "Microsoft Says Its New AI System Diagnosed Patients 4 Times More Accurately Than Human Doctors." Here’s my take: This new study is interesting because Microsoft's MAI Diagnostic Orchestrator arranged multiple AI models in a "chain-of-debate style" that mimics how human doctor collaborate. The AI achieved 80 percent accuracy compared to doctors' 20 percent in their test. What's particularly compelling is the methodology. Rather than replacing the diagnostic process, Microsoft replicated it: analyzing symptoms, ordering tests, and conducting further analysis until a diagnosis was reached. And as I have long said, better care need not be more expensive care. Consistent with this, the Microsoft system reduced costs by 20 percent by selecting more efficient diagnostic pathways. While there were limitations — the doctors in this study weren't allowed to use additional tools that they'd usually rely on in practice — the results are still quite encouraging. As Mustafa Suleyman, CEO of Microsoft's AI division, puts it, the "orchestration mechanism" of multiple AI agents working together represents a promising path forward. The potential for improving quality and reducing cost in health care is massive, and it's exciting to see how quickly the technology is advancing that could help us do that. https://lnkd.in/edbhneMz

This paper has been getting popular in healthcare circles, and I wanted to share my quick take. First, super exciting to see Microsoft validating an approach we've also taken. We recently launched our own sequential diagnosis module, designed to reason like a clinician. It asks the right questions, builds a dynamic differential diagnosis, explains its thinking at every step, and generates a tailored management plan. Early feedback from doctors has been incredibly positive, and it’s only getting better with use. Second, here's what Microsoft found: The core insight Most AI benchmarks in medicine are still too static. Real doctors diagnose iteratively, step by step, refining hypotheses with each clue. This paper introduces a realistic benchmark and an AI system that mirrors this process… and outperforms clinicians. What they built 1. Sequential Diagnosis Benchmark (SDBench) - Based on 304 real-world NEJM CPC cases - Agents (AI or human) start with a short abstract - Must ask questions or order tests iteratively - Every action incurs a cost—just like real life - Goal: Reach an accurate diagnosis efficiently 2. MAI Diagnostic Orchestrator (MAI-DxO) A smart framework simulating a virtual panel of 5 “doctors”: 2.1 Dr. Hypothesis → manages the differential 2.2 Dr. Test-Chooser → picks high-yield tests 2.3 Dr. Challenger → counters bias 2.4 Dr. Stewardship → keeps it cost-effective 2.5 Dr. Checklist → quality control Works across LLMs (OpenAI, Claude, Gemini, etc.) Why this matters - Validates the power of structured, stepwise reasoning in AI - Proves orchestration beats raw model output - Shows real potential for safer, cheaper, more accessible diagnosis, especially where clinical expertise is scarce --- We’ve just released a closed beta for our own diagnostics module. If you're a clinician and want to try it, DM me. Happy to give you early access!