LLM Solutions for Automated System Failure Remediation

Most LLM agents in production never get better at their job. They serve thousands of users, encounter new failure modes daily, and just... stay the same. Updating them means retraining, which means downtime, which means disrupted service. MetaClaw, a new framework from the AIMing Lab, breaks that cycle. The core idea: let the agent evolve continuously through two mechanisms that reinforce each other. When the agent fails a task, an LLM evolver analyzes the failure trajectory and synthesizes a new reusable skill - immediate improvement, zero downtime. Then, during user-inactive windows detected by a scheduling system that monitors inactivity and calendar data, the framework triggers gradient-based policy updates via cloud LoRA fine-tuning and reinforcement learning with a process reward model. The better policy generates better trajectories. Better trajectories produce better skills. Better skills feed back into policy optimization. A flywheel, not a one-shot fix. On their benchmarks, skill-driven adaptation alone improved accuracy by up to 32% relative. The full pipeline pushed Kimi-K2.5 from 21.4% to 40.6% accuracy and lifted composite robustness by 18.3% - all while running on a proxy-based architecture that requires no local GPUs. For anyone running LLM agents in production across shifting workloads, the pattern here is worth studying: separate fast adaptation from slow optimization, and let each feed the other. ↓ 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

Agents don't just fail at the semantic layer. They fail at the infrastructure layer too — and that's the part many teams aren't prepared for. Out-of-memory errors. Spot instances preempted mid-run. Network timeouts that wipe out precious context. These aren't edge cases. They're the norm at scale. The insight I've been hammering on lately: 🏗️ Infrastructure as context. When an agent hits an OOM error, that error is signal. If you surface it correctly — catching it as a structured exception, passing it back into the agent loop — the agent can respond to it. It can request more memory, adjust its resource config, and retry. All without human intervention. This is what we mean by self-healing. Not magic. Not perfect. Just: give the agent the right context about why it failed, and it can often dig itself out. The three primitives that make this possible: → 📋 Replay logs — granular per-step checkpoints so a crashed agent picks up exactly where it left off, not from scratch → 🎒 Global caching — shared across runs, so you're not re-paying for identical tool calls or LLM API hits → ✍️ Intermediate state persistence — serialized automatically, so the agent's memory survives infra failures And to top it all off, we get infrastructure-level self-healing by giving agents a runtime that allows them to catch OOM errors in the outer loop, adjust the resources of its subagents and tool call processes, and running them in secure sandboxes with network and IO restrictions. We put all of this into practice with a customer running 250k software products through a deep-research agent pipeline — 200 steps per run, 100 LLM calls per product. They used spot instances heavily. Checkpoint recovery turned preemptions into a non-issue. Failure recovery time dropped 50%. The lesson: don't aim for failure-proof. Aim for cheap failures and fast recovery. Observability is necessary. Durability is what actually lets agents recover. Learn more about these ideas here 👇 https://lnkd.in/epS6dxwJ #MLOps #AgentEngineering #LLM #MLInfrastructure #AIInfrastructure #Flyte

When’s the last time your LLM provider went down? 🤔 Well, OpenAI had a partial outage late Monday night and Anthropic had elevated error rates as recently as yesterday. At Assembled, our customers don’t tolerate downtime like this, so we built automated fallbacks, making our system incredibly resilient to LLM provider outages without our customers needing to know. We detect failures in real time, instantly reroute requests to secondary or on-prem models, and keep your support workflows humming—even when the big providers falter. 🚀 We wrote a blog post on the nuts and bolts of how we: - Monitor health across multiple LLM providers - Seamlessly fail over to backup models or local inference - Maintain sub-second response times and 99.99% uptime Check it out here 👉 https://lnkd.in/g3s7y4fc

Machine Learning models are critical not only for many customer-facing products like recommendation algorithms but also very important for unlocking value in backend tasks to enable efficient operations and save business costs. This blog, written by the machine learning engineering team at Netflix, shares the team's approaches to automatically leverage machine learning to remediate failed jobs without human intervention. -- Problem: At Netflix, millions of workflow jobs run daily in their big data platform. Although failed jobs represent a small portion, they still incur significant costs given the large base. The team currently has a rule-based approach to categorize error messages. However, for memory configuration errors, engineers still need to remediate the jobs manually due to their intrinsic complexity. -- Solution: The team uses the existing rule-based classifier as the first pass to classify errors, and then develops a new Machine Learning service as the second pass to provide recommendations for memory configuration errors. This Machine Learning system has two components: one is a prediction model that jointly estimates the probability of retry success and the retry cost, and the other is an optimizer that recommends a Spark configuration to minimize the linear combination of retry failure probability and cost. -- Result: Such a solution demonstrated big success, with more than 56% of all memory configuration errors being remediated and successfully retried without human intervention. This also decreased the compute cost by about 50% because the new configurations make the retry successful or disable unnecessary retries. Whenever there is a place with inefficiencies, it's helpful to think about a better solution. Machine learning is the way to introduce intelligence into such solutions, and this blog serves as a nice reference for those interested in leveraging machine learning to improve their operational efficiency. #machinelearning #datascience #operation #efficiency #optimization #costsaving #snacksweeklyondatascience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gj6aPBBY    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gRcaifKR 

Today, I am pleased to announce the release of our paper "Solver-in-the-Loop: MDP-Based Benchmarks for Self-Correction and Behavioral Rationality in Operations Research." This research, done with my PhD student Ruicheng Ao and my former Postdoc Xinshang Wang, focuses on using LLMs to automate OR model debugging. The challenge: how to debug an OR model when the solver returns an INFEASIBLE message. Using LLMs, we are able to automatically query the IIS (Irreducible Infeasible Subsystem), diagnose the root cause, repair the formulation, re-run the solver, and iterate until the model is corrected. So, what used to take hours and days can now take seconds. Our training approach combines supervised fine-tuning (SFT) on expert trajectories with reinforcement learning using GRPO, where the solver provides verifiable reward signals for outcome, diagnostic accuracy, and efficiency. The figures below show performance for our Trained Local Models and for API baselines (with no training). Here's what each metric means: - RR@5: Recovery Rate within 5 steps—the percentage of problems successfully fixed. - DA: Diagnostic Accuracy—whether the model correctly identified the actual problematic constraint. - Steps: Average number of iterations needed for recovery. Testing with trained Qwen, Llama, and DeepSeek models shows Recovery Rate of 90%+, DA of 60%+, and an average of ~2 steps for complete recovery. The conclusions are clear: ·      Domain training beats scale (8B > frontier APIs). ·      Composite reward function prevents trial-and-error. ·      LLM & Solver-in-the-loop enable quick repair of OR formulations. For more detailed information, with additional insights on classical OR models, please review the paper at https://lnkd.in/ekWdD_A2