How AI Models Are Trained for Reasoning Tasks

If you’re an AI engineer trying to understand how reasoning actually works inside LLMs, this will help you connect the dots. Most large language models can generate. But reasoning models can decide. Traditional LLMs followed a straight line: Input → Predict → Output. No self-checking, no branching, no exploration. Reasoning models introduced structure, a way for models to explore multiple paths, score their own reasoning, and refine their answers. We started with Chain-of-Thought (CoT) reasoning, then extended to Tree-of-Thought (ToT) for branching, and now to Graph-based reasoning, where models connect, merge, or revisit partial thoughts before concluding. This evolution changes how LLMs solve problems. Instead of guessing the next token, they learn to search the reasoning space- exploring alternatives, evaluating confidence, and adapting dynamically. Different reasoning topologies serve different goals: • Chains for simple sequential reasoning • Trees for exploring multiple hypotheses • Graphs for revising and merging partial solutions Modern architectures (like OpenAI’s o-series reasoning models, Anthropic’s Claude reasoning stack, DeepSeek R series and DeepMind’s AlphaReasoning experiments) use this idea under the hood. They don’t just generate answers, they navigate reasoning trajectories, using adaptive depth-first or breadth-first exploration, depending on task uncertainty. Why this matters? • It reduces hallucinations by verifying intermediate steps • It improves interpretability since we can visualize reasoning paths • It boosts reliability for complex tasks like planning, coding, or tool orchestration The next phase of LLM development won’t be about more parameters, it’ll be about better reasoning architectures: topologies that can branch, score, and self-correct. I’ll be doing a deep dive on reasoning models soon on my Substack- exploring architectures, training approaches, and practical applications for engineers. If you haven’t subscribed yet, make sure you do: https://lnkd.in/dpBNr6Jg ♻️ Share this with your network 🔔 Follow along for more data science & AI insights

Breakthrough in AI Reasoning: How Tree Search is Revolutionizing Vision-Language Models Researchers from the Chinese Academy of Sciences and Alibaba Cloud have just unveiled a game-changing approach to making Large Vision-Language Models significantly smarter at visual reasoning tasks. The Challenge: Current multimodal AI systems struggle with two critical issues - they often generate responses that don't align with user instructions, and existing knowledge retrieval methods provide rigid, formulaic examples that fail to capture underlying logical patterns. The Innovation - RCTS Framework: Reasoning Context Generation: The system automatically enriches knowledge bases by generating detailed reasoning contexts for visual question-answer pairs using a self-consistent evaluation mechanism. Instead of simple Q&A formats, it creates comprehensive step-by-step thought processes that reveal the logical patterns behind correct answers. Hybrid Multimodal Retrieval: Under the hood, the framework employs dual encoders - separate text and vision encoders that create unified embeddings. This allows the system to retrieve relevant examples based on both visual content and textual similarity, maintaining the integrity of multimodal information throughout the process. Monte Carlo Tree Search with Heuristic Rewards: Here's where it gets fascinating - the system treats example selection as a sequential decision-making problem. It uses tree search algorithms to explore different combinations of retrieved examples, evaluating each path using two sophisticated reward mechanisms: - Self-consistency rewards that verify if the reasoning context generates consistent answers - Mutual heuristic rewards that assess whether examples positively contribute to solving other related questions The Technical Breakthrough: The framework transforms the traditional "retrieve and use" approach into an intelligent "retrieve, re-rank, and optimize" pipeline. The tree search systematically explores combinations of contextual examples, backpropagating reward values to identify the most beneficial sequences for in-context learning. Results That Matter: Testing across multiple challenging datasets including ScienceQA, MMMU, and MathV showed remarkable improvements - up to 4.2% performance gains over existing methods, with the system achieving 91.44% accuracy on ScienceQA using Qwen2-VL models. This research represents a fundamental shift from basic retrieval-augmented generation to sophisticated contextual reasoning optimization. The implications for AI applications in education, scientific analysis, and complex visual understanding are profound. The complete methodology is training-free and adaptable across domains, making it immediately applicable to existing vision-language systems. We're witnessing the evolution from AI that simply "knows" to AI that truly "understands" through structured reasoning.

One of the biggest challenges in deploying LLMs in real workflows is reasoning. Not answering trivia, but actual structured thinking: planning, breaking problems into steps, updating based on intermediate results. LLMs sometimes produce convincing responses that collapse when you inspect the logic. This shows up in math, code generation, question answering, and increasingly, in agentic use cases. Projects fail when models cannot reason reliably over multiple steps. This failure happens because most LLMs are trained to predict the next token, not to reason through a process. They pick the most likely next word, based on patterns they have seen. They are not optimizing for whether each step is logically valid or whether the final result is correct. Even fine-tuned models often reproduce patterns without deeply validating the steps in between. That said, agents have solved this problem to a great extent. A new paper from Stanford and Ceramic AI proposes a surprisingly effective solution: 𝗧𝗵𝗶𝗻𝗸, 𝗣𝗿𝘂𝗻𝗲, 𝗧𝗿𝗮𝗶𝗻. The model generates multiple reasoning paths. Only the ones that lead to correct answers are kept. Then the model is fine-tuned on those filtered traces. This loop is repeated. Over time, the model improves its ability to generate correct, logically coherent solutions, entirely from its own outputs. There is no need for external labels, teacher models, or human ranking. This is a very effective method because pruning to correct final answers is a simple form of reward. Mathematically, the paper shows that this kind of filtered fine-tuning is equivalent to reinforcement learning with a binary reward signal. It avoids the complexity and instability of full RL pipelines but delivers the same benefits. The model learns to prefer better thinking, not just better phrasing. The results are significant!! Gemma-2B improves from 41.9 to 57.6 percent accuracy on GSM8K. Gemma-9B reaches 82 percent, outperforming LLaMA-70B. Even LLaMA-70B improves from 78 to 91 percent, surpassing GPT-4o. 𝗔𝗹𝗹 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗻𝗲𝘄 𝗱𝗮𝘁𝗮 𝗼𝗿 𝗲𝘅𝘁𝗲𝗿𝗻𝗮𝗹 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸. 𝗝𝘂𝘀𝘁 𝘀𝗺𝗮𝗿𝘁 𝗳𝗶𝗹𝘁𝗲𝗿𝗶𝗻𝗴 𝗮𝗻𝗱 𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴. If you’re working on reasoning-heavy tasks, you can try this with open models and modest compute. The steps are simple: 1. Generate several reasoning paths per example 2. Keep only the ones that lead to a correct final answer 3. Fine-tune the model on those filtered examples 4. Repeat with the new model to improve further It works best on tasks with verifiable outcomes like math, code, or structured QA. You need a base model that can already reason somewhat, and a way to check correctness. But you do not need GPT-4, and you do not need human labels. This method pushes us toward a future where models do not just produce good outputs, but learn to produce better reasoning. It is simple, scalable, and grounded in solid learning theory. And it is something teams can start applying today.

I’ve been exploring a question that has sparked endless debates in the AI community: how do we actually teach LLMs to reason, without leaning on endless human labels or black-box pipelines? My latest blog post dives into this problem head-on, walking through how I built a reasoning LLM completely from scratch - 100% local, no labeled data, no human feedback loops using Group Relative Policy Optimization (GRPO). For those unfamiliar, GRPO is a reinforcement fine-tuning method that replaces expensive human preference labels with deterministic reward functions. That makes it especially powerful for domains like math and logic, where correctness is inherently verifiable. I shared the full pipeline: loading a base open-weight model (Qwen3-4B-Base in my case), applying LoRA for parameter-efficient fine-tuning with UnslothAI, creating a structured reasoning dataset from the Open R1 Math corpus, and then defining simple but effective reward functions that check answers, numbers, and formats. With HuggingFace’s TRL library handling the GRPO trainer, the loop becomes elegant: generate → score → reinforce. The results speak for themselves—what started as a generic base model quickly transformed into a reasoning-focused specialist, capable of producing step-by-step solutions with consistency and accuracy. I also reflect on when to choose reinforcement fine-tuning over supervised fine-tuning, why reward shaping often matters more than raw model size, and how GRPO opens the door for label-free reasoning across other domains like code, logic, or theorem proving. For me, the biggest takeaway was how empowering it felt to run the entire workflow locally, with complete transparency and reproducibility. If you’re curious about the future of reasoning LLMs beyond the hype, this piece offers a grounded, hands-on narrative. https://lnkd.in/g2BdmPJr

The recently released open-source model, DeepSeek-R1, is comparable to OpenAI O1 in multiple benchmarks. This O1-level complex reasoning ability, as mentioned in the paper, is due to "emergent long Chain-of-Thought (CoT) from large-scale reinforcement learning." This might sound alienating, but it should feel rather familiar after a brief reflection. Let's understand by contrasting the training recipes of standard LLMs vs reasoning "thinking" LLMs. Simplified recipe of training SOTA LLMs (DeepSeek-V3, Llama, Qwen, ...): 📌 Pre-training: Starting from random weights, train on ~10-20 Trillion token data 📌 Post-training: ✔️ Supervised fine-tune (SFT) with millions of (prompt, response) pairs ✔️ Perform preference learning (RLHF/PPO/DPO) with millions of human preferences or labels from reward models Before going over the recipe for thinking reasoning models, let's consider why we are building these new types of models. For some hard problems, like Olympiad-level math problems, rather than giving one answer, giving multiple answers or trying different approaches proved to be helpful. Even better if there is an internal thought process (long CoT) that tries an approach, regularly introspects itself and changes the approach, or explores new strategies if the previous one is not satisfactory. See the contrast here: standard CoT breaks down the problem into multiple steps but tries only one approach. This came to be known as inference-time scaling (at least, one way of doing this scaling). As in, we generate more tokens based on the complexity of the problem (could be tens of thousands of tokens) while inferring from a model. These extra generated tokens for exploration could be treated as ‘internal thinking’ tokens. (See an LLM having an 'aha moment' in one of the images attached). Usually, these thinking models are trained in math, coding, or logic-heavy domains. These tasks/questions have a correct answer and the correctness acts as a reward/preference for RL. Simplified recipe of training reasoning LLMs: (DeepSeek-R1, possibly OpenAI O1/O3) 1️⃣ After pre-training, perform SFT on very few warm-up data rather than millions of (prompt, response) pairs. 2️⃣ Perform RL for many iterations with correctness as the reward model. This is where long CoT emerged just from trial and error of RL training. 3️⃣ Perform SFT on large curated data for broad tasks. 4️⃣ Perform RL again with correctness as the reward model for math/code tasks. And LLM-as-a-judge type generative reward model for other tasks. It is important to note that emergent long CoT, internal thinking, with large-scale RL was only possible in domains where we know the exact answers. It is still a research exploration acquiring complex reasoning abilities outside of these domains.

Interestingly, it's still hard to tell when AI models gain better reasoning – during pre-training, mid-training, or RL. Here's a very useful study from Carnegie Mellon University that digs deep into this issue. Let me unpack it as a quick guide for you ⟶ The main question is: Does reinforcement learning (RL) actually add new reasoning abilities, or just refine what models already know? The researchers found the answer through a fully controlled experimental setup with synthetic tasks, where reasoning is split into clear, checkable steps and the training data can be precisely controlled and modified. They evaluate models in 2 ways: • Extrapolation (depth): Can the model solve harder problems made by combining more steps than it saw before? • Context transfer (breadth): Can the model apply the same reasoning in new surface context? ➡️ In fact, pre-training, mid-training, and RL, each play distinct roles. And here is where each of them actually helps: 🔹 1. RL truly improves reasoning only in specific conditions: It works when pre-training hasn’t already maxed out the model, and when RL focuses on tasks that are just at the edge of what the model can handle: - If tasks are already familiar, RL adds little. - If they’re too unfamiliar, RL also fails. This explains why RL looks ineffective on common tasks like math or coding (already well learned), but looks powerful on new synthetic tasks where the model still has room to grow. ❗So, to get the most out of RL: • Train on problems the model usually fails at first, but can solve sometimes with enough attempts. • As the model improves, keep updating the training set so it always targets the model’s current limits. 🔹 2. Generalizing across contexts needs some pre-training first: Models can’t jump to totally new contexts with RL alone. But just a tiny amount of pre-training exposure (~1%) is enough. After that, RL can reliably transfer the reasoning to new settings. Plus, more exposure helps creativity on harder tasks. 🔹 3. Mid-training matters a lot: Adding this structured training phase between pre-training and RL gives bigger gains – more so than using RL alone, even with the same compute budget. But what if you have a fixed amount of compute: how much should you spend on mid-training, and how much on RL? • In easier or nearby tasks, models do best when most compute goes to mid-training. This gives stable reasoning skills. • In very hard or far-out tasks, more compute should go to RL to explore new behavior, as long as there is some mid-training first. 🔹 4. Don't underestimate process-aware rewards: Rewarding that corrects intermediate steps (not just final answers) reduces reward hacking and leads to more faithful reasoning, so: • Combine sparse “answer-correct” rewards with dense feedback on reasoning steps. • As long as the step-level supervision is high quality, this consistently improves performance.

I just finished reading three recent papers that every Agentic AI builder should read. As we push toward truly autonomous, reasoning-capable agents, these papers offer essential insights, not just new techniques, but new assumptions about how agents should think, remember, and improve. 1. MEM1: Learning to Synergize Memory and Reasoning Link: https://bit.ly/4lo35qJ Trains agents to consolidate memory and reasoning into a single learned internal state, updated step-by-step via reinforcement learning. The context doesn’t grow, the model learns to retain only what matters. Constant memory use, faster inference, and superior long-horizon reasoning. MEM1-7B outperforms models twice its size by learning what to forget. 2. ToT-Critic: Not All Thoughts Are Worth Sharing Link: https://bit.ly/3TEgMWC A value function over thoughts. Instead of assuming all intermediate reasoning steps are useful, ToT-Critic scores and filters them, enabling agents to self-prune low-quality or misleading reasoning in real time. Higher accuracy, fewer steps, and compatibility with existing agents (Tree-of-Thoughts, scratchpad, CoT). A direct upgrade path for LLM agent pipelines. 3. PAM: Prompt-Centric Augmented Memory Link: https://bit.ly/3TAOZq3 Stores and retrieves full reasoning traces from past successful tasks. Injects them into new prompts via embedding-based retrieval. No fine-tuning, no growing context, just useful memories reused. Enables reasoning, reuse, and generalization with minimal engineering. Lightweight and compatible with closed models like GPT-4 and Claude. Together, these papers offer a blueprint for the next phase of agent development: - Don’t just chain thoughts; score them. - Don’t just store everything; learn what to remember. - Don’t always reason from scratch; reuse success. If you're building agents today, the shift is clear: move from linear pipelines to adaptive, memory-efficient loops. Introduce a thought-level value filter (like ToT-Critic) into your reasoning agents. Replace naive context accumulation with learned memory state (a la MEM1). Storing and retrieving good trajectories, prompt-first memory (PAM) is easier than it sounds. Agents shouldn’t just think, they should think better over time.

Can you tell reasoning models to think longer before producing an answer? Apparently yes, and there are multiple ways presented by CMU and Stanford! I recently explored two papers tackling the same challenge in AI reasoning from completely different angles: how to control and optimize the "thinking time" of large language models to balance computational resources and performance. The Problem Both Papers Address: Modern reasoning LLMs like OpenAI's o1 and DeepSeek's r1 can generate extended chain-of-thought sequences to solve complex problems. However, the uncontrolled nature of this reasoning leads to inefficiencies—sometimes wasting resources with lengthy reasoning, other times stopping too early on difficult problems. Two Different Solutions: Reinforcement Learning vs. Minimal Data L1: Reinforcement Learning for Precise Control Carnegie Mellon researchers Aggarwal and Welleck's "L1: Controlling How Long A Reasoning Model Thinks" takes a sophisticated reinforcement learning approach: >> Length Controlled Policy Optimization (LCPO): Trains models to generate correct answers while adhering to user-specified length constraints >> Two variants: L1-Exact (produces reasoning of exactly the requested length) and L1-Max (doesn't exceed the requested length) Results: Outperforms previous methods by up to 100% relative and 20% absolute performance on math tasks S1: Minimal Data and Test-Time Control Muennighoff et al.'s "s1: Simple test-time scaling" takes a remarkably lightweight approach: >>Minimal dataset: Just 1,000 carefully curated high-quality examples (s1K dataset) >>Budget forcing: Simple test-time mechanism to control reasoning length Efficiency: Training takes only 26 minutes on 16 H100 GPUs >>Results: Outperforms OpenAI's o1-preview on competition math by up to 27% Despite their different approaches, both methods demonstrate: >> Test-time scaling: Performance improves with allocated computation >> Strong generalization: Both methods work across various reasoning tasks >> Efficiency improvements: Both outperform previous approaches with fewer resources Perhaps most impressively, L1's 1.5B parameter model matches GPT-4o at equal reasoning lengths, while S1 outperforms o1-preview while using just 1/800th the training data of comparable models. Together, they suggest that reasoning in LLMs may be more accessible than previously thought—whether through careful optimization of smaller models or minimal fine-tuning of larger ones. Both papers are open-source with code available: L1: https://lnkd.in/gj6Pw4ki S1: https://lnkd.in/gPJsZ2iC

𝗧𝗵𝗲 𝗳𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗔𝗜 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗺𝗶𝗴𝗵𝘁 𝗹𝗼𝗼𝗸 𝗹𝗲𝘀𝘀 𝗹𝗶𝗸𝗲 𝗰𝗼𝗹𝗱 𝗰𝗮𝗹𝗰𝘂𝗹𝗮𝘁𝗶𝗼𝗻 𝗮𝗻𝗱 𝗺𝗼𝗿𝗲 𝗹𝗶𝗸𝗲 𝗶𝗻𝘁𝗲𝗿𝗻𝗮𝗹 𝗰𝗵𝗮𝗼𝘀. Not in a bad way. A new paper from Google and the University of Chicago shows that top-performing reasoning models like DeepSeek-R1 don’t just think harder, they simulate something more profound: 𝘈 𝘤𝘰𝘮𝘮𝘪𝘵𝘵𝘦𝘦 𝘰𝘧 𝘥𝘪𝘴𝘢𝘨𝘳𝘦𝘦𝘪𝘯𝘨 𝘦𝘹𝘱𝘦𝘳𝘵𝘴 𝘪𝘯𝘴𝘪𝘥𝘦 𝘵𝘩𝘦𝘮𝘴𝘦𝘭𝘷𝘦𝘴. Here’s what’s wild: When solving a problem, these models: - Question themselves ~7 times - Shift perspectives ~3.5 times - Engage in internal conflict ~3+ times 𝗮𝗹𝗹 𝗽𝗲𝗿 𝗽𝗿𝗼𝗯𝗹𝗲𝗺. Researchers discovered a neural "switch" they call the conversational surprise marker. Turning it up doubled performance, from 27% to 55% on hard reasoning tasks. Even when trained without prompts for dialogue or debate, models spontaneously evolved internal argumentation, because that’s what works. This behavior, what the authors call a "society of thought", mirrors how we humans reason best: by weighing diverse perspectives, entertaining doubt, and challenging our assumptions. 𝗧𝗵𝗲 𝗶𝗺𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻? Scaling reasoning isn’t just about bigger models. It’s about structured internal diversity, simulated voices with distinct expertise, personalities, and even emotional roles. For AI teams, this opens up new design frontiers: - Reward behaviors that mirror high-performing teams. - Think in terms of internal collaboration, not monologues. - Embrace the messiness of thought. 𝗧𝗵𝗲 𝘀𝗺𝗮𝗿𝘁𝗲𝘀𝘁 𝗺𝗼𝗱𝗲𝗹𝘀 𝗮𝗿𝗲𝗻'𝘁 𝗺𝗼𝗻𝗼𝗹𝗶𝘁𝗵𝗶𝗰 𝘁𝗵𝗶𝗻𝗸𝗲𝗿𝘀. 𝗧𝗵𝗲𝘆'𝗿𝗲 𝗺𝗶𝗻𝗶-𝘀𝗼𝗰𝗶𝗲𝘁𝗶𝗲𝘀 𝗶𝗻 𝗱𝗶𝗮𝗹𝗼𝗴𝘂𝗲. #AI #MachineLearning #Reasoning #LLMs #CognitiveScience #ArtificialIntelligence

🔍 𝗡𝗲𝘄 𝗥𝗲𝘀𝗲𝗮𝗿𝗰𝗵 𝗼𝗻 𝗖𝗵𝗮𝗶𝗻-𝗼𝗳-𝗧𝗵𝗼𝘂𝗴𝗵𝘁 (𝗖𝗼𝗧) 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗶𝗻 𝗔𝗜 A recent study, "𝘈 𝘛𝘩𝘦𝘰𝘳𝘦𝘵𝘪𝘤𝘢𝘭 𝘜𝘯𝘥𝘦𝘳𝘴𝘵𝘢𝘯𝘥𝘪𝘯𝘨 𝘰𝘧 𝘊𝘩𝘢𝘪𝘯-𝘰𝘧-𝘛𝘩𝘰𝘶𝘨𝘩𝘵: 𝘊𝘰𝘩𝘦𝘳𝘦𝘯𝘵 𝘙𝘦𝘢𝘴𝘰𝘯𝘪𝘯𝘨 𝘢𝘯𝘥 𝘌𝘳𝘳𝘰𝘳-𝘈𝘸𝘢𝘳𝘦 𝘋𝘦𝘮𝘰𝘯𝘴𝘵𝘳𝘢𝘵𝘪𝘰𝘯," explores how Chain-of-Thought can be more accurate when intermediate steps are connected rather than isolated. 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 𝗦𝘁𝗮𝘁𝗲𝗺𝗲𝗻𝘁:  Few-shot Chain-of-Thought (CoT) prompting has improved the performance of large language models. However, investigations to understand CoT have generally isolated CoT into separated in-context learning steps (what the authors call Stepwise In-Context Learning (ICL)). This overlooks how language models work since all previous context is included when predicting the next tokens and generating new content. The researchers call their work Coherent CoT. Moreover, models can self-correct intermediate errors when they maintain full context. The results indicate that the transformer architecture is more sensitive to errors in intermediate reasoning steps than the final outcome. 𝗞𝗲𝘆 𝗙𝗶𝗻𝗱𝗶𝗻𝗴𝘀: • Chain-of-Thought (CoT) works better when reasoning steps stay connected vs isolated.  • The authors introduce "Coherent CoT," which explicitly integrates earlier reasoning steps for improved information propagation. Instead of propagating information between prompts, Coherent CoT colocates this information more compactly.  • AI models reason better when they keep all previous context, allowing for self-correction of intermediate mistakes. • By showing the model examples of both correct and incorrect reasoning paths, accuracy improves, especially in tracking and disambiguation tasks. • Using this connected reasoning approach, models like GPT-3.5, GPT-4, Gemini Pro, and DeepSeek showed 5-6% accuracy gains. 𝗪𝗵𝘆 𝗜𝘁 𝗠𝗮𝘁𝘁𝗲𝗿𝘀:  This “Coherent CoT” approach shows that models are more effective and reliable when they don’t lose sight of earlier steps. Error-aware examples help models understand mistakes, leading to better and more consistent results in complex and unfamiliar tasks.