Large Language Models Insights

Large Language Models (LLMs) are powerful, but how we 𝗮𝘂𝗴𝗺𝗲𝗻𝘁, 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲, 𝗮𝗻𝗱 𝗼𝗿𝗰𝗵𝗲𝘀𝘁𝗿𝗮𝘁𝗲 them truly defines their impact. Here's a simple yet powerful breakdown of how AI systems are evolving: 𝟭. 𝗟𝗟𝗠 (𝗕𝗮𝘀𝗶𝗰 𝗣𝗿𝗼𝗺𝗽𝘁 → 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲)   ↳ This is where it all started. You give a prompt, and the model predicts the next tokens. It's useful — but limited. No memory. No tools. Just raw prediction. 𝟮. 𝗥𝗔𝗚 (𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹-𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗲𝗱 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻)   ↳ A significant leap forward. Instead of relying only on the LLM’s training, we 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗲 𝗿𝗲𝗹𝗲𝘃𝗮𝗻𝘁 𝗰𝗼𝗻𝘁𝗲𝘅𝘁 𝗳𝗿𝗼𝗺 𝗲𝘅𝘁𝗲𝗿𝗻𝗮𝗹 𝘀𝗼𝘂𝗿𝗰𝗲𝘀 (like vector databases). The model then crafts a much more relevant, grounded response.   This is the backbone of many current AI search and chatbot applications. 𝟯. 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗟𝗟𝗠𝘀 (𝗔𝘂𝘁𝗼𝗻𝗼𝗺𝗼𝘂𝘀 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 + 𝗧𝗼𝗼𝗹 𝗨𝘀𝗲)   ↳ Now we’re entering a new era. Agent-based systems don’t just answer — they think, plan, retrieve, loop, and act.   They: - Use 𝘁𝗼𝗼𝗹𝘀 (APIs, search, code) - Access 𝗺𝗲𝗺𝗼𝗿𝘆 - Apply 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗰𝗵𝗮𝗶𝗻𝘀 - And most importantly, 𝗱𝗲𝗰𝗶𝗱𝗲 𝘄𝗵𝗮𝘁 𝘁𝗼 𝗱𝗼 𝗻𝗲𝘅𝘁 These architectures are foundational for building 𝗮𝘂𝘁𝗼𝗻𝗼𝗺𝗼𝘂𝘀 𝗔𝗜 𝗮𝘀𝘀𝗶𝘀𝘁𝗮𝗻𝘁𝘀, 𝗰𝗼𝗽𝗶𝗹𝗼𝘁𝘀, 𝗮𝗻𝗱 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗺𝗮𝗸𝗲𝗿𝘀. The future is not just about 𝘸𝘩𝘢𝘵 the model knows, but 𝘩𝘰𝘸 it operates. If you're building in this space — RAG and Agent architectures are where the real innovation is happening.

The 2025 Landscape of LLMs — Updated View of the Big Players in the Game of AI About 18 months ago, I shared my first version of the Large Language Model landscape, and a lot has changed since then. The space has evolved rapidly, but at the same time, we’re starting to see clear patterns emerge. This updated view focuses on the leading AI research labs, their latest models, and how those models can be accessed. It’s not meant to list every single LLM out there—but it does cover about 95% of what’s being used in real-world scenarios today. Here are some key insights: 🔹 No more clear front-runner: We’ve gone from “everyone chasing one leader” to a fairly even playing field. For most use cases, model differences are small and often not that relevant. 🔹 Model choice is the new normal: Customers now expect the ability to test, compare, and switch between models with ease. This shift is driving interest in evaluation frameworks and model routing tools. 🔹 Reasoning-first models are rising: Many providers are clearly moving toward models optimized for reasoning—fueling the surge of Agentic AI architectures. 🔹 Proprietary still leads, but just barely: Open-source and open-weight models are quickly closing the gap. 🔹 The U.S. is still ahead, but international competition is heating up—fast. 🔹 Cloud and APIs dominate: With few exceptions (hello Grok/XAI 👀), nearly every model is accessible via API across the major cloud platforms. 🔹 Serverless is the default: Most organizations prefer calling models via API over hosting or fine-tuning them—unless the use case is highly specialized. 🔹 Everyone else? Still less than 5% of the market. We’re entering a phase where model access, interoperability, and orchestration matter more than the model itself. And this landscape helps make sense of where we are and where we’re going. #LLMs #AI #MachineLearning #GenerativeAI #AgenticAI #OpenSource

Don't just blindly use LLMs, evaluate them to see if they fit into your criteria. Not all LLMs are created equal. Here’s how to measure whether they’re right for your use case👇 Evaluating LLMs is critical to assess their performance, reliability, and suitability for specific tasks. Without evaluation, it would be impossible to determine whether a model generates coherent, relevant, or factually correct outputs, particularly in applications like translation, summarization, or question-answering. Evaluation ensures models align with human expectations, avoid biases, and improve iteratively. Different metrics cater to distinct aspects of model performance:  Perplexity quantifies how well a model predicts a sequence (lower scores indicate better familiarity with the data), making it useful for gauging fluency. ROUGE-1 measures unigram (single-word) overlap between model outputs and references, ideal for tasks like summarization where content overlap matters. BLEU focuses on n-gram precision (e.g., exact phrase matches), commonly used in machine translation to assess accuracy. METEOR extends this by incorporating synonyms, paraphrases, and stemming, offering a more flexible semantic evaluation. Exact Match (EM) is the strictest metric, requiring verbatim alignment with the reference, often used in closed-domain tasks like factual QA where precision is paramount. Each metric reflects a trade-off: EM prioritizes literal correctness, while ROUGE and BLEU balance precision with recall. METEOR and Perplexity accommodate linguistic diversity, rewarding semantic coherence over exact replication. Choosing the right metric depends on the task—e.g., EM for factual accuracy in trivia, ROUGE for summarization breadth, and Perplexity for generative fluency. Collectively, these metrics provide a multifaceted view of LLM capabilities, enabling developers to refine models, mitigate errors, and align outputs with user needs. The table’s examples, such as EM scoring 0 for paraphrased answers, highlight how minor phrasing changes impact scores, underscoring the importance of context-aware metric selection. Know more about how to evaluate LLMs: https://lnkd.in/gfPBxrWc Here is my complete in-depth guide on evaluating LLMs: https://lnkd.in/gjWt9jRu Follow me on my YouTube channel so you don't miss any AI topic: https://lnkd.in/gMCpfMKh

AI agents are not yet safe for unsupervised use in enterprise environments The German Federal Office for Information Security (BSI) and France’s ANSSI have just released updated guidance on the secure integration of Large Language Models (LLMs). Their key message? Fully autonomous AI systems without human oversight are a security risk and should be avoided. As LLMs evolve into agentic systems capable of autonomous decision-making, the risks grow exponentially. From Prompt Injection attacks to unauthorized data access, the threats are real and increasingly sophisticated. The updated framework introduces Zero Trust principles tailored for LLMs: 1) No implicit trust: every interaction must be verified. 2) Strict authentication & least privilege access – even internal components must earn their permissions. 3) Continuous monitoring – not just outputs, but inputs must be validated and sanitized. 4) Sandboxing & session isolation – to prevent cross-session data leaks and persistent attacks. 5) Human-in-the-loop, i.e., critical decisions must remain under human control. Whether you're deploying chatbots, AI agents, or multimodal LLMs, this guidance is a must-read. It’s not just about compliance but about building trustworthy AI that respects privacy, integrity, and security. Bottom line: AI agents are not yet safe for unsupervised use in enterprise environments. If you're working with LLMs, it's time to rethink your architecture.

LLM memory is missing something fundamental. During pre-training, Llama 70B compresses the entire internet into 140GB of model weights. But just putting Steve Jobs’ Wikipedia page into the context window creates an 80GB key-value cache. If we want models that can efficiently reason over millions of tokens of context, we cannot simply dump everything into a context window. We need to continue training models at test-time, using long-context as training data to compress massive amounts of information directly into the model weights. Incredibly excited to share work led by my student Arnuv Tandon, in partnership with NVIDIA AI, that has been over a year in the making: End-to-End Test-Time Training for Long Context. As the title suggests, we continue training language models at test-time using the same next-token prediction objective as pre-training — allowing our model to scale with context length like full attention without maintaining a key and value for every token in the sequence. With linear complexity, our method is 2.7x faster than full attention at 128K tokens while achieving better performance. We believe test-time training is the key to unlocking a future with long-horizon agents, robots with human-like memory, and truly personal AI with your own model weights. Read the full paper: https://lnkd.in/g3f2BFcx Read the NVIDIA blog post: https://lnkd.in/gnWvS3Uk

2024 was a great year for building general-purpose LLMs, with some specialized fine-tuning for math and code. So far, 2025 seems to be the year of diverging into two key areas: (1) Reasoning models (focused on math and code), and (2) Agents (essentially LLM-based workflow automation). This year is going to be another eventful one for LLM research and development! I plan to write more about reasoning models soon. In the meantime, if you're looking for focused reading this weekend, I've re-compiled my take on the noteworthy LLM research papers of 2024. The topics include mixture-of-experts, new scaling laws for precision, scaling inference-time compute, and more. It's all packed into a PDF-friendly, 47-page article with a table of contents for easy navigation: https://lnkd.in/gFUT9cnk Topics: 1. January: Mixtral’s Mixture of Experts Approach  1.1 Understanding MoE models  1.2 The relevance of MoE models today 2. February: Weight-decomposed LoRA  2.2 LoRA Recap  2.2 From LoRA to DoRA  2.3 The future of LoRA and LoRA-like methods 3. March: Tips for Continually Pretraining LLMs  3.1 Simple techniques work  3.2 Will these simple techniques continue to work? 4. April: DPO or PPO for LLM alignment, or both?  4.1 RLHF-PPO and DPO: What Are They?  4.2 PPO Typically Outperforms DPO  4.3 How are PPO and DPO used today? 5. May: LoRA learns less and forgets less  5.1 LoRA learns less  5.2 LoRA forgets less  5.3 The LoRA trade-off  5.4 Future approaches to finetuning LLMs 6. June: The 15 Trillion Token FineWeb Dataset  6.1 Comparison to other datasets  6.2 Principled dataset development  6.3 The relevance of FineWeb today 7. July: The Llama 3 Herd of Models  7.1 Llama 3 architecture summary  7.2 Llama 3 training  7.3 Multimodal Llamas  7.4 Llama 3 impact and usage 8. August: Improving LLMs by scaling inference-time compute  8.1 Improve outputs by using more test-time computation  8.2 Optimizing test-time computation techniques  8.3 Test-time computation versus pretraining a larger model  8.4 Future relevance of test-time compute scaling 9. September: Comparing multimodal LLM paradigms  9.1 Multimodal LLM paradigms  9.2 Nvidia’s hybrid approach  9.3 Multimodal LLMs in 2025 10. October: Replicating OpenAI O1’s reasoning capabilities  10.1 Shortcut learning vs journey learning  10.2 Constructing long thoughts  10.3 Distillation – the quick fix?  10.4 The state of AI research  10.5 The future of LLMs in the light of o1 (and o3) 11. November: LLM scaling laws for precision  11.1 Chinchilla scaling laws refresher  11.2 Low-precision training  11.3 Precision scaling laws takeaways  11.4 Model scaling laws in 2025 12. December: Phi-4 and learning from synthetic data  12.1 Phi-4 performance  12.2 Synthetic data learnings  12.4 Future importance of synthetic data Conclusions and outlook  Multimodal LLMs  Computational efficiency  State space models  Scaling  What I am looking forward to

If you’re an AI engineer trying to optimize your LLMs for inference, here’s a quick guide for you 👇 Efficient inference isn’t just about faster hardware, it’s a multi-layered design problem. From how you compress prompts to how your memory is managed across GPUs, everything impacts latency, throughput, and cost. Here’s a structured taxonomy of inference-time optimizations for LLMs: 1. Data-Level Optimization Reduce redundant tokens and unnecessary output computation. → Input Compression:  - Prompt Pruning, remove irrelevant history or system tokens  - Prompt Summarization, use model-generated summaries as input  - Soft Prompt Compression, encode static context using embeddings  - RAG, replace long prompts with retrieved documents plus compact queries → Output Organization:  - Pre-structure output to reduce decoding time and minimize sampling steps 2. Model-Level Optimization (a) Efficient Structure Design → Efficient FFN Design, use gated or sparsely-activated FFNs (e.g., SwiGLU) → Efficient Attention, FlashAttention, linear attention, or sliding window for long context → Transformer Alternates, e.g., Mamba, Reformer for memory-efficient decoding → Multi/Group-Query Attention, share keys/values across heads to reduce KV cache size → Low-Complexity Attention, replace full softmax with approximations (e.g., Linformer) (b) Model Compression → Quantization:  - Post-Training, no retraining needed  - Quantization-Aware Training, better accuracy, especially <8-bit → Sparsification:  - Weight Pruning, Sparse Attention → Structure Optimization:  - Neural Architecture Search, Structure Factorization → Knowledge Distillation:  - White-box, student learns internal states  - Black-box, student mimics output logits → Dynamic Inference, adaptive early exits or skipping blocks based on input complexity 3. System-Level Optimization (a) Inference Engine → Graph & Operator Optimization, use ONNX, TensorRT, BetterTransformer for op fusion → Speculative Decoding, use a smaller model to draft tokens, validate with full model → Memory Management, KV cache reuse, paging strategies (e.g., PagedAttention in vLLM) (b) Serving System → Batching, group requests with similar lengths for throughput gains → Scheduling, token-level preemption (e.g., TGI, vLLM schedulers) → Distributed Systems, use tensor, pipeline, or model parallelism to scale across GPUs My Two Cents 🫰 → Always benchmark end-to-end latency, not just token decode speed → For production, 8-bit or 4-bit quantized models with MQA and PagedAttention give the best price/performance → If using long context (>64k), consider sliding attention plus RAG, not full dense memory → Use speculative decoding and batching for chat applications with high concurrency → LLM inference is a systems problem. Optimizing it requires thinking holistically, from tokens to tensors to threads. Image inspo: A Survey on Efficient Inference for Large Language Models ---- Follow me (Aishwarya Srinivasan) for more AI insights!

DeepSeek's recent developments have ignited significant discussion in the AI community, and I wanted to take a minute to share some thoughts. If you haven’t heard, the company's latest model, R1, showcases a reasoning capability comparable to OpenAI's o1, but with a notable distinction: DeepSeek claims that their model was trained for much less cost.   It isn’t clear yet if DeepSeek is the real deal or a DeepFake, but regardless of what we learn in the coming days, it’s clear that this is a wake up call -- the path of bigger and bigger LLMs that rely on ever-increasing GPUs and massive amounts of energy is not the only path forward. In fact, it’s clear there is very limited upside to that approach, for a few reasons:   ⭐️ First, pure scaling of LLMs at training time has reached the point of diminishing or maybe near zero returns. Bigger models trained with more data are not resulting in meaningful improvements.   🤔 Further, enterprises don’t need huge, ask-me-anything LLMs for most use cases. Even prior to DeepSeek, there's a noticeable shift towards smaller, more specialized models tailored to specific business needs. As more enterprise AI use cases emerge, it becomes more about inference -- actually running the models to drive value. In many cases, that will happen at the edge of the internet, close to end users. Smaller models that are optimized to run on commodity hardware are going to create more value, long-term, than over-sized LLMs.   💡 Finally, the LLM space is ripe for optimization. The AI models we have seen so far have focused on innovation by scaling at any cost. Efficiency, specialization, and resource optimization are once again taking center stage, a signal that AI’s future lies not in brute force alone, but in how strategically and efficiently that power is deployed.

Today is a delightful day in open-source AI! Meta's Llama-2 release is a major milestone, but we also need to stay grounded. Happy to share my notes: ▸ Llama-2 likely costs $20M+ to train. Meta has done an incredible service to the community by releasing the model with a commercially-friendly license. AI researchers from big companies were wary of Llama-1 due to licensing issues, but now I think many of them will jump on the ship and contribute their firepower. ▸ Meta's team did a human study on 4K prompts to evaluate Llama-2's helpfulness. They use "win rate" as a metric to compare models, in similar spirit as the Vicuna benchmark. 70B model roughly ties with GPT-3.5-0301, and performs noticeably stronger than Falcon, MPT, and Vicuna.  I trust these real human ratings more than academic benchmarks, because they typically capture the "in-the-wild vibe" better. ▸ Llama-2 is NOT yet at GPT-3.5 level, mainly because of its weak coding abilities. On "HumanEval" (standard coding benchmark), it isn't nearly as good as StarCoder or many other models specifically designed for coding. That being said, I have little doubt that Llama-2 will improve significantly thanks to its open weights. ▸ Meta's team goes above and beyond on AI safety issues. In fact, almost half of the paper is talking about safety guardrails, red-teaming, and evaluations. A round of applause for such responsible efforts! In prior works, there's a thorny tradeoff between helpfulness and safety. Meta mitigates this by training 2 separate reward models. They aren't open-source yet, but would be extremely valuable to the community. ▸ I think Llama-2 will dramatically boost multimodal AI and robotics research. These fields need more than just blackbox access to an API.  So far, we have to convert the complex sensory signals (video, audio, 3D perception) to text description and then feed to an LLM, which is awkward and leads to huge information loss. It'd be much more effective to graft sensory modules directly on a strong LLM backbone. ▸ The whitepaper itself is a masterpiece. Unlike GPT-4's paper that shared very little info, Llama-2 spelled out the entire recipe, including model details, training stages, hardware, data pipeline, and annotation process. For example, there's a systematic analysis on the effect of RLHF with nice visualizations. Quote sec 5.1: "We posit that the superior writing abilities of LLMs, as manifested in surpassing human annotators in certain tasks, are fundamentally driven by RLHF." Congrats to the team again 🥂!

Prompt formatting can have a dramatic impact on LLM performance, but it varies substantially across models. Some pragmatic findings from a recent research paper: 💡 Prompt Format Significantly Affects LLM Performance. Different prompt formats (plain text, Markdown, YAML, JSON) can result in performance variations of up to 40%, depending on the task and model. For instance, GPT-3.5-turbo showed a dramatic performance shift between Markdown and JSON in code translation tasks, while GPT-4 exhibited greater stability. This indicates the importance of testing and optimizing prompts for specific tasks and models. 🛠️ Tailor Formats to Task and Model. Prompt formats like JSON, Markdown, YAML, and plain text yield different performance outcomes across tasks. For instance, GPT-3.5-turbo performed 40% better in JSON for code tasks, while GPT-4 preferred Markdown for reasoning tasks. Test multiple formats early in your process to identify which structure maximizes results for your specific task and model. 📋 Keep Instructions and Context Explicit. Include clear task instructions, persona descriptions, and examples in your prompts. For example, specifying roles (“You are a Python coder”) and output style (“Respond in JSON”) improves model understanding. Consistency in how you frame the task across different formats minimizes confusion and enhances reliability. 📊 Choose Format Based on Data Complexity. For simple tasks, plain text or Markdown often suffices. For structured outputs like programming or translations, formats such as JSON or YAML may perform better. Align the prompt format with the complexity of the expected response to leverage the model’s capabilities fully. 🔄 Iterate and Validate Performance. Run tests with variations in prompt structure to measure impact. Tools like Coefficient of Mean Deviation (CMD) or Intersection-over-Union (IoU) can help quantify performance differences. Start with benchmarks like MMLU or HumanEval to validate consistency and accuracy before deploying at scale. 🚀 Leverage Larger Models for Stability. If working with sensitive tasks requiring consistent outputs, opt for larger models like GPT-4, which show better robustness to format changes. For instance, GPT-4 maintained higher performance consistency across benchmarks compared to GPT-3.5. Link to paper in comments.