Daily Dose of Data Science

DevOps vs. MLOps vs. LLMOps, explained visually: . . Many teams are trying to apply DevOps practices to LLM apps. But DevOps, MLOps, and LLMOps solve fundamentally different problems. DevOps is software-centric. You write code, test it, and deploy it. The feedback loop is straightforward: Does the code work or not? MLOps is model-centric. Here, you're dealing with data drift, model decay, and continuous retraining. The code might be fine, but the model's performance can degrade over time because the world changes. LLMOps is foundation-model-centric. Here, you're typically not training models from scratch. Instead, you're selecting foundation models and then optimizing through three common paths: - Prompt Engineering - Context/RAG Setup - Fine-Tuning But here's what really separates LLMOps: The monitoring is completely different. In MLOps, you track data drift, model decay, and accuracy. In LLMOps, you're watching for: - Hallucination detection - Bias and toxicity - Token usage and cost - Human feedback loops This is because you can't just check if the output is "correct." You need to ensure it's safe, grounded, and cost-effective. The evaluation loop in LLMOps also feeds back into all three optimization paths simultaneously. Failed evals might mean you need better prompts, richer context, OR fine-tuning. So it's not a linear pipeline anymore. One more thing: prompt versioning and RAG pipelines are now first-class citizens in LLMOps, just like data versioning became essential in MLOps. And the ops layer you choose should match the system you're building. 👉 Over to you: What does your LLM monitoring stack look like right now? ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

Cut your LLM inference costs by up to 95%. No prompt rewrites. No model changes. headroom is a Python library, proxy, and MCP server that compresses tool outputs, logs, files, and RAG chunks before they reach the LLM. It achieves 60-95% token reduction with no measurable quality loss. The problem: as agentic workloads scale, token consumption compounds fast. Context windows fill with verbose tool responses, bloated logs, and uncompressed file chunks. headroom intercepts all of it upstream, before the LLM ever sees it. Three deployment options: - Drop-in Python library - Proxy (works with any LLM client) - MCP server for agent pipelines No architectural overhaul required. Compression happens at the data layer, invisible to the rest of your stack. 100% open-source. Link to the GitHub repo in comment. ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

A tricky LLM interview question: (answer below) Evicting 90% of the KV cache can free almost none of the memory it was using. The KV cache grows with every token a model generates. Each token appends its KV vectors across every layer, and nothing is freed while generation continues. If a 32K-token CoT caches ~32K tokens of KV vectors, a Qwen3-32B with 4-bit weights will run out-of-memory around 24K tokens on a 24GB GPU. One obvious solution is to keep the important tokens and drop the rest, since attention is sparse enough to allow it. But this does not solve the memory problem yet. The reason is paged attention. Under the hood, it splits GPU memory into fixed physical blocks, each one holds the KV for about 16 tokens. This block returns to the allocator only when EVERY slot inside it is empty. Since the eviction logic selects tokens by importance, and such tokens are scattered across blocks... ...so despite eviction, almost every block is left with some tokens. For instance, if the logic evicts 14k of 16k tokens across 1,000 blocks, most likely every block will still have a token. This means the allocator frees almost nothing. Placing the new tokens into those freed slots is not ideal because it breaks the cache's layout. Say token 16,001 arrives, and it's placed in the slot the 40th token used to hold. The cache now reads position 38, then 16,001, then 41, so the cache is no longer in token order. Attention can still compute the right answer from that, but only if every slot now carries a separate note recording positions. This introduces another bookkeeping cost. There's another problem. Eviction methods pick which tokens to keep based on attention scores. But fast attention kernels like FlashAttention never save these scores. They compute attention in small pieces and throw the full score grid away, which is also why they're fast. The workaround is to fall back to eager attention, which gives up the speed FlashAttention provides. NVIDIA published TriAttention to solve these problems. It never needs attention scores. Instead, it scores tokens from the geometry of the model's key and query vectors before RoPE is applied. For the memory problem, it runs a compaction pass every 128 decoded tokens. The surviving tokens slide forward to close the holes eviction creates, so whole blocks empty out and return to the allocator while the cache stays in token order. On long reasoning traces, the approach matches full-attention accuracy while decoding 2.5x faster and using 10.7x less KV memory. KV cache compression is a big infrastructure problem. The number that decides whether it works is the count of freed blocks, not the count of evicted tokens. You can find the NVIDIA paper in the comments. 👉 Over to you: What is your solution to this? ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

From AGI → ASI Google mapped what comes after human-level AI. There's a brilliant new paper from Google DeepMind, and it skips the fight everyone else is having. Almost every AI debate today is about reaching human-level intelligence. This one asks the harder question instead. Once we have an AI as capable as a person, where does it go next? Their reasoning is uncomfortable because it's so simple. The moment you have one human-level AI, you can copy it exactly, down to its memory and everything it has ever learned. So you don't end up with one digital worker. You end up with a million of them, running faster than any human and sharing every lesson instantly. A group like that could outthink any company or research lab long before any single AI becomes a lone genius. That's the heart of it. Scale alone might carry us from "as smart as a person" to "smarter than entire teams of the best experts." The paper maps four ways this could play out, and they can all happen at once. Keep scaling what already works. Invent a new approach when the current one runs dry. Let AI improve AI in a loop that feeds itself. Or let armies of AIs organize into something smarter than any one of them, the way a company outperforms any single employee. Then comes the honest part, which is rare for a report like this. They list everything that could slow it all down. Running out of good training data. Costs and energy becoming impossible to sustain. Research simply getting harder as the easy wins disappear. But the most interesting roadblock isn't a wall. It's a question. Today's AI learns from everything humans have already figured out, and it's brilliant at remixing our ideas. What it has never done is invent a genuinely new idea from scratch. The test they borrow from Demis Hassabis says it best. Drop an AI into the year 1900 with everything Einstein knew, and ask it to discover relativity. Right now it can't, and something is clearly still missing. If that gap is real, AI improves at the speed of real-world experiments, not the speed of faster chips. Here's the shift in thinking we keep coming back to. The popular picture is one big moment where everything changes overnight. This paper argues for something different. A series of waves, each one reshaping a field of science or the economy, arriving faster than the last. And the most useful work right now isn't guessing when. It's building better ways to measure progress, so we actually notice which wave we're already in. One last detail we loved. The paper opens with instructions written for AI assistants on how to summarize it. The authors simply assumed you'd ask a model to read it for you. Link to the paper in the comments. ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

Turn any paper into running code. Just swap arxiv → autoarxiv in the paper url. That hands the paper to an AI agent from alphaXiv. It reads the abstract, the claims, and the linked GitHub repo, then clones the codebase and works through the usual setup pain like dependencies, broken paths, environment config, and hardware assumptions. From there it designs a minimal reproduction. That means a smaller model, fewer steps, and a single GPU instead of a cluster, scaled down just enough to test whether the headline claim holds. The whole run is live and fully logged. Loss curves, metrics, and training progress are all observable as it happens. What comes back is a clean signal on whether the minimal run matches the paper's reported result, plus an estimate of what a full replication would cost in compute and time. A lot of research code dies in setup before anyone verifies a single number. This moves reproduction from a weekend of debugging to a url change. Pick a paper and try it now. ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

Web scraping will never be the same. (100% open-source visual search at scale) PixelRAG is a retrieval system that skips HTML parsing completely. Instead of scraping a page into text and embedding chunks, it screenshots the page and retrieves the image. A vision-language model reads the answer straight off the pixels. Why that matters: parsing is where web RAG quietly loses information. - A single HTML-to-text parser can drop 40%+ of a page. - Tables, charts, and layout get flattened or thrown out. - Swapping parsers alone can move accuracy ~10 points on the same docs. PixelRAG indexes the page a person actually sees. The team built a visual index of all of Wikipedia, 30M+ screenshots, and it still beats the strongest text RAG baseline by 18.1% on text-only QA. The repo also ships a Claude Code plugin that gives Claude eyes. It lets Claude screenshot any URL and read the rendered page instead of scraping the DOM. So you can hand it a live page, an arXiv paper, or your local site and ask what it actually looks like. One setup script. No MCP server, no backend. How the pipeline works: - Renders each document (web, PDF, image) to image tiles. - Embeds them with Qwen3-VL-Embedding, LoRA fine-tuned on screenshots. - Builds a FAISS index and serves a search API. A stronger reader model lifts accuracy with no re-indexing, since the index is just pixels. Everything is open-source under Apache-2.0. We have shared the GitHub repo in the comments. ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

Claude Code’s architecture, explained visually: . . Claude Code is a lot more than a CLI that invokes the Claude models. The actual system has six layers, and the model is just one node inside the loop. 1) Input layer handles session management, permission gating, and YAML-based trust tiers before anything reaches the model. 2) Knowledge layer holds the skill registry, context compressor, task graph, and cross-session memory store. This is where harness intelligence lives outside the weights. The context compressor doesn’t summarize your conversation the way ChatGPT does. Instead, it runs structured extraction on file paths, code snippets, and error histories while pruning redundant tool outputs. The goal is to keep the context usable, not just smaller. 3) Execution layer runs tool dispatch through a typed registry with one handler per tool, like bash, read, write, grep, glob, and revert. 4) Integration layer connects the MCP runtime to external servers (filesystem, git, custom). Tools register inward, and memory writes outward to a markdown file (agent_memory.md) that persists across sessions. 5) Multi-agent layer is the most underappreciated piece, and it works very differently from what most people assume. Claude Code supports two levels of parallelism: - Subagents are lightweight workers that run inside your session. They get their own context window, do a focused task (search the codebase, explore a file tree), and return results to the parent. They can’t talk to each other, and they can’t spawn their own subagents. - Agent teams go further. One session acts as a team lead, and it spawns independent teammates, each running as a full Claude Code instance with its own context window. Each teammate gets git worktree isolation. It’s a separate working directory with its own branch, sharing the same repository history. This means agents can write to overlapping parts of the codebase without file conflicts. When they finish, worktrees with no changes are cleaned up automatically. 6) Observability layer wraps everything. An event bus with lifecycle hooks logs all tool calls and messages, creating a complete audit trail of the agent’s actions and decisions. Finally, the master agent loop sits at the center of all six layers. It assembles context, calls the model, receives a tool request, executes it, feeds the result back in, and repeats. Every iteration is one turn. Within a turn, the model might request a tool call. That request flows through the permission system, gets executed, and the output feeds back into the loop as the next input. The loop itself is single-threaded on purpose. All the intelligence lives in the layers around it, not in the loop logic. Anthropic calls it a dumb loop because the model reasons, and the harness mediates. This is the architecture behind Claude Code. ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

The AI engineering master stack in 2026! (bookmark this) It covers ten layers from the model itself to running it safely in production. 1) Foundations set how the model represents input through tokens, embeddings, transformer, attention, context window, positional encoding, and mixture of experts. 2) Model behavior covers how a trained model responds through pretraining, post-training, sampling, temperature, reasoning models, multimodality, and test-time compute. 3) Prompt engineering shapes the output through the prompt alone using system prompts, few-shot, chain-of-thought, structured outputs, prompt caching, self-consistency, and meta-prompting. 4) Retrieval feeds the model data it was never trained on through chunking, vector databases, hybrid search, reranking, retrieval eval, query rewriting, and GraphRAG. 5) Agents let the model take actions instead of only answering through function calling, ReAct, planning, reflection, multi-agent, computer use, and human-in-the-loop. 6) Context engineering controls what fills the context window across steps through context management, compaction, memory, MCP, agent harness, just-in-time retrieval, and structured note-taking. 7) Fine-tuning changes the weights when prompting and context fall short through SFT, LoRA, RLHF, DPO, distillation, GRPO, and RLVR. 8) Inference optimization makes the model cheap and fast to serve through quantization, KV cache, batching, speculative decoding, vLLM serving, FlashAttention, and PagedAttention. 9) Evaluation measures whether the system is actually correct through benchmarks, LLM-as-judge, golden datasets, hallucination detection, regression tests, trajectory evaluation, and red teaming. 10) LLMOps and safety keep the system reliable and safe in production through observability, cost tracking, guardrails, PII redaction, feedback loops, prompt injection defense, and model routing. This grid below is the overview, but each layer is deep enough to be its own field with dedicated tooling. Over to you: What else would you add here? ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

Researchers made K-means 200x faster! (beats approaches like cuML and FAISS) Flash-KMeans is an IO-aware implementation of KMeans that rethinks the algorithm around modern GPU bottlenecks. By attacking the memory bottlenecks directly, Flash-KMeans achieves: - 30x speedup over cuML - 200x speedup over FAISS At the million-scale, Flash-KMeans can complete a k-means iteration in milliseconds. Several reasons why this is important: K-means has always been an offline primitive. Something you run once to preprocess data and move on. These speedups change that. ↳ Vector indices like FAISS use k-means to build search indices. Faster k-means means you can re-index dynamically as data changes, not batch it overnight. ↳ LLM quantization methods need k-means to find optimal weight codebooks, per layer, repeatedly. What takes hours could now take minutes. ↳ MoE models need fast token routing at inference time. Millisecond k-means makes it viable to run this inside the inference loop, not just in preprocessing. The 200x over FAISS is the number to internalize. FAISS is the industry standard. Most production vector search systems sit on top of it. Link to the paper and code in the comments. ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.

You'll regret using Claude Code without this. Claude Code stops at the boundary of your terminal. It has no visibility into Datadog traces, Jira tickets, Slack decision history, or cloud infra state. So when something breaks in production, Claude Code isn't there. When the same architectural debate resurfaces because nobody documented the original reasoning, Claude Code can't help. This is an institutional memory problem, which needs a different agent. And it has now actually been shipped into the CodeRabbit Agent. The agent lives inside Slack and connects to the actual stack (like GitHub, Jira, Linear, Notion, Datadog, Sentry, PostHog, AWS, GCP, and 20+ other tools via native integrations), covering the full SDLC beyond code generation. Once you mention it in a Slack thread, it can pull your Datadog traces, find which PR caused the issue, open a fix, and document the whole incident in Linear. All inside the same thread where the team sees every step." One thing that's nicely engineered into the memory layer is the long-term knowledge retention. CodeRabbit Agent retains decisions, patterns, and learnings across three tiers: thread, channel, and org-wide, respecting Slack's privacy boundaries. So what the team decided in one thread is available when the same problem surfaces months later. The knowledge compounds instead of leaking. Under the hood, four layers make this work: 1) Context: It pulls live state from GitHub, Jira, Linear, Notion, Datadog, and AWS or GCP. 2) Memory: It retains decisions across thread, channel, and org-wide tiers, so what was decided once does not disappear. 3) Multiplayer: It operates inside the Slack thread, steerable in plain English, resumable across devices, and visible to the whole team. 4) Guardrails: It runs every action inside channel-scoped controls on access, memory, tool use, and cost. These are complementary layers, and together they form a shared, persistent, governed intelligence layer that compounds over time. Your coding agent makes individual engineers faster. This agent acts as a second brain to make the teams smarter. Try CodeRabbit Agent for free, link in the first comment. ____ Find us → Daily Dose of Data Science Every day, we share tutorials and insights on DS, ML, LLMs, and RAGs.