Rishabh Misra’s Post

Cut PyTorch VRAM usage in half — without sacrificing performance! We just published a deep dive on how we optimised the McByte tracker in PyTorch from ~16 GB of VRAM to under 8 GB by restructuring the similarity computation and reducing intermediate memory allocations — a huge win for real-time vision workloads and constrained hardware. Read the full post: https://lnkd.in/dJRCRWJk

I spent weeks getting GPUs to work with Slurm, and here's what nobody tells you about ML infrastructure. Setting up AWS ParallelCluster with GPU nodes may sound straightforward, but it is not. The challenges include: - GRES config - gres.conf - slurm.conf GPU type mappings - EFA networking - EFS mount targets Every failed job leads to another debug cycle with questions like, "Why didn't the node come up?" and "Why doesn't Slurm see the GPU?" After 555 jobs, I finally had a usable Slurm dataset. On the Kubernetes side, I managed 1.2 million jobs across EKS clusters, facing different challenges such as pod specs, queue state snapshots, and resource fragmentation metrics. During experiments across both schedulers, I discovered an unexpected insight: the costly GPU infrastructure that everyone struggles to set up often lacks the simplest information: "When will my job start?" When I built models to predict queue delays: - A logistic regression with GPU telemetry achieved an AUROC of 0.815, outperforming gradient boosting without it, which scored 0.498. The plumbing matters more than the model. - Calibration—whether "90% confidence" truly means 90%—collapsed 22 times when transferring between Slurm and Kubernetes. While the model ranks well, the probabilities can be misleading. - The solution is inexpensive: isotonic recalibration, which takes milliseconds and requires no retraining. This work began with a simple question: can we provide users with a reliable answer to "When will my job start?" I have just published first preprint on TechRxiv: [Calibration Under Extreme Imbalance: A Multi-Cluster Benchmark for Operational Queue Delay Prediction](https://lnkd.in/erGsUKvW

Slurm can be a bit intimidating at first, but once configured, it's easy to use and can give your AI teams a solid framework to run distributed training at scale very efficiently. We can help you set up Slurm clusters on our Ori infrastructure to run your training jobs or any high compute jobs you need. You can also do it yourself - just check out this guide below.

In het weekend probeer ik altijd wat bij te lezen (early morning - late night) in AI land. Ik kwam deze leuke open source tool tegen. https://lnkd.in/eUGNmNPy A terminal tool that right-sizes LLM models to your system's RAM, CPU, and GPU. Detects your hardware, scores each model across quality, speed, fit, and context dimensions, and tells you which ones will actually run well on your machine. Ships with an interactive TUI (default) and a classic CLI mode. Supports multi-GPU setups, MoE architectures, dynamic quantization selection, and speed estimation.

Running an Small Language Model Without a GPU (CPU-Only Inference): An Experiment I wanted to explore the practical implications of running an SLM on minimal infrastructure without relying on a GPU. I started by selecting Gemma 3 1B as the model. Gemma 3 1B is a small, text-only, open-weight language model from Google’s Gemma family. I used llama.cpp as the inference engine to run the model. For the hardware, I used an cloud-based VPS with 4 CPU cores and 8 GB RAM, running Ubuntu. I chose a 4-bit quantized GGUF version of Gemma 3 1B (Q4_K_M) from Hugging Face: https://lnkd.in/ghx8mc49 On my server, the SLM inference runs as a system service. The llama.cpp server API (llama-server) is securely exposed over the internet with API key–based authentication. With this setup, I could query the SLM remotely from my laptop over the internet via an OpenAI-compatible Chat Completions API. On testing in Postman, each query took around 40+ seconds to deliver a response. CPU usage reached 100% across all four cores. Despite this, the system continued to operate reliably under a dozen sequential queries, and memory usage remained within limits, with no crashes or OOM issues. This latency is not viable! So, I switched to a smaller 4-bit quantized GGUF version of Gemma 3 270M (Q4_K_M), which has significantly fewer parameters than the 1B variant. Also available on Hugging Face: https://lnkd.in/ggTQHVNJ After configuring the new model on my server, I tested again. The results were significantly better, with responses arriving in the 5–15 second range, depending on query complexity. That’s it for now. I plan to evaluate other small language models and vary system specifications to better understand their impact on inference latency and overall performance. Microsoft’s Phi family looks like a strong candidate for future experiments.

llm-d v0.5 is here 🎉 The journey from "peak performance" to "operational rigor" is what defines production-grade AI. While our last release (v0.4) was about breaking latency records, v0.5 is about sustaining that performance under the unpredictable pressure of real-world scale. With today’s release, we’re introducing the features required to turn state-of-the-art research into resilient, cost-efficient infrastructure. ⚙️ What’s New in v0.5? ⚫️ Hierarchical KV Offloading: We’ve decoupled cache capacity from GPU memory. Our new storage architecture introduces a three-tier hierarchy (GPU, CPU, and Disk), allowing you to sustain throughput even when your active working set dwarfs your cluster's RAM. ⚫️ Advanced Scheduling & LoRA Support: Cache-aware routing now fully supports LoRA adapters and active-active high availability, preventing "thundering herd" issues in multi-tenant environments. ⚫️ Resilient Networking (UCCL): Our new transport backend provides 2.4x greater resilience to network congestion, ensuring consistent tail latency in shared production environments. ⚫️ The "Research Paper Principle": We’ve overhauled our developer experience. Every performance claim in v0.5 is backed by a reproducible, version-controlled configuration you can validate with a single command. ⚫️ Scale-to-Zero: New autoscaling updates allow inference pools to scale down to zero during idle periods, significantly reducing costs for intermittent workloads. 📊 Impact Highlight: Breaking the Memory Ceiling [Image: Figure 1 - KV-cache throughput under growing user concurrency on Llama-3.1-70B] As shown in our latest benchmarks, standard GPU-only deployments experience a sharp performance collapse once HBM is saturated. llm-d v0.5 creates a "performance floor." By leveraging storage-backed offloading, we sustained ~185k tok/s even as concurrency scaled to 250 users—a 13.9x improvement over baseline performance. 🤝 Built with the Community We are proud to see our UCCL backend merged into NIXL 0.9 and our KV-Connector work integrated with vLLM. Ready to scale? Check out the full release notes and the updated benchmarking tools on our GitHub. 🛠️ 🔗 https://lnkd.in/gRruvFQU

LLM decode can spend more time on CPU kernel-launch overhead than on GPU compute. CUDA Graphs flip that by replaying an entire forward pass with a single call—but only if your shapes and memory addresses stay stable. I wrote a deep dive on why decode is a perfect fit, why context/mixed batches are a combinatorial nightmare, and how TensorRT-LLM uses monolithic vs piecewise graphing to get most of the win without breaking correctness. Read: https://lnkd.in/dE8QSYZZ

I watched a senior engineer spend three weeks quantizing an LLM to 4-bit. The P99 latency got worse. The issue wasn’t the technique; it was treating quantization as a storage problem instead of a memory-bandwidth problem. At Twitter, I spent a month debugging why our "optimized" models ran slower than the originals. The models were smaller. The math was correct. Yet latency regressed. The missing piece: the *unpacking tax*. Here’s the reality most benchmarks hide: Time ≈ Total bytes moved / Memory bandwidth On paper, moving from FP16 (16-bit) to INT4 (4-bit) means 4× less data moving across the memory bus per token. In a memory-bound regime, that translates to 3–4× higher throughput. But there’s a catch. GPUs don’t compute in 4-bit or 8-bit. Those weights are dequantized back to FP16/BF16 in the local cache before computation. That dequantization costs clock cycles and creates production surprises: → High batch sizes: Time saved on memory movement dominates = throughput improves → Batch size of 1: Unpacking overhead dominates = latency gets worse Quantization is not a free win. It’s a tradeoff. If you’re choosing a method, align it with your deployment reality: → GPTQ: Effective for static weights, but sensitive to outliers → AWQ: Preserves critical weights at higher precision for better quality → GGUF: Excellent for CPU/Metal inference, less relevant for H100/A100 clusters This is Part 4 of a deep dive into inference optimization. Previous posts: Memory Wall: https://lnkd.in/gdT26UTV KV Cache: https://lnkd.in/gKkrqVzf Paged Attention: https://lnkd.in/gX5JNZhn Next up: I will break down the closest thing to "cheating physics" in ML - Speculative Decoding. What’s the most expensive quantization mistake you’ve seen in production - latency, quality, or operability?

GPU pipelines don’t stall at compute — they stall at upload. Most discussions around GPU utilization focus on kernels, batch size, or VRAM limits. But in real systems, the dominant idle source often sits earlier: disk → shard → manifest → pinned buffer → VRAM → compute If the upload path can’t keep pace, the GPU starves regardless of how efficient the kernels are. Typical hidden upload bottlenecks: • fragmented reads and small files • late materialization / decode • pageable host memory copies • repacking before transfer • launch waiting on data readiness The result is familiar in telemetry: idle_gap ≈ data_ready_delay + launch_latency So even with optimized kernels: data_ready_delay dominates → utilization collapses The practical fix isn’t more compute tuning — it’s upload alignment: • contiguous shard ingestion • pre-packed pinned buffers • direct host→VRAM staging • launch gated on ready events Once upload keeps pace, GPU idle gaps shrink and adaptive launch control actually has signal to work with. In many pipelines, the GPU isn’t the bottleneck. The uploader is.

Found a useful tool recently - llmfit, helps when you want to explore local LLMs but stuck in analysis paralysis. It detects your CPU, RAM, and GPU, then scores 157 models from 30 providers showing which ones fit your system. It shows parameters, score, speed estimates (tokens/s), memory usage, quantization, and a fit rating (Perfect/Good/Marginal). The score is a weighted composite of quality, speed, fit, and context window - with weights adjusted per use case (coding, reasoning, chat, etc.). brew install llmfit llmfit Integrates with Ollama - see installed models and download new ones directly from the TUI. https://lnkd.in/gPYeFmpW #ai #llm #cli #devtools #ollama