LiteLLM AI Gateway

We're excited to launch the newly revamped https://www.litellm.ai/ The new site marks LiteLLM AI Gateway evolution into the go-to AI Gateway for platform teams. Route across models, roll out access, and manage costs with minimal latency overhead

We added two cost views to the Cost Optimization dashboard on LiteLLM AI Gateway: spend by tool, and prompt cache leakage. Spend by tool attributes each request's cost to the tools that ran on it, so you can see which MCP tools are actually driving spend. Cache leakage shows, per virtual key or per Claude model, how much you're paying for input tokens you could have cached and didn't, with a potential-savings estimate that sits right next to the caching setting that fixes it. Get started here: https://lnkd.in/dgpU2i8b Improvements on this release 👇 • #33190 - admit gateway DCR session bearers at the aggregate /mcp scope • #34322 - use the official Google Drive streamable HTTP MCP server • #33884 - fall through to an ephemeral DCR mint when passthrough authorize has no client_id • #34250 - resolve word-form budget_duration so it no longer silently resets daily • #34259 - discover autorouter models via /v1/models so an AI-API-only key works • #34429 - reject failed atomic budget reservations under fail_closed_budget_enforcement • #34417 - add overwrite_user_with_key_hash to stamp the outgoing user param with the key hash • #32350 - add RESTful PATCH /v2/organization/{organization_id} • #34325 - bake non-root prisma engines so migrations run offline for any uid • #33422 - preserve reasoning through prompt hooks on the Responses API • #34347 - send type in tool_choice disable_parallel_tool_use for Bedrock Converse • #34319 - strip remaining output_format schema keywords Anthropic rejects • #34338 - forward SageMaker stream events as they arrive to cut TTFT • #32841 - edit fallback chains from router settings in the UI • #31743 - find logs by request id across pages and dates

We're shipping per-session incremental guardrail scanning on LiteLLM AI Gateway. Set only_scan_new_messages: true on a Bedrock guardrail and each turn only sends the messages it hasn't already scanned in that session, instead of re-scanning the whole growing context every request. This is for AI Platform Teams running Bedrock Guardrails on long agent or Claude Code sessions, where that re-scan on every turn normally drives guardrail throttling. Claude Code sessions get this automatically. No session id falls back to a full scan, and a blocked segment is never cached, so it gets re-checked on retry. Off by default. Get started here: https://lnkd.in/daNCVfMD Improvements on this release 👇 • #33278 — add only_scan_new_messages for per-session incremental guardrail scanning • #33261 — share CLI SSO login sessions across workers without enable_redis_auth_cache • (#33253) · CLI SSO login now works on a multi-worker proxy with Redis attached • #34133 — emit Nova Sonic realtime session.created/session.updated · provider fix (Bedrock realtime) • #34160 — reflect REDIS_* env cache config in the UI and stop it overwriting the stored password • #34156 — surface env-var-sourced theme and logging-callback settings

We're excited to launch day-0 support for Gemini 3.6 Flash and Gemini 3.5 Flash-Lite. These models are great for coding and agentic work, especially because they're more affordable than the previous generation. Gemini 3.6 Flash is Google's new default Flash: $1.50 / $7.50 per 1M input/output tokens (cheaper output than 3.5 Flash), and about 17% fewer output tokens per task. Flash-Lite is the low-latency one, at $0.30 / $2.50 per 1M and ~350 tokens/sec, built for agentic search and high-throughput document work. Here's how to get started: https://lnkd.in/dJf5mg2f Improvements on this release 👇 • #34035 - route OCR, messages, and responses-websockets through Rust by default • #34107 - honor pre-computed Entra ID auth for Azure /messages • #33844 - add deepkeep as a custom guardrail • #34189 - block provider key exposure via URL-valued model destinations • #33908 - sanitize model_armor error details by default • #34072 - store enterprise IdP identity assertion at SSO login for MCP EMA egress • #34065 - let an admin-pinned issuer drive MCP OAuth discovery for url-less servers • #34059 - add Google Sheets, Drive, Calendar, and Docs to the MCP OpenAPI registry • #31007 - add configurable budget_reset_time of day • #34130 - make DB config-reload interval configurable • #34124 - honor request-level num_retries over the global setting • #34129 - stop per-deployment num_retries double-counting as provider max_retries • #34199 - bump litellm 1.94.0 → 1.95.0

Thrilled to announce a NEW COST OPTIMIZATION DASHBOARD on LiteLLM AI Gateway. This enables admins to know how much they're saving and what's actually working. s/o Jordan Toor for the idea This covers the cost saved from prompt compression, prompt caching and auto routing. Try now - https://lnkd.in/gnW5upTg

On LiteLLM AI Gateway, MCP keys that are scoped to a toolset now call precisely from the tools it grants. This is enforced identically on the MCP protocol path and the REST path. Toolset grants now resolve inside the shared permission primitives every path already consults (get_allowed_mcp_servers_for_key, get_allowed_tools_for_server), so tools/list and tools/call can no longer drift, and the toolset acts as a ceiling — a key can't reach the sibling tools on the same server it was never granted. This is great for AI Platform Teams exposing MCP servers to agents who need a curated toolset to be a real authorization boundary. Get started with MCP permission management here: https://lnkd.in/ghdtN9QS Improvements on this release 👇 • #33612 - expand MCP toolset grants in the shared permission primitives so tools/call honors them • #33586 - make the preemptive-401 MCP OAuth challenge mode-aware (gateway-managed vs client-credentials M2M) • #33153 - per-server outcomes for aggregate MCP tools/list and truthful single-server REST statuses • #33584 - keep the MCP reference intact when the semantic tool filter narrows tools • #33573 - add enable_anthropic_prompt_caching for automatic cache_control injection (Anthropic + Bedrock) • #33587 - ingest and round-trip SCIM entitlements and roles user attributes • #33746 - add router plugin reference catalog • #32441 - enforce max_parallel_requests as a per-slot concurrency gauge • #33736 - bill partial streamed spend when the client disconnects mid-stream • #32438 - track LLM completion usage and spend for /v1/rag/query • #33644 - resolve router_settings.plugins dotted paths and load plugins from installed packages • #33719 - self-heal on missing thinking-signature errors from Bedrock/Vertex • #33714 - bill Fireworks AI prompt-cache hits at the cache_read rate • #33565 - stop stale auth-cache re-publish to Redis so key updates propagate across replicas • #33554 - model_armor: restore reference attachments and remove the attachment count cap • #33533 - surface Gemini grounding toolUsePromptTokenCount in Usage