Red Hat AI Inference: The open foundation for enterprise AI

Red Hat® AI Inference optimizes inference across hybrid cloud environments, acting as the engine for agentic AI and internal Model-as-a-Service (MaaS) patterns. This solution provides the operational control organizations need to run any model on any accelerator and scale predictably. With AI Inference, central IT and platform teams become the organization's AI provider, taking advantage of available resources to serve more users and agents.

As part of the Red Hat AI portfolio, AI Inference is powered by vLLM, a high-performance inference runtime that lets users run any AI model on any hardware accelerator across datacenters, clouds, and edge environments. To help IT and platform teams scale AI workloads efficiently and manage token economics, AI Inference includes llm-d, which intelligently distributes inference processing across a fleet of accelerators, preventing bottlenecks and maximizing compute efficiency.

AI Inference accelerates time to value with a curated collection of validated, optimized open models from the Red Hat AI repository hosted on Hugging Face. These models are ready for production deployment with improved efficiency and no loss of accuracy. Advanced model compression capabilities help reduce hardware requirements and costs through techniques like quantization and speculative decoding, applied to both foundational and custom models.

The platform exposes gen AI-specific telemetry—from time-to-first-token (TTFT), key-value (KV)-cache hit rate, throughput, and graphics processing unit (GPU) utilization—to existing monitoring dashboards. It gives organizations the operational transparency they need to meet service-level objectives and control costs.

AI Inference runs on Red Hat OpenShift® and third-party Kubernetes distributions.

Table 1. Key benefits

Powered by vLLM and llm-d, AI Inference delivers a fully integrated platform that makes the most of inference at both the individual accelerator and the entire infrastructure.

• Open hybrid cloud runtime. Run your choice of models across various accelerators on Kubernetes and Linux environments—in a datacenter, a cloud, or at the edge.
• Distributed inference. Take advantage of a fully integrated platform powered by llm-d to route and balance inference traffic across a fleet of accelerators for consistent performance and better infrastructure utilization.
• Model-optimization toolkit. Reduce hardware requirements and costs while maintaining accuracy through techniques like quantization and speculative decoding—applied to both foundational and custom models.
• Validated model repository. Access a curated collection of leading gen AI models— validated and maximized for AI Inference—from the Hugging Face repository. These models are ready for production deployment with increased efficiency and preserved accuracy.
• Enterprise Kubernetes deployment. Deploy distributed inference on Red Hat OpenShift and third-party Kubernetes platforms. Third-party deployments are covered under Red Hat’s third-party support policy.

Learn about how AI Inference helps organizations expand token economics, scale predictably, and run their preferred choice of models on any accelerators and cloud.

• Visit the Red Hat AI Inference product page.
• Try AI Inference with a no-cost, 60-day trial.