Inference platforms & gateways
15 systems in the inference platforms & gateways category of the AI Agent Infrastructure Landscape, grouped by maturity tier.
Tier 1 — battle-tested (10)
- Anyscale Ray-based AI platform
Commercial platform built on Ray — distributed training, fine-tuning, serving. Anyscale Endpoints provides hosted OSS inference; ray cluster is the substrate underneath OpenAI's training stack.
- Baseten ML model serving cloud
Inference platform for ML models — Truss package format, Chains for multi-model workflows, autoscaling GPU/CPU serving. Targeted at production teams shipping LLM/diffusion endpoints.
- Fireworks AI Fast OSS LLM inference cloud
Fast inference platform for OSS LLMs — custom Cuda kernels, speculative decoding, multi-LoRA serving. Targets latency-sensitive production use cases.
- LiteLLM Universal LLM API + Proxy
BerriAI's open-source LLM gateway — unified OpenAI-format API for 100+ providers; LiteLLM Proxy adds budgets, fallbacks, observability, rate-limiting.
- Modal Serverless compute / inference
Serverless cloud for AI/ML — Python-decorator workflow defines GPU/CPU functions deployed to managed infra; widely used for training jobs, batch inference, and agent backends.
- OpenRouter Multi-provider LLM gateway
Unified API + marketplace for 300+ models — pay-as-you-go credits, automatic fallback, model arena. Used as default LLM router by many JS/TS agents.
- Replicate Model-as-API inference cloud
Hosted-API platform for OSS models — Cog format packages models with their environment; thousands of community models behind a single REST API.
- TensorRT-LLM NVIDIA inference compiler for LLMs
NVIDIA's TensorRT extension for LLM inference — compiles transformers to optimised kernels for Hopper/Blackwell. Used in NIM and downstream deployments.
- Together AI Inference + fine-tuning cloud
Cloud platform for OSS LLM inference and fine-tuning — serverless inference, dedicated endpoints, fine-tuning API, training cluster.
- vLLM Fast LLM inference engine
Open-source LLM inference engine with PagedAttention — high-throughput batching, paged KV-cache. UC Berkeley origin; de-facto OSS inference stack.
Tier 2 — production-ready (5)
- OctoAI (now NVIDIA) GenAI inference (acquired)
Inference cloud spun out of OctoML — acquired by NVIDIA in 2024. Originally TVM-compiler heritage; offered API for OSS LLM/diffusion serving.
- Portkey AI Production AI gateway
AI gateway + observability — load balancing across 200+ LLMs, retries, caching, eval, prompts. Open-source Gateway + commercial Portkey Cloud.
- RunPod GPU cloud for AI
GPU-cloud platform — Pods (long-running), Serverless (per-request), and Hub (community templates). Popular for fine-tuning and inference of OSS models.
- SGLang Structured LLM-program runtime
Fast LLM serving framework with RadixAttention (cache reuse across requests with shared prefixes) + structured programming primitives. UC Berkeley.
- Text Generation Inference (TGI) HF inference server
Hugging Face's production inference server for LLMs — continuous batching, FlashAttention, quantization (bitsandbytes, GPTQ).