Inference platforms & gateways

Commercial platform built on Ray — distributed training, fine-tuning, serving. Anyscale Endpoints provides hosted OSS inference; ray cluster is the substrate underneath OpenAI's training stack.

Inference platform for ML models — Truss package format, Chains for multi-model workflows, autoscaling GPU/CPU serving. Targeted at production teams shipping LLM/diffusion endpoints.

Fast inference platform for OSS LLMs — custom Cuda kernels, speculative decoding, multi-LoRA serving. Targets latency-sensitive production use cases.

BerriAI's open-source LLM gateway — unified OpenAI-format API for 100+ providers; LiteLLM Proxy adds budgets, fallbacks, observability, rate-limiting.

Serverless cloud for AI/ML — Python-decorator workflow defines GPU/CPU functions deployed to managed infra; widely used for training jobs, batch inference, and agent backends.

Unified API + marketplace for 300+ models — pay-as-you-go credits, automatic fallback, model arena. Used as default LLM router by many JS/TS agents.

Hosted-API platform for OSS models, part of Cloudflare since 2025-12-01 (still runs as a distinct brand) — Cog format packages models with their environment; thousands of community models behind a single REST API.

NVIDIA's TensorRT extension for LLM inference — compiles transformers to optimised kernels for Hopper/Blackwell. Used in NIM and downstream deployments.

Cloud platform for OSS LLM inference and fine-tuning — serverless inference, dedicated endpoints, fine-tuning API, training cluster.

Open-source LLM inference engine with PagedAttention — high-throughput batching, paged KV-cache. UC Berkeley origin; de-facto OSS inference stack.

Inference cloud spun out of OctoML — acquired by NVIDIA in 2024. Originally TVM-compiler heritage; offered API for OSS LLM/diffusion serving.

AI gateway + observability — load balancing across 200+ LLMs, retries, caching, eval, prompts. Open-source Gateway + commercial Portkey Cloud. Acquired by Palo Alto Networks (closed about 2026-06); now sold as Prisma AIRS AI Gateway (GA …

GPU-cloud platform — Pods (long-running), Serverless (per-request), and Hub (community templates). Popular for fine-tuning and inference of OSS models.

Fast LLM serving framework with RadixAttention (cache reuse across requests with shared prefixes) + structured programming primitives. UC Berkeley.

Hugging Face's LLM inference server (continuous batching, FlashAttention, quantization). Entered maintenance mode, then archived 2026-03-21; HF recommends vLLM, SGLang, llama.cpp or MLX instead.

Open-source, Envoy-powered control plane giving agents one OpenAI-compatible API for every model and one router for MCP tools, with provider credentials, failover and token limits. It was formerly Envoy AI Gateway and is now an Agentic A…

Open-source Rust proxy for AI agents covering LLM routing with credentials and failover, native MCP and A2A traffic, and token-usage limits. Contributors include Solo.io, Microsoft and AWS, and kgateway uses it as its AI data plane.

Open-source Go AI gateway from Maxim AI with one OpenAI-compatible API for many models, failover, load balancing, semantic caching, virtual keys and budgets, and a built-in MCP gateway for agents.

Gateway in front of 20+ model providers with an OpenAI-compatible unified API, caching, rate and spend limits, dynamic routing, guardrails, DLP and BYOK, plus logging and analytics.

Enterprise AI gateway providing unified access to 1,600+ models with policy control, failover, routing, guardrails, budgets and monitoring. It can be deployed in VPC, on-prem or air-gapped environments.

Vercel's unified endpoint for text, image, video and speech models across providers, used from the AI SDK and OpenAI-compatible clients.