SGLang

Fast LLM serving framework with RadixAttention (cache reuse across requests with shared prefixes) + structured programming primitives. UC Berkeley.

At a glance

Optimised for: searched not found

Anti-fit: searched not found

searched not found

searched not found

~9k stars; LMSYS-affiliated; competitor to vLLM for OSS inference engines; Apache-2.0.

Other inference platforms & gateways in the catalog, ranked by inbound references.

LiteLLM T1
BerriAI's open-source LLM gateway — unified OpenAI-format API for 100+ providers; LiteLLM Proxy adds budgets, fallbacks, observability, rate-limiting.
vLLM T1
Open-source LLM inference engine with PagedAttention — high-throughput batching, paged KV-cache. UC Berkeley origin; de-facto OSS inference stack.
Anyscale T1
Commercial platform built on Ray — distributed training, fine-tuning, serving. Anyscale Endpoints provides hosted OSS inference; ray cluster is the substrate underneath OpenAI's training stack.
Baseten T1
Inference platform for ML models — Truss package format, Chains for multi-model workflows, autoscaling GPU/CPU serving. Targeted at production teams shipping LLM/diffusion endpoints.
Fireworks AI T1
Fast inference platform for OSS LLMs — custom Cuda kernels, speculative decoding, multi-LoRA serving. Targets latency-sensitive production use cases.
Modal T1
Serverless cloud for AI/ML — Python-decorator workflow defines GPU/CPU functions deployed to managed infra; widely used for training jobs, batch inference, and agent backends.

vLLM competes with — ; LMSYS-affiliated; competitor to vLLM for OSS inference engines; Apache-2.0.