Catalog Systems Inference platforms & gateways TensorRT-LLM

TensorRT-LLM

https://github.com/NVIDIA/TensorRT-LLM

NVIDIA's TensorRT extension for LLM inference — compiles transformers to optimised kernels for Hopper/Blackwell. Used in NIM and downstream deployments.

Active · Verified 2026-10-27 · fresh as of 2026-10-27 · 18 cited facts, 1 with a third-party source · 1 recorded change · Suggest a correction

At a glance

Type
NVIDIA inference compiler for LLMs repo fix
Tier
T1
Section
Inference platforms & gateways
Created
2023 repo fix
Latest release
v1.2.1 (2026-04-20) — tensorrt-llm; GitHub pre-release v1.4.0rc1 (2026-10-22) registry fix
License
Apache-2.0 repo fix
GitHub
NVIDIA/TensorRT-LLM — ~15.0k stars, last commit 2026-10 repo fix
Pricing
searched not found fix
Funding
searched not found fix
Freshness
2026-10-27 fix
Layer
model-infra repo fix
Licensing
open-source repo fix
Deployment locus
local repo fix
Platform vs product
platform repo fix
SDK languages
python repo fix
Protocols
openai-compatible vendor fix
Developer activity
OpenRank 215.57 (12-mo avg 249.91) · 279 participants — 2026-09 third-party fix

When to use

Optimised for: searched not found

Anti-fit: searched not found

Pros & cons

Pros

searched not found

Cons

searched not found

Claims & capabilities

searched not found

Technical surface

API surface
searched not found
Backend storage
not applicable — not a memory product
Deployment
Self-hosted only — inference library/runtime for NVIDIA GPUs
Embedding model
not applicable — not a memory product
Multi-tenancy
searched not found
MCP
searched not found
A2A
searched not found
OpenTelemetry
searched not found

Similar systems

Other inference platforms & gateways in the catalog, ranked by inbound references.

  • LiteLLM T1

    BerriAI's open-source LLM gateway — unified OpenAI-format API for 100+ providers; LiteLLM Proxy adds budgets, fallbacks, observability, rate-limiting.

  • OpenRouter T1

    Unified API + marketplace for 300+ models — pay-as-you-go credits, automatic fallback, model arena. Used as default LLM router by many JS/TS agents.

  • Together AI T1

    Cloud platform for OSS LLM inference and fine-tuning — serverless inference, dedicated endpoints, fine-tuning API, training cluster.

  • Fireworks AI T1

    Fast inference platform for OSS LLMs — custom Cuda kernels, speculative decoding, multi-LoRA serving. Targets latency-sensitive production use cases.

  • vLLM T1

    Open-source LLM inference engine with PagedAttention — high-throughput batching, paged KV-cache. UC Berkeley origin; de-facto OSS inference stack.

  • Portkey AI T2

    AI gateway + observability — load balancing across 200+ LLMs, retries, caching, eval, prompts. Open-source Gateway + commercial Portkey Cloud. Acquired by Palo Alto Networks (closed about 2026-06); now sold as Prisma AIRS AI Gateway (GA 2026-07-16).

Sources (17 cited facts, 1 third-party)

Each fact links to the page it came from; the badge says who published it (how to read sources).

What changed

Changes to this record's lifecycle, licence, release, pricing and hosting facts, from the catalog's git history. Subscribe (Atom)

  1. 2026-10-27
    • Latest release first recorded as “v1.2.1 (2026-04-20) — tensorrt-llm; GitHub pre-release v1.4.0rc1 (2026-10-22)” source
    • Deployment first recorded as “Self-hosted only — inference library/runtime for NVIDIA GPUs” source
    #272

Related systems

Referenced by (1)

  • LlamaIndex (framework) integrates with — LlamaIndex has an official LLM integration package llama-index-llms-nvidia-tensorrt in its repo. (verified 2026-10-29)