Throughput-Optimized Deployment Profile

A model deployment profile that maximises sustained queries per second, e.g., via FP8 precision and multi-GPU tensor parallelism, at the cost of higher per-request latency.

Responsibility. Configures an inference microservice for maximum throughput on the available GPUs.

Also known as: vllm-fp8-throughput-optimized, High-throughput profile, Multi-GPU tensor-parallel profile

Variant of Model Deployment Profile abstract

When to choose. Choose when maximising queries per second matters more than minimum time-to-first-token and multiple GPUs are available.