Infrastructure · Software component

Least-Connections Load Balancer

Software componentInfrastructureInfrastructurearc:LeastConnectionsLoadBalancer

A load balancer that tracks in-flight requests per replica and sends each new request to the replica with the fewest active requests, adapting to variable request durations.

Responsibility. Routes each request to the replica currently handling the fewest active requests.

Also known as: Least connections

Variant of Load Balancer abstract

When to choose. Choose as the default when request durations vary widely (e.g., 90% under 2 s, 10% over 10 s).

Direct neighbourhood (hover for relationship types)

Relationships

receives data from dynamic

Container Health Prober Ch4.7

alternative to variability

Design guidance

SHOULD be the default algorithm for agent workloads with highly variable processing times.
MUST couple connection counting with health checks and reset counts for crashed replicas, otherwise a dead replica appears idle and attracts all traffic.

Quantitative guidance

As stated by the sources; verify before use.

Same document workload: least connections balanced replicas at 60-70% CPU and cut P95 latency from 45 s to 28 s (within SLO) (Ch4.7).

Classification

Patterns: Least-connections routing
Quality attributes: Performance efficiency (ISO/IEC 25010)Flexibility (ISO/IEC 25010)
Risks mitigated: Long-running requests clustering on one replica

Sources

Ch4.7: T. Nguyen, "Scaling Strategies," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.7. ISBN: 9798244538229.