Variation points

Choose for simple, deterministic tasks where a straightforward tool call suffices and latency or budget constraints cannot absorb multi-step reasoning or planning overhead.

Choose when an agent must integrate an extensive, growing catalog of capabilities (tens of functions across many domains) whose composition cannot be pre-programmed and new capabilities must be added without changing orchestration code. Avoid for focused agents with fewer than five capabilities, or for workflows needing iterative refinement loops, conditional branching on intermediate results, or strictly deterministic execution (prefer explicit state-graph orchestration).

Choose for complex workflows with mostly predictable steps in stable environments, resource-constrained settings, or supervisor-worker coordination; avoid for highly dynamic, real-time, simple, or open-ended exploratory tasks.

Choose when solution paths are unpredictable and require adaptive investigation (research, debugging, non-standard support); avoid for simple deterministic, latency-critical, budget-constrained, or formally verified tasks.

Choose in heterogeneous systems where agent availability fluctuates or several agents offer similar capabilities with different trade-offs, requiring runtime discovery.

Choose when task delegation boundaries are well-defined and agent capabilities are static, so no runtime capability discovery is needed.

Choose when high throughput is critical, agents run asynchronously at variable rates, resilience to temporary failures is essential, state changes must reach many subscribers, and eventual consistency is acceptable.

Choose when immediate feedback or strong consistency is essential, interactions are brief and synchronous, the agent network is small, and explicit communication trails aid compliance.

Choose when agent dependencies can be declared at workflow-configuration time; static validation makes runtime deadlock structurally impossible and fails fast, with no scanning cost or detection latency.

Choose only when dependencies cannot be declared in advance; it requires periodic scanning of active agents and lets a deadlock persist until the next scan detects it.

Choose for relatively static knowledge bases where daily updates suffice, and for structured sources with efficient timestamp-based change detection.

Choose for rapidly changing domains (e.g., news, stock trading) and semi-structured event sources where near-real-time freshness justifies added complexity.

Choose for initial population, after chunking or schema parameter changes, or to remediate quality issues from earlier permissive filters.

Choose as the default for scheduled (hourly or daily) runs that keep the knowledge base fresh.

Choose when workflow stages can be pre-partitioned by clear decision criteria, e.g., routine verification to the agent and judgment-heavy decisions to humans.

Choose when no fixed ownership is appropriate and initiative should flow dynamically to whichever party has the relevant capability, capacity and confidence at each stage.

Choose when execution hits a specific ambiguity or misunderstanding that a brief clarifying exchange can resolve before control returns to the primary party.

Choose when one party holds information the other needs but should not control when or how it is acted upon; the recipient keeps full initiative.

Choose for long-running processes (hours or days) whose state must survive instance failure and be resumable by any instance.

Choose for multi-agent or multi-instance systems needing shared, low-latency state for horizontal scaling.

Choose when state needs both persistence and relational context (e.g., ticket, related tickets, consulted articles).

Choose for short-lived, agent-local workflows that complete in seconds to minutes on a single instance; state is lost if the instance fails.

Choose when persistence is needed but relational queries over state are not.

Choose when state must converge regardless of the order of concurrent updates.

Choose when instances can detect conflicting updates and retry them.

Choose when instances must hold exclusive access before modifying state.

Choose when allocation should adapt dynamically to agents' workload, expertise and confidence, or when agents compete for shared resources.

Choose when collaborators must be discovered at runtime by advertised capability rather than hardcoded.

Choose when task types and agent capabilities are stable and predictable routing matters more than dynamic allocation.

Choose when LLM reasoning about the next step must be constrained by explicit rules.

Choose when deciding what to do next requires reasoning, e.g., tasks without predetermined solution paths.

Choose when decision logic can be expressed as explicit rules over state fields.

Choose over spawning all agents concurrently whenever agents consume each other's outputs, since probabilistic start ordering violates dependencies under load.

Choose when multi-step agent workflows need branching, error handling or durations of hours or days that exceed individual function timeout limits.

Choose when agents can analyse the same source information independently; reduces both tokens and latency.

Choose for enterprise agent ecosystems with dozens of specialized capabilities, evolving service catalogs and existing-system integration needing central orchestration; avoid for focused single-purpose agents where plugin abstraction is over-engineering.

Choose when a task decomposes into a fixed sequence of processing stages, each needing a different prompting strategy, with intermediate outputs inspected between stages; for simple linear workflows without branching, simple function composition suffices.

Choose when tasks lack predetermined solutions and need conditional branching (logic trees), crash recovery and traceable transitions; when workflows exhibit decision-tree routing on classification or intermediate results, when the graph is valuable documentation, when observability matters, or when workflows change frequently. Avoid for simple linear workflows or when the learning-curve cost is unjustified for small teams or short-lived projects.

Choose when human readability matters, clients are diverse (any HTTP client), performance requirements are moderate, and REST infrastructure can be reused.

Choose when throughput and latency are critical, type safety matters, bidirectional streaming is needed, and both client and server implementations are controlled.

Choose when the upstream source can signal publication of updates, so freshness need not wait for a TTL.

Choose when the source updates on a known schedule and no change notifications are available.

Choose when profiling shows GPU idle gaps during I/O-bound tool calls in multi-step agent loops (e.g., ReAct) under concurrent load.

Choose for independent data fetching from multiple sources where latency matters and tools share no mutable state and have no coordination requirements.

Choose when tools have dependencies (Step N+1 needs Step N's output), when guaranteed execution ordering is needed, when tools modify shared data, or when orchestration logic depends on seeing results sequentially.

Choose when the agent needs direct querying or updating of structured data stores.

Choose for distributed systems needing reliable, buffered delivery that survives temporary component failure.

Choose for synchronous CRUD operations and cloud-service integration; the most common mechanism.

Choose for event-driven, asynchronous workflows where external systems should notify the agent instead of being polled.

Choose when many distinct problem types need structured reasoning but resources are insufficient to hand-craft examples for each, or when the question distribution is unknown or evolving; accept added pipeline complexity and slightly lower quality than hand-crafted few-shot.

Choose when reasoning structure and consistency matter more than deployment speed (e.g., medical diagnosis, legal analysis, financial modelling) and expert time can be invested in two to three exemplary demonstrations per problem type.

Choose when immediate deployment is needed across diverse, unpredictable problem types without time or budget to craft examples, accepting variable reasoning quality on ambiguous problems.

Choose when fine-grained control is needed at scale: explicit priorities for business-critical rules, automatic specificity for most rules, and recency so updated rules take effect without re-prioritising.

Choose when simplicity and determinism matter most and the rule base can be maintained as a carefully sequenced decision list with specific rules placed before general fallbacks.

Choose when knowledge evolves rapidly and newer rules encode more current understanding (e.g., new fraud patterns); requires rule modification timestamps and clear documentation.

Choose when explicit control is needed, e.g., safety-critical systems where emergency rules must pre-empt optimisation or routine rules; requires careful priority assignment and maintenance.

Choose when rules form natural general-to-specific hierarchies where nuanced policies should override broad defaults; needs a separate policy for equal-specificity conflicts.

Choose when binary goals suffice and no trade-offs exist; offers faster development and easier debugging.

Choose when real-time decisions must be made in milliseconds and a 'good enough' answer is acceptable, and when the heuristic's systematic bias causes minimal harm in the context (e.g., routine, clear-cut cases).

Choose when the problem mixes decision types with incompatible requirements, needs both adaptation and transparency, is safety-critical in an open world, or combines structured knowledge with unstructured perception (high decomposability, hard constraints, mixed data availability).

Choose when the strategy space is too large to specify manually, the environment is dynamic or partially unknown, and sufficient training data and computation are available.

Choose for sequential decisions where current actions affect future options and the state is (or has been augmented to be) Markovian.

Choose when the problem is non-Markovian or history-dependent (e.g., treatment response depending on prior treatments) and state augmentation is insufficient.

Choose when multiple stakeholders hold diverse or disputed preferences, when trade-offs must be explored before committing, or when objectives are incommensurable and resist a common scale.

Choose when rules capture the domain completely and consistently produce correct decisions, and transparency and predictability are paramount.

Choose when decisions involve complex trade-offs among multiple objectives with no clear priority ordering, probabilistic outcomes, or continuous optimization where the degree of success matters more than binary goal achievement; use scalarized weights when a single decision-maker has clear preferences.

Choose when users need to know the minimal changes that would flip the decision (why-not / recourse).

Choose when users need the key factors behind a decision and their relative importance.

Choose when comparing the case to k similar past decisions conveys the rationale better than factor weights.

Choose when the decision model is opaque and a model-agnostic local approximation of top feature contributions is needed.

Choose when regulatory or safety rules must never be overridden by learned or utility scores.

Choose when training data showing which combinations produce the best outcomes is available.

Choose when component accuracies on similar cases are known and redundancy should catch errors through disagreement.

Choose when demonstrations must cover diverse problem types and reasoning chains are to be generated automatically rather than written manually.

Choose when the demonstration pool is large or noisy and a compact core subset satisfying sufficiency and necessity is needed.

Choose when accuracy gains justify per-query retrieval cost; computational cost becomes high on large demonstration pools.

Choose when state coordinates are available and per-node evaluation must be near-free; the sweet spot for grid and road pathfinding.

Choose only when inference is cheap relative to node expansion cost, so higher accuracy reduces total search time.

Choose when states are high-dimensional (images, sensors) or actions are continuous, so tables cannot be stored or visited.

Choose when the state-action space is small and discrete enough to enumerate and visit.

Choose in regulated industries where business rules and domain constraints can be encoded as formal logic and compliance must be verifiably met.

Choose when model internals are accessible and verification must reflect what actually happened inside the model rather than its verbal explanation; requires per-domain calibration.

Choose when domain-specific labeled correct/incorrect reasoning steps (human-annotated or synthetic) are available and higher in-domain accuracy is required, e.g., compliance, legal or policy reasoning.

Choose for mathematical, symbolic or other formally specified domains where errors have severe consequences (financial calculation, cryptographic protocols, safety-critical systems) and reasoning can be translated into formal statements.

Choose when reasoning contains arithmetic or algebraic manipulations that can be extracted and independently re-computed with a symbolic mathematics library.

Choose when labeled reasoning-correctness data is unavailable and moderate agreement with human judgment suffices, using a properly prompted general LLM.

Choose when reducing self-reinforcing bias justifies added complexity and cost; requires critic criteria aligned with downstream consumers.

Choose when implementation simplicity matters and the generating model has adequate domain knowledge; beware it may reinforce its own misconceptions.

Choose when errors must be caught before they propagate through the chain; substantially improves final reasoning quality compared to single-pass generation with post-hoc evaluation.

Choose when failures are global or rare (<10% of executions), unpredictable, memory is constrained, spaces are small, latency can be tolerated, or optimality matters more than continuity.

Choose when failures are frequent (>30%), predictable from observable conditions (time of day, sensor state), stable, few (2-5 modes), and replanning pauses of 3-5 s are intolerable.

Choose when changes are frequent but localized edge-cost modifications on a stable topology, state spaces are large, memory permits search-tree persistence, and optimal solutions matter.

Choose when discrepancies are localized, adaptation must be fast, and plan stability matters (multi-robot committed trajectories, user-communicated ETAs); good-enough solutions suffice.

Choose for safety-critical systems, medical treatment, retirement finance and other domains where worst-case outcomes carry severe consequences.

Choose only for small-stakes decisions with minimal outcome variance or explicitly risk-neutral domains aggregated across many independent events.

Choose for venture-style portfolios, innovation strategies with asymmetric upside, or winner-take-all competitive scenarios.

Choose for easy problems or small sample counts (k=3-5) where vote distributions are already clear and quality-assessment overhead is not justified.

Choose for medium-to-hard problems with larger sample counts (k≈10-40) or cost-constrained high-stakes deployments where extracting more signal from fewer samples justifies ~5% quality-scoring overhead.

Choose for code generation where implementations vary textually but test cases define correctness.

Choose when no trained value network is available and simulations are cheap, favouring heuristic rollout policies where domain knowledge exists.

Choose when training data exists or self-play can generate it and upfront training cost plus per-evaluation inference latency are acceptable.

Choose when the simplest structure suffices and clear step boundaries are needed for automated parsing and step-by-step evaluation.

Choose when evaluation must extract and assess specific reasoning components, e.g., whether verification actually checks results or conclusions quantify confidence.

Choose for simple sequential workflows (3-10 steps) or optimization-dominant problems where provably optimal solutions matter more than fast feasible plans and the action space is small.

Choose when the goal has naturally nested structure, reusable decomposition patterns exist, stakeholders need multiple abstraction views, or staged commitment under uncertainty is needed, and the world is stable during planning.

Choose for high-level decomposition of open-ended goals where broad knowledge matters and encoding deep domain methods in prompts would be too expensive.

Choose when heuristics are hard to design, the state space is enormous, near-optimal solutions are acceptable for faster computation, transitions are stochastic or partially observable, or asymmetric tree growth pays off (game playing, robotic task planning, chemical retrosynthesis, navigation among dynamic crowds).

Choose for one-shot planning problems with expensive forward-model simulations (e.g., computationally intensive physics), where lower memory use outweighs MCTS's progressive statistical refinement.

Choose for highly dynamic domains where strategic decisions become invalid before tactical decomposition finishes, or where problem structure is undefined.

Choose when a problem decomposes into independent subproblems whose solutions must be synthesized, or intermediate results should inform rather than compete with each other.

Choose when exploration and backtracking matter but synthesis across branches does not.

Choose for open-ended exploration of vast, unstructured solution spaces where large conceptual leaps are needed (e.g., creative writing plans).

Choose for highly constrained, structured solution spaces where candidates should collectively and systematically cover the space (e.g., arithmetic operations).

Choose when intermediate states can be checked for objective feasibility (e.g., mathematical reachability, constraint satisfaction) and absolute thresholds are needed for pruning.

Choose when evaluation criteria are subjective or rating scales poorly defined (e.g., narrative coherence), or when a single best candidate must be selected.

Choose for moderate conversation lengths (roughly 10-20 exchanges) where perfect recall of prior turns matters; switch to sliding-window or summarizing variants when history approaches token limits.

Choose for long-running conversations where historical context determines current interpretation and users reference earlier turns, accepting retrieval latency and the need to decide which history to fetch.

Choose for goal-oriented conversations in which certain turns establish critical constraints or preferences that later turns build on, provided an accurate relevance model is available.

Choose as the simplest bound when losing early conversation detail is acceptable.

Choose when early context (user goals, key facts, binding decisions) remains relevant later and naive truncation would lose it.

Choose for multi-step agents whose trajectories accumulate dead ends, restatements and stale turns.

Choose for large-scale deployments (100+ instances), frequent autoscaling that would churn sessions, or when no session may be lost on instance failure.

Choose when latency requirements are extreme (sub-100 ms), scale is modest (tens of instances), and session recreation after instance failure is acceptable.

Choose when every action might be significant later (e.g., debugging complex technical problems where the solution hinges on minor details); avoid for routine standard-pattern interactions where storage overhead outweighs retrieval value.

Choose when most interactions follow standard patterns and storage must be reduced while preserving the most informative experiences (failures, unusual outcomes, high-impact states).

Choose when thoughts need multiple parents so independent reasoning chains can merge.

Choose when reasoning branches diverge but never need to reconverge.

Choose when a simple gate suffices and the model's self-assessed certainty is reasonably calibrated; combine with pattern rules and periodic audits to catch miscalibration.

Choose when query types differ in their optimal retrieval strategy and more sophisticated routing than a single confidence gate is warranted.

Choose when answers must combine sub-answers from multiple documents and cite the supporting sources for verification.

Choose when a corpus contains exact, near and semantic duplicates together (typical in production); orders cheap levels first to minimise cost.

Choose when duplicates are identical after cleaning and pairwise similarity comparison would be computationally prohibitive.

Choose when near-duplicates differ by minor edits, dates or formatting (versioning, format conversion); tune the threshold between conservative and aggressive.

Choose when content varies slightly (whitespace, minor edits, reformatting) so exact hashing misses duplicates.

Choose when duplicates express the same information in different words (rewrites, summaries, translations); most expensive level.

Choose for real-time or semi-structured event sources when near-instant knowledge updates justify the added complexity.

Choose for unstructured document repositories such as shared drives, Git repositories or cloud object storage.

Choose for semi-structured sources exposed through REST or GraphQL APIs (e.g., ticketing, CRM, wiki platforms).

Choose for structured sources (relational/NoSQL databases, warehouses) whose schemas and audit timestamps enable selective, batch delta queries.

Choose only when content has no usable structure or timing data; the chapter presents it as the naive baseline that fractures semantic units.

Choose when document structure (sections, subsections) must be preserved so retrieved chunks keep their structural context, accepting more complex retrieval logic.

Choose when important context spans chunk or section boundaries and continuity must be preserved, accepting multiplied storage.

Choose for text documents with structure, splitting at section headers and paragraph breaks rather than arbitrary token counts.

Choose for audio transcripts lacking paragraph breaks or headers, when retrieved segments must link back to exact moments in the recording.

Choose when chunk coherence matters more than uniform chunk size.

Choose only when GPU infrastructure is unavailable and query volume is low (below the few-thousand-queries-per-day GPU break-even); pair with an embedding cache.

Choose when building initial development or baseline systems, when no data-sovereignty or air-gap constraint applies, and when per-token API cost is acceptable for the query volume.

Choose for rapid deployment on existing text RAG with mostly general imagery (photos, simple diagrams); avoid for information-dense charts needing precise values or OCR.

Choose when each modality needs its best-suited embedding model (e.g., domain BERT for text, CLIP for images, DePlot for charts) and components must be upgradable independently.

Choose when data sovereignty, zero-trust or air-gapped operation is required, when long documents must be embedded, or when volume is high enough (beyond a few thousand queries daily) that GPU per-query efficiency beats API pricing; requires operating GPU and serving infrastructure.

Choose when all modalities are grounded to text (captions, linearized tables, transcripts) so a single text embedding model and text vector index suffice.

Choose for lower-precision applications; matches by edit distance with manual review of uncertain mappings.

Choose for high-precision requirements; resolves mentions to knowledge-base IDs via an entity linking service.

Choose when the same real-world entity carries different identifiers across several source systems and string similarity alone is insufficient to unify them.

Choose when a simple rule suffices to spot charts, e.g., images containing axis labels, legends or grid patterns, as in the financial-report worked example.

Choose when routing at scale and a vision-language model is already deployed, reusing it for meta-classification instead of adding a separate classification model.

Choose for information-dense images (charts, plots, graphs, tables) where users need exact figures and computation, e.g., financial or benchmark charts.

Choose for natural images, complex scenes, and diagrams with readable text where descriptive detail (objects, spatial relationships, context, visible text) matters more than exact numbers.

Choose when rapid development and schema flexibility (properties on nodes and edges, multiple labels, schema evolution) matter more than formal semantics; the industry-standard choice for agent systems.

Choose when interoperability across systems requires standardized ontologies and formal W3C semantics (e.g., academic knowledge bases like Wikidata).

Choose as a reliable baseline for explicit subject-verb-object relationships; it misses implicit relationships stated without verbs.

Choose when implicit relationships (e.g., appositives like 'Google, a major Anthropic investor') must be captured for higher recall.

Choose instead of full hybrid fusion for extremely latency-sensitive applications (sub-50 ms p99) that still need keyword precision.

Choose in latency-sensitive RAG agents where retrieval dominates time-to-first-token.

Choose when queries require following typed relationships across multiple hops (e.g., conflict-of-interest chains, investor overlap).

Choose when queries need both semantic similarity and relationship traversal.

Choose alongside dense vector search when sparse keyword matching can return results faster, run in parallel to cut retrieval latency.

Choose when queries need conceptual understanding or semantic similarity across varied terminology (simple Q&A, documentation search, customer support).

Choose for cost-sensitive applications with moderate accuracy needs and as the default starting point to establish baseline retrieval performance; open-source variants when self-hosting without API dependencies is required.

Choose for specialized applications with technical or industry-specific terminology, where domain models outperform general-purpose ones.

Choose for broad coverage across diverse content types.

Choose when retrieval accuracy directly drives user experience and passages are short to medium length, justifying higher per-token cost.

Choose for enterprise deployments processing long technical documents, contracts or papers that routinely exceed 8,000 tokens, typically self-hosted for data sovereignty.

Choose for search-specific applications over short passages where ranking precision and near-duplicate discrimination matter more than long context.

Choose only for small collections (under about 100K vectors) where exact results are needed.

Choose for large collections (beyond ~10 million vectors) where search latency outweighs higher memory usage.

Choose as a balanced default for mid-size collections needing good recall with manageable memory.

Choose when collections and query rates are modest enough that CPU-based index building and search latency are acceptable.

Choose when query throughput, storage efficiency, multi-tenancy or on-premises constraints demand capabilities simpler vector databases lack and an infrastructure team can absorb the operational complexity.

Choose for local development, notebooks, proofs of concept, education and embedded applications with modest scale (well under one million vectors).

Choose when filtered vector search is the core workload (recommendations under business-rule constraints, document search with access control, analysis within temporal windows); available self-hosted or managed.

Choose for large (up to billion-scale) collections, continuous ingestion with frequent index rebuilds, or multi-hop agent workflows querying many times per request under tight latency.

Choose when rapid deployment and operational simplicity outweigh infrastructure control and cloud-hosted storage is acceptable.

Choose when an organization already operates the relational database at scale, datasets are moderate (<1M vectors), and transactional consistency across relational and vector operations is required.

Choose when a team comfortable operating infrastructure needs to deploy anywhere (on-premises for data sovereignty, own cloud for cost control) without sacrificing developer experience.

Choose for broad client compatibility, debugging and moderate query volumes (below ~100 requests per second).

Choose when query volume exceeds ~100 requests per second or real-time applications need the lowest latency.

Choose by default for any transformer deployment, especially long contexts on memory-bandwidth-limited GPUs; benchmark against standard attention for very short sequences.

Choose when much larger effective contexts are needed and occasionally missing long-range dependencies that full attention would capture is acceptable.

Choose only when benchmarks show fused kernels are slower, e.g., very short sequences (under 256 tokens) with small batches.

Choose when memory is constrained or batch size must be preserved (e.g., edge devices, high-batch serving).

Choose for complex queries or premium users where the quality improvement justifies cost and sharding complexity.

Choose for complex queries where accuracy improvements justify the substantially higher token cost.

Choose for simple or domain-specific queries where benchmarking shows it meets quality thresholds.

Choose for queries classified as moderately complex.

Choose for large language model generation, where the engine's continuous or in-flight batching outperforms request-level batching.

Choose for uncommon or heterogeneous GPUs (RTX 4090, A10G, V100, T4), edge and mixed research clusters, or development prioritizing iteration speed.

Choose when pre-compiled engines exist for the deployed GPU (e.g., A100 40/80GB, H100 80GB, L40S 48GB).

Choose for non-autoregressive models (classifiers, encoders, vision, tree-based models, custom Python logic) that benefit from server-side dynamic batching; OpenVINO for Intel CPU or edge CPU-only infrastructure.

Choose for models on backends that accept batched tensors (TensorFlow, PyTorch, ONNX Runtime, TensorRT); do not use for engines with internal continuous batching.

Choose for autoregressive LLM generation where sequences finish at different times and latency and throughput must be balanced without artificial batch-assembly delays.

Choose when the served model is stateful (recurrent networks, language models with hidden state, streaming speech recognition, conversational models carrying context) so every request of a sequence must reach the same model instance.

Choose for offline batch workloads (overnight reports, bulk document processing, scheduled evaluation) where no user waits.

Choose for production with SLA or compliance requirements (SOC 2, HIPAA, FedRAMP) and mission-critical services where 10-15% latency gains justify reduced flexibility.

Choose for research, experimentation, custom fine-tuned model pipelines, multi-model task switching and prototyping where experimentation velocity outweighs production stability.

Choose for mixed or semi-interactive workloads, such as customer service chatbots where users tolerate 300-500 ms latency and traffic allows some batching.

Choose when serving heterogeneous users with tiered SLAs (premium <50ms, standard <200ms, best-effort batch) on shared infrastructure.

Choose for interactive query-time workloads such as visual question answering where sub-second responses are required.

Choose for interactive applications (chat, code completion, real-time moderation) with short, factual responses where deterministic, concise output is acceptable.

Choose for offline RAG preprocessing (bulk captioning or extraction) where images processed per second matters more than per-image latency.

Choose (with buffers pre-allocated for maximum length) only when sequence lengths are bounded and predictable, accepting higher initial memory use.

Choose by default for production serving, especially workloads with high variance in generation length; benefit is minimal for fixed-length outputs.

Choose in production when recency, priority and recomputation cost must be balanced together.

Choose when all requests have similar priority.

Choose for interactive sessions where recent activity signals continued engagement and idle sessions can be suspended.

Choose when requests carry differentiated SLAs (e.g., premium versus free tier).

Choose when data residency, security policy, and cost considerations permit sending prompts to a cloud-hosted LLM provider.

Choose when data residency requirements, security policies, or cost considerations prevent using cloud-hosted LLM APIs, or for high-throughput tool-heavy agents.

Choose as the default when accuracy cannot be compromised (e.g., precise numerical reasoning where even 1% degradation is unacceptable).

Choose for maximum compression and throughput in use cases that accept ~3-7% accuracy loss.

Choose on GPUs with FP8 support when roughly doubled throughput with minimal accuracy loss is acceptable for the model.

Choose only when 2-5% accuracy loss is acceptable and maximum memory and cost reduction is needed.

Choose when 1-2% accuracy reduction is tolerable (e.g., conversational or human-reviewed moderation workloads) in exchange for roughly halved infrastructure cost; requires representative calibration.

Choose when targeting Ampere-or-newer hardware exclusively and near-FP32 accuracy is required without a quantization workflow.

Choose when query phrasing varies widely and higher classification accuracy justifies the added latency and cost of a classifier model.

Choose when query classes follow clear lexical patterns and zero added latency and cost matter more than robustness to phrasing variation.

Choose when audit requires persisting all outputs, capacity exceeds practical RAM, cache updates must be atomic with other data, or SQL querying of cached data is needed.

Choose for horizontally scaled systems with >20% query repetition or where cache must survive instance restarts.

Choose for single-instance deployments, session-scoped data, or sub-millisecond latency needs where even Redis overhead matters.

Choose when users ask the same questions in different wording and incremental savings exceed embedding costs.

Choose for cloud deployment needing a balance of accuracy and compute.

Choose for edge deployment on embedded GPU devices where memory and power are constrained.

Choose for demanding applications requiring the highest visual reasoning accuracy.

Choose when substantial multi-GPU infrastructure is available and all model weights are to be modified.

Choose when GPU memory is constrained and near-full-fine-tuning quality is needed by training only a small adapter.

Choose when expert demonstrations are available and autonomous exploration is too slow, costly, or unsafe.

Choose when a reward signal is available and the agent can safely explore (e.g., in simulation), with ample data and compute.

Choose when expert demonstrations (e.g., thousands of hours of human driving) are available and explicit preference specification is impractical.

Choose when users' selections among presented options and feedback are continuously observable, enabling personalization without explicit preference configuration.

Choose first for simpler preference-learning tasks where direct optimization suffices and lower compute is desired.

Choose for complex multi-dimensional preferences where explicit reward modeling provides interpretability benefits.

Choose when human preference must be grounded by verifiable correctness, e.g., to prevent optimizing toward fluent hallucinations or annotator bias.

Choose when harmlessness rewards must scale without human labels and reflect explicit, inspectable principles rather than implicit annotator preferences.

Choose when reward hacking via single-pattern exploitation is a concern and the cost of training multiple reward models is acceptable.

Choose for high-stakes decisions such as lending where the policy must satisfy a chosen fairness metric (e.g., equalized odds) while keeping legitimate business criteria.

Choose when desired behaviour is a subjective judgment (e.g., tone, trade-offs) that rigid rules cannot capture and annotator bias is controlled.

Choose when user segments legitimately hold divergent preferences and sufficient preference data can be collected for each segment.

Choose when an expert-provided initial rule set exists and should be refined from accumulated cases where its decisions proved wrong.

Choose when rules must be induced from historical labelled examples (e.g., past approved/denied applications) rather than from an existing expert rule set.

Choose when maximum performance and operational simplicity outweigh API-management features, authentication/rate limiting live elsewhere, and routing plus load balancing suffice.

Choose when centralised policy management, varied authentication schemes (OAuth2, JWT, API keys), traffic transformation or complex routing justify added latency and configuration complexity.

Choose when consistent latency matters more than cost (24/7 warm instances, no cold starts), Kubernetes already runs other services to amortise overhead, the team has strong Kubernetes expertise, continuous background processing is needed, or debugging requires persistent logs and long-term metric retention.

Choose when traffic shows high variance or long idle periods, deployment velocity is critical, the team lacks infrastructure expertise, cost must scale with actual usage, or the system processes discrete events rather than continuous streams.

Choose for latency-sensitive deployments with explicit response-time SLOs.

Choose for stateless agents with unpredictable, high-variance traffic (e.g., multi-tenant SaaS) that tolerate 60-90 s scale-up lag.

Choose for agents that primarily orchestrate external API or tool calls, where CPU does not reflect capacity.

Choose CPU for compute-bound agents dominated by inference or reasoning loops, and memory for agents holding large context windows or document embeddings.

Choose when traffic patterns are extremely predictable, making schedule-based capacity simpler and equally effective.

Choose for datasets under ~10 million documents, infrequent (monthly or rarer) processing, simple quality filters without deduplication, or limited capacity to manage GPU infrastructure.

Choose for billions of documents where CPU pipelines take multiple days, regular full reprocessing of large corpora, fuzzy deduplication at scale, or when GPU infrastructure already exists.

Choose for models under ~13B that fit on one GPU; scales nearly linearly.

Choose for training models whose weights plus optimizer state exceed GPU memory (e.g., 70B with 280 GB state on 4x80GB).

Choose for cross-node scaling over slower interconnects where memory efficiency matters more than per-sample latency.

Choose for memory-constrained models (30-40B+) and latency-sensitive inference, confined to a single-node high-bandwidth GPU interconnect domain.

Choose for specialised applications needing ultra-low latency when hardware programming expertise is available.

Choose for robotics and autonomous systems needing programmable acceleration across many model types, accepting higher power draw.

Choose for mobile and embedded products where dedicated AI acceleration must consume minimal power.

Choose for battery-powered devices running standard neural network architectures.

Choose for sensor-scale, battery-powered deployments where memory and power budgets are minimal.

Choose for large models (70B+), high-concurrency single-tenant deployments, or latency-critical workloads needing full memory bandwidth and large continuous batches.

Choose for multi-tenant SaaS with SLA guarantees, mission-critical agents, regulatory isolation needs, or production agents that must not be slowed by co-located batch jobs.

Choose for internal development, trusted co-located workloads, long-running batch jobs, or cost-constrained deployments that tolerate latency variance and cannot afford partitioning overhead.

Choose for 8+ GPU nodes and high-batch tensor-parallel inference where all-reduce bandwidth dominates.

Choose for smaller or lower-utilization deployments and data-parallel workloads where interconnect bandwidth is not the bottleneck.

Choose for tensor-parallel inference or training of >40B models with sustained utilization (100+ GPU-hours/month).

Choose for robotics and autonomous systems needing programmable acceleration across many model types, accepting higher power draw.

Choose for critical real-time production workloads requiring guaranteed availability.

Choose for the predictable, continuous baseline load present even in low-traffic periods.

Choose for batch, development, and embarrassingly parallel background tasks that tolerate interruption.

Choose for multi-tier offerings and diverse model sizes (7B-70B) when the team accepts extra scheduling and configuration complexity for better utilisation.

Choose for homogeneous workloads and SaaS platforms with many similar-sized tenants where operational simplicity is the priority.

Choose for 70B+ models, high-concurrency single-tenant or latency-critical deployments needing a full GPU.

Choose when inference replicas hold non-equivalent state (KV caches, loaded adapters) and multi-agent routing must track which pods cache which contexts.

Choose when all instances are identical and all requests equivalent, so raw throughput and simplicity outweigh request-aware routing.

Choose when request-based routing, application-aware health checking, TLS termination, or cookie-based session affinity is required.

Choose as the default when request durations vary widely (e.g., 90% under 2 s, 10% over 10 s).

Choose only when simpler heuristics (least connections, weighting) fail to maintain SLOs and observability infrastructure can supply fresh metrics.

Choose only when all requests have similar processing cost and all replicas identical capacity.

Choose as the simplest cache-locality option for conversational agents whose follow-up messages come from the same client.

Choose when replicas run on heterogeneous infrastructure with measurable capacity differences, or for weighted gradual rollouts.

Choose when validation completes before cutover and instant switch-over with instant failback to a fully running previous version is required.

Choose when a new version should be validated on real user traffic with limited exposure before full rollout.

Choose gradual one-at-a-time replacement when safety matters more than update speed; replacing all instances at once is faster but risky.

Choose for variable, unpredictable or infrequently triggered agent workloads.

Choose for latency-sensitive agents requiring consistent performance; avoid enabling it for all functions, since it pays for idle capacity and removes serverless's main cost benefit.

Choose for predictable, constant high-volume agent workloads where reserved capacity costs less than pay-per-execution billing.

Choose for moderate quality degradation, performance issues or non-critical bugs.

Choose for critical issues: error rate > 5%, latency > 200% of baseline, system crash/unavailability or detected data corruption.

Choose when agents or stores need persistent identity, ordered initialization (e.g., coordinator before workers) or reconnection to specific persistent volumes.

Choose for stateless agent replicas that load the same model, keep no local state and execute tasks independently (most worker agents).

Choose when comparing versions on live outcome metrics with users actually served by the new version, controlling for temporal and population effects.

Choose when a new version must be assessed on production inputs before committing to deployment, without executing its actions.

Choose to test utility-driven decisions across diverse and edge-case scenarios before deployment.

Choose when comprehensive per-trace ground truth justifies ~2-5% of inference cost and ~500 ms added latency per request.

Choose when a representative baseline-quality estimate is needed with no production latency impact.

Choose for production systems where evaluation should focus on problem areas; recommended default (5-10% with error oversampling).

Choose when managers must review multi-agent collaboration effectiveness and escalation patterns across thousands of conversations.

Choose when the audience is budget owners or executives who need trend, model-mix, unit-economics and budget-versus-actual views rather than operational health panels.

Choose to decide whether multi-step agent latency is dominated by LLM inference or external tool latency, and whether multi-agent parallelism is effective or serialized.

Choose for routine and production diagnosis of where time is spent across CPU-GPU boundaries, since event-driven tracing with CPU sampling keeps overhead at about 1-3%.

Choose when system profiling shows inference dominates and the time split across attention kernels, KV-cache access, quantized operations and batching must be understood.

Choose only for controlled deep-dive optimization of individual kernels (warp divergence, memory access, register use); its 10-100x slowdown precludes continuous production use.

Choose for non-real-time applications, cost-sensitive monitoring and historical analysis.

Choose for critical or high-volume applications that need real-time alerts.

Choose when agents must not receive full credit for correct answers reached through flawed reasoning and the bottleneck dimension should be surfaced.

Choose when dimensions carry different importance, e.g., high-stakes medical reasoning weighting correctness over efficiency or consumer chatbots weighting relevancy and informativeness.

Choose when valid answers have a single canonical form; it fails on correct paraphrases.

Choose when correct answers may be paraphrased (e.g., differently worded delivery dates) but wrong answers must still fail.

Choose for cheap detection of required/forbidden language (e.g., acknowledgment phrases, unauthorized guarantees).

Choose when the quality dimension is qualitative and resists simple keyword or rule checks.

Choose when success depends on structured constraints such as refund limits, dates or eligibility.

Choose when correct answers may be phrased differently from the ground truth and accuracy must be judged by meaning.

Choose when the agent exposes a tool-call log and operational efficiency matters.

Choose when deployment requires auditability, interpretability or trust calibration, not just correct outcomes.

Choose for highly constrained applications with well-defined, limited acceptable outputs (template-based bots, approved code sets, near-zero risk tolerance); avoid for general-purpose applications needing natural, flexible responses.

Choose when microsecond latency, implementation simplicity and deterministic, auditable behaviour matter, e.g., as a fast first-pass check for blatant violations before more expensive ML classifiers; avoid as sole defense where context matters or adversaries use misspellings and character substitutions.

Choose when adversaries consistently evade keyword filters or when context and semantics matter (coded language, microaggressions), accepting labeled-data needs, millisecond latency and residual adversarial susceptibility.

Choose as a fast, deterministic check for explicit regulated phrasing (e.g., 'you should buy', price predictions); insufficient alone because paraphrased advice evades surface patterns.

Choose when adversaries or the model paraphrase regulated content, so advice-giving intent must be detected regardless of phrasing, accepting higher false positives and compute cost.

Choose when threat modelling demands the strongest possible isolation per customer and its cost and operational complexity are acceptable.

Choose for adversarial multi-tenant, batch or high-value workloads (e.g., financial services handling customer funds) where security outweighs VM boot latency and memory overhead; also the fallback for gVisor-incompatible workloads.

Choose for low-risk development environments running trusted code; insufficient for multi-tenant, sensitive-data or autonomous production workloads because of shared-kernel escape risk.

Choose when stronger isolation than standard containers is needed with modest overhead, e.g., interactive real-time services that cannot tolerate VM boot latency, provided the workload's syscalls are supported.

Choose for deeper analysis in high-stakes applications (financial services, healthcare, government).

Choose for fast initial screening of all traffic; effective against documented attacks but not novel ones.

Choose when lower latency is needed and content must not be exposed to third parties (critical for HIPAA or GDPR), and ML infrastructure expertise is available.

Choose when ease of integration outweighs dependency on an external service and per-request pricing.

Choose for robust, scalable detection on user-facing content, including implicit bias without explicit markers; requires labeled training examples and periodic retraining.

Choose for high-stakes decisions needing maximum detection accuracy, accepting a full LLM inference of latency and per-evaluation cost.

Choose for fast, interpretable, immediate filtering of explicit bias indicators; insufficient alone because paraphrasing evades it and it misses contextual bias.

Choose for identifiers that are PII only in context (patient IDs, account numbers, internal IDs next to field labels).

Choose when documents mix structured, named and contextual PII (e.g., regulated healthcare content); avoids single-strategy blind spots.

Choose for unstructured PII such as person names, organizations and locations that regex cannot distinguish from normal text; costlier than patterns.

Choose for structured PII with predictable formats (SSNs, credit cards, phone numbers, emails); cheapest strategy but misses names and contextual identifiers.

Choose once policies have been calibrated in monitor and soft modes and their thresholds reflect organizational reality.

Choose when first deploying policies, to observe agent behaviour, detect over-triggering or never-triggering policies, and calibrate thresholds before enforcement.

Choose after baseline calibration when severe-consequence, low-error-tolerance policies (production database deletion, SSN exfiltration, exceeding authority limits) must be enforced while other policies still need tuning.

Choose when the generating model's token probability distributions are observable during parameter generation, so uncertainty can be measured without a separate model.

Choose when labeled examples of correct and hallucinated tool calls are available to train a well-calibrated verifier, and a second model check per call is acceptable.

Choose for clearly prohibited categories (e.g., illegal activity) where erring on the side of safety outweighs refusing some acceptable requests.

Choose when the violating portion can be removed or rewritten so the remaining response stays useful.

Choose for ambiguous cases with possible legitimate justification (fiction, security research) or high-stakes dilemmas, accepting slower responses for greater accuracy.

Choose for borderline outputs approaching policy boundaries where immediate blocking is not required; use blocking mode for inviolable legal, ethical or safety constraints.

Choose when preserving user autonomy matters and the concern (e.g., possible misinformation, low confidence) warrants flagging rather than blocking.

Choose when external validation is needed for customers, regulators or certification (e.g., ISO/IEC 42001), typically annually with a formal report.

Choose for continuous assurance: monthly or quarterly comprehensive audits with immediate finding reporting and verification that controls operate as designed.

Choose when actions are reversible and organizational risk tolerance favors velocity over oversight (aggressive).

Choose when actions are irreversible or risk tolerance is low (conservative; prevents unauthorized actions).

Choose when the decision must still be made promptly and higher authority is available to make it.

Choose for time-critical decisions whose delay harms users (e.g., emergency claims or treatments) when approval queues exceed their SLA.

Choose when the request remains valid later and neither declining nor escalating further is appropriate.

Choose (post-processing) to correct a trained model's decisions without retraining, e.g., as an adaptive intervention on production drift; post-hoc correction usually gives weaker fairness-accuracy trade-offs than in-training constraints.

Choose (in-processing) when fairness should be a first-class optimization goal; often yields better fairness-accuracy trade-offs than post-hoc correction because the model learns representations that satisfy the constraint.

Choose (pre-processing) when training data underrepresents groups or encodes historical imbalance; reweighting preserves the original distribution, while resampling/augmentation change the dataset.

Choose for globally deployed systems whose jurisdictions differ in law and cultural norms (e.g., political speech, religious expression), accepting higher design, evaluation and maintenance cost.

Choose when one consistent value set can serve all users, explicitly acknowledging limited scope (particular value commitments) rather than claiming universal validity.

Choose when users legitimately differ in how they weight values such as privacy versus convenience, and customization can be bounded by non-negotiable principles.

Choose when processing is genuinely optional and not necessary for the service (e.g., marketing, optional research participation, analytics cookies); avoid when withdrawal would disrupt ongoing operations such as longitudinal research.

Choose when processing is genuinely necessary, not merely convenient, for contract performance (e.g., shipping address and payment for a purchase, employee bank details, treatment as healthcare service delivery).

Choose when a specific statute mandates the processing (e.g., tax record retention, workplace safety documentation, medical record retention); vague references to 'legal compliance' are insufficient.

Choose when an organizational or third-party interest (e.g., fraud prevention, medical research with safeguards, behavioural analytics) does not override individual rights and consent would be operationally fragile.

Choose for governmental functions, public health initiatives or publicly funded research exercising official authority, identifying the specific public interest or legal provision.

Choose only narrowly, where failure to process could result in serious harm or death (e.g., treating an unconscious patient, child protection).

Choose when fairness metrics must be released with provable guarantees that no individual's participation or outcome can be inferred, and sample sizes are large enough to tolerate added noise.

Choose when several institutions (hospital networks, bank consortia) individually lack the data volume or diversity for statistically powerful bias detection but cannot centralize raw data.

Choose when parties must jointly compute aggregate fairness metrics while no party may learn any other party's demographic data or predictions, accepting cryptographic overhead.

Choose when likelihood and impact can only be judged in bands and qualitative judgment about context and acceptability matters more than numerical precision.

Choose when numerical evidence is needed to track trends, compare mitigation alternatives and trigger action at defined thresholds, e.g., continuous re-scoring.

Choose for IDEs, productivity and design tools where power users value keyboard speed and the agent can predict intent from context; avoid when discoverability for new users, multi-turn dialogue, or extensive explanation is required.

Choose for general Q&A, customer support, simple task automation, and exploratory conversations; avoid for workflows needing visual process representation, dense structured data, or parallel information streams.

Choose for post-hoc analysis (call-center QA, meeting summaries, voicemail), maximum accuracy (legal, medical), speaker diarization or overlapping multi-speaker audio, where higher latency is acceptable.

Choose when real-time interaction is required (live customer-service calls, voice assistants, live accessibility captioning) and latency matters more than perfect accuracy.

Choose when users can be expected to seek detail themselves (low time pressure, routine or low-stakes decisions); insufficient alone because users rarely seek additional information spontaneously.

Choose when confidence may fall below thresholds, decisions may deviate from historical patterns, or high-stakes scenarios arise, so critical detail must surface without user initiative.

Choose for expert analysts and compliance or regulatory auditors conducting deep audits or fairness reviews of high-stakes decisions.

Choose for regular users with moderate expertise (loan officers, credit analysts, moderators) and medium-stakes decisions: key factors, feature weights, confidence, comparisons and governing thresholds.

Choose for novice users, customers and affected parties, and for routine low-stakes decisions: plain-language summary of primary factors with actionable improvement or recourse guidance.

Choose for most agent streaming workflows: the user submits a query, the agent streams a response, and no mid-stream client-to-server interaction is needed.

Choose for genuinely interactive scenarios: agents expecting mid-stream user feedback or interruption, collaborative multi-user agent sessions, or real-time dashboards whose pushed updates need client acknowledgment.

Used in early RLHF work; generally avoid for preference learning because annotators interpret scales inconsistently across people and over time.

Choose when more information per annotation is worth higher annotator cognitive load and potentially lower judgment consistency.

Choose as the default (gold standard) elicitation format: comparative judgments are more reliable than absolute ratings and scale well.

Choose when the decision carries significant consequences but does not require immediate action.

Choose when agents detect high-severity situations (security incidents, system failures affecting critical operations) requiring urgent human attention.

Choose when stakeholders need awareness of a boundary event but the action may proceed (e.g., internal transfers above $10,000 with enhanced logging).

Choose when high feedback volume matters more than information density; single-click ratings generate high volume but low information per response.

Choose when periodic overall satisfaction, frequency and feature-request data are needed (Ref10.03), accepting lower response rates as context fades.

Choose when training data should focus precisely on current weaknesses rather than random optional feedback.

Choose when rich training signals are needed and lower participation due to higher user effort is acceptable.

Choose for high-consequence irreversible decisions (financial transactions, contractual commitments, changes affecting many users) where errors must be prevented before impact, accepting reduced throughput.

Choose for lower-stakes reversible actions where operational velocity matters; requires the ability to reverse completed actions when reviews reveal errors.

Choose when decisions carry a usable confidence score and escalation should depend on agent certainty; the text presents risk gating as complementary for high-impact actions.

Choose when certain case characteristics (unusual amount, rare scenario, conflicting signals, policy exception) warrant specialist judgement regardless of agent confidence or aggregate risk score.

Choose when escalation must reflect action impact (financial amount, deletion, external calls, customer reach) regardless of agent confidence.

Choose for specialized domains (healthcare, finance, autonomous vehicles, legal) where judging clinical accuracy, regulatory compliance or safety requires expertise; costs more than general annotators.

Choose for general-purpose content where volume and cost matter; unsuitable for specialized medical, legal, financial or safety-critical content.