Orchestration · Software component
Agent Controller
Software componentOrchestrationOrchestration & ToolsVariation point (abstract)arc:AgentController
An abstract agent runtime that runs the control loop deciding, for one agent, how reasoning, tool actions, and observations are sequenced toward a goal.
Responsibility. Runs the reasoning-action control loop for one agent.
Also known as: Agent, Agent runtime, Agent loop, Support agent, Agent runtime loop, ReAct loop, Compliance Agent, Agent pod, Agent replica, Agent instance, Stateless agent replica, Stateless agent instance, LLM agent reasoning core, High-stakes decision agent
Variants
| Variant | When to choose |
|---|---|
| Direct Tool-Calling Controller | Choose for simple, deterministic tasks where a straightforward tool call suffices and latency or budget constraints cannot absorb multi-step reasoning or planning overhead. |
| Function-Calling Controller | Choose when an agent must integrate an extensive, growing catalog of capabilities (tens of functions across many domains) whose composition cannot be pre-programmed and new capabilities must be added without changing orchestration code. Avoid for focused agents with fewer than five capabilities, or for workflows needing iterative refinement loops, conditional branching on intermediate results, or strictly deterministic execution (prefer explicit state-graph orchestration). |
| Plan-and-Execute Controller | Choose for complex workflows with mostly predictable steps in stable environments, resource-constrained settings, or supervisor-worker coordination; avoid for highly dynamic, real-time, simple, or open-ended exploratory tasks. |
| ReAct Agent Controller | Choose when solution paths are unpredictable and require adaptive investigation (research, debugging, non-standard support); avoid for simple deterministic, latency-critical, budget-constrained, or formally verified tasks. |
Relationships
deployed on structural
- Agent Hosting Platform abstract Ch4.2
- CPU Compute Node Ch4.7
- Cluster Namespace Ch4.3
- Container Orchestrator Ch1.8 Ch4.1 +1
- Execution Sandbox abstract Ch9.3
- GPU Node abstract Ch4.7
- Staging Environment Ch3.1A Ch4.1 +3
exposes structural
is configured by structural
invokes dependency
- Approval Proposal Builder Ch10.4
- Benchmark Environment abstract Ch3.2
- Code Execution Runner Ch10.5
- Confidence Estimator Ch1.1A
- Context Window Manager abstract Ch1.4
- Cross-Source Consistency Verifier Ch3.2
- Decision Engine abstract Ch5.10
- Dynamic Tool Loader Ch3.10
- Embedding Service abstract Ch1.8
- Failure Analyzer Ch3.8
- Guardrail Orchestrator Ch7.1B
- Hybrid Retriever abstract Ch1.8
- Inference Server Ch9.4
- LLM Inference Service Ch1.4 Ch1.8 +7
- LLM Provider Adapter Ref3.07
- Memory Retriever Ch1.4
- Model Router Ch1.8 Ch3.10 +2
- OpenAI-Compatible Inference API Ch7.3
- Oversight Gate abstract Ch1.1A Ch1.1B +2
- Perception Interpreter Ch1.4
- Plan Template Adapter Ch4.7
- Policy Plugin Gateway Ch9.2
- Progress Validator Ch1.6
- Prompt Context Builder Ch8.3
- Reasoning Engine Ch1.2 Ch1.5A +1
- Request Batcher Ch2.9 Ch3.4
- Retriever abstract Ch1.7A Ch2.9 +1
- Retry Handler Ch2.8
- Sandbox Execution API Ch9.3
- Task Planner abstract Ch1.7B Ch3.8
- Tool Call Dispatcher abstract Ch2.6 Ch3.4 +1
- Tool Executor Ch1.1A Ch1.2 +6
- Tool Protocol Client Ch7.3
- Tool Protocol Server Ref1.01 Ref7.14
- Tool Selector Ch1.7B
is invoked by dependency
reads dependency
- Conversation State Store abstract Ch1.8
- Episodic Memory Store Ch1.7B
- External Session State Store Ch4.7 Ref7.17
- Instance-Local Session State Ch4.7
- Object Store Ch4.7
- Response Cache abstract Ch1.8 Ch2.9
- Secrets Vault Ch4.4 Ch4.7
- Semantic Memory Store Ch1.7B
- State Checkpoint Store abstract Ch1.4 Ch1.7B
- Task Queue Ch1.8
- Tool Registry Ch1.2
- Tool Result Cache Ch1.1A Ch1.2
- Working Memory Buffer abstract Ch1.4 Ch1.7B +1
- World Model State Ch1.4
writes dependency
emits telemetry to dynamic
escalates to dynamic
is routed to by dynamic
- A/B Test Traffic Splitter Ch3.1A Ch3.3 +1
- Approval Outcome Router Ch10.4 Ref10.01
- Blue-Green Deployment Switcher Ch4.1
- Layer-7 Load Balancer Ch4.2 Ch4.7
- Load Balancer abstract Ch1.8 Ch4.7 +2
- Rollout Manager abstract Ch4.1
- Multimodal Content Router Ch7.5
- Service Mesh Proxy Ch4.3
- Session Affinity Load Balancer Ch4.3
is triggered by dynamic
receives data from dynamic
sends data to dynamic
is constrained by control
- AI Governance Policy Ch9.8
- Action Policy Engine Ch1.2 Ch10.5
- Agent-Specific Policy Ch10.5
- Constitution abstract Ch9.5
- Container Security Context Ch4.3
- Deployment Manifest Ch4.7
- Compliance Policy Rule Set Ch9.5 Ch10.5
- Execution Timeout Policy Ch2.8
- Feature Flag Service Ch9.8
- Iteration Limit Policy Ch1.6 Ref8.06
- Least-Privilege Permission Set Ch9.2 Ch9.3
- Operational Norm Set Ch9.6
- Output Format Specification Ch8.2A
- Physical Safety Envelope Ch10.5
- Rate Limiter Ch9.2
- Retry Policy Ch3.8
- Risk Acceptance Record Ref9.07
- Scoped Access Token Ch9.2
- Time-Bound Permission Grant Ch9.2
- Token Budget Enforcer Ch3.10 Ch10.5 +1
- Transparency Policy Ch9.3
is guarded by control
- Action Policy Engine Ref9.09
- Circuit Breaker Ch2.8
- Client Rate Limiter Ref8.06
- Fact Checking Rail abstract Ref8.06
- Guardrail Orchestrator Ch8.2B Ch10.4
- Input Rail Ch9.5 Ref7.14 +1
- Output Rail Ch3.2 Ch8.2A +2
- PII Redactor Ch3.2
- Risk Gate Ch9.4 Ref9.02
- Service Mesh Proxy Ch4.3
- State Cycle Detector Ch1.6
- Rule Constraint Filter Ch9.6
is orchestrated by control
is overridden by control
is scaled by control
requires approval from control
is audited by assurance
is evaluated by assurance
- A/B Test Traffic Splitter Ch3.4
- Agent Test Runner Ch4.2
- Evaluation Harness Ch3.1A Ch3.1B +8
- Feedback Collector abstract Ch3.8
- Citation Verifier Ch8.2A
- Human Evaluator Ch3.3 Ch10.5
- Interaction Safety Scanner Ch3.3
- LLM Judge abstract Ch3.8 Ch8.2A
- Load Test Runner Ch4.1
- Online Evaluator Ch3.3 Ch4.2
- Policy Adherence Evaluator Ch9.2
- Principle Adherence Evaluator Ch9.6
- Red Team Tester Ch9.6
- Regression Gate Ch8.3
- Release Approver Ch4.2
- Smoke Tester Ch4.2
- Task Success Evaluator abstract Ch3.3
- Tool Call Accuracy Evaluator Ch3.8
- Trajectory Matching Evaluator Ch3.8
is monitored by assurance
- Agent Circuit Breaker Ch10.4
- Alignment Drift Monitor Ch9.6
- Execution Profiler Ch3.4 Ch7.3
- Experiment Guardrail Monitor Ch3.1A Ch4.2
- Fairness Monitor Ch10.5
- Human Supervisor Ch1.1A Ch1.1B +3
- Quality Drift Detector Ch4.1 Ch10.5
- SLO Monitor Ch4.2
- Token Cost Meter Ch1.8 Ch7.3
- Tool Usage Analyzer Ch4.1
- Agent Behavior Anomaly Detector Ch3.3 Ch9.6 +2
Design guidance
- SHOULD select the control pattern (ReAct, Plan-and-Execute, direct tool calling) from problem characteristics as a strategic architecture decision.
- SHOULD start simple and add complexity only when evidence demonstrates necessity.
- SHOULD use structured perception output (category, entities) to select which memories to retrieve rather than loading all memories into context.
- SHOULD update working context immediately while persisting to long-term stores asynchronously.
- MUST stop on sufficient information or on an iteration limit rather than looping indefinitely.
- SHOULD store no session state locally; conversation history, preferences and intermediate results SHOULD live in external stores so any replica can serve any request.
- SHOULD be right-sized: use the smallest model and GPU type that meets measured quality thresholds.
- SHOULD request GPU resources equal to limits to receive guaranteed resources and avoid noisy-neighbour contention.
- SHOULD select trace collection depth per request context (full, sampled, confidence-conditional, user-triggered) rather than tracing everything.
- MUST process each request without relying on replica-local state from previous interactions, externalizing conversation history, preferences and intermediate artifacts.
- MAY keep short-TTL local caches of recent context if it can reconstruct context when routed to a different replica.
- MUST emit structured logs to a shared aggregation system so request paths can be traced across replicas.
Quantitative guidance
As stated by the sources; verify before use.
- Externalized state retrieval adds 5-20 ms per request; local+Redis two-layer caching yields sub-10 ms average (Ch1.8).
- Doubling stateless replicas (3 to 6) doubles throughput (Ch1.8).
- Right-sizing from large (8 cores/32 GB) to medium (4 cores/16 GB) instances can halve cost with no performance impact when GPU-bound (Ch1.8).
- Worked deployment: 3 replicas each handling ~100 requests/min; traffic from 20 req/min overnight to 300 req/min peak with 500 req/min spikes (Ch4.7).
- Losing one of three replicas reduces capacity 33% while maintaining availability (Ch4.7).
Classification
- Patterns
- Agent control loopPerceive-remember-act loopAgentic RAGReActStateless replicaExternalized stateHorizontal scaling (scale out)Vertical scaling (scale up)
- Technologies
- LangChainKubernetes DeploymentNVIDIA NIMLangGraphLlamaIndex
- Quality attributes
- Flexibility (ISO/IEC 25010)Cost efficiencyPerformance efficiency (ISO/IEC 25010)Functional suitability: correctness and validity (ISO/IEC 25010 | NIST AI RMF: valid)Interaction capability (ISO/IEC 25010)Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)
- Risks mitigated
- Stateless treatment of repeated situationsInfinite loopsSingle point of failureSession loss on instance failure
Sources
- Ch1.1A: T. Nguyen, "Designing User Interfaces for Intuitive Human-Agent Interaction," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.1A. ISBN: 9798244538229.
- Ch1.1B: T. Nguyen, "Human-in-the-Loop Patterns and Accessible Design," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.1B. ISBN: 9798244538229.
- Ch1.2: T. Nguyen, "Core Agent Patterns," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.2. ISBN: 9798244538229.
- Ch1.4: T. Nguyen, "Memory and Perception Systems," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.4. ISBN: 9798244538229.
- Ch1.5A: T. Nguyen, "Stateful Orchestration - Introduction and Core Concepts," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.5A. ISBN: 9798244538229.
- Ch1.6: T. Nguyen, "Stateful Orchestration - Pitfalls, Integration, and Synthesis," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.6. ISBN: 9798244538229.
- Ch1.7A: T. Nguyen, "Relational Reasoning with Knowledge Graphs - The Fundamentals, Integration, and Extraction," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.7A. ISBN: 9798244538229.
- Ch1.7B: T. Nguyen, "Relational Reasoning with Knowledge Graphs - Hybrid RAG+KG Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.7B. ISBN: 9798244538229.
- Ch1.8: T. Nguyen, "Scalability and Production Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 1.8. ISBN: 9798244538229.
- Ch2.5: T. Nguyen, "Semantic Kernel," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 2.5. ISBN: 9798244538229.
- Ch2.6: T. Nguyen, "Tool Integration and Function Calling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 2.6. ISBN: 9798244538229.
- Ch2.9: T. Nguyen, "Streaming and Real-Time Responses," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 2.9. ISBN: 9798244538229.
- Ch3.1A: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1A. ISBN: 9798244538229.
- Ch3.1B: T. Nguyen, "Implement Evaluation Pipelines and Task Benchmarks - Guided Practice," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.1B. ISBN: 9798244538229.
- Ch3.2: T. Nguyen, "Compare Agent Performance Across Tasks and Datasets," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.2. ISBN: 9798244538229.
- Ch3.8: T. Nguyen, "Action Accuracy Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.8. ISBN: 9798244538229.
- Ch3.10: T. Nguyen, "Efficiency Metrics," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 3.10. ISBN: 9798244538229.
- Ch4.1: T. Nguyen, "Introduction to AI Agent Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.1. ISBN: 9798244538229.
- Ch4.2: T. Nguyen, "Deployment and Scaling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.2. ISBN: 9798244538229.
- Ch4.3: T. Nguyen, "Container Orchestration and Edge Deployment," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.3. ISBN: 9798244538229.
- Ch4.4: T. Nguyen, "Performance Profiling and Optimization," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.4. ISBN: 9798244538229.
- Ch4.7: T. Nguyen, "Scaling Strategies," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 4.7. ISBN: 9798244538229.
- Ch5.10: T. Nguyen, "Utility-Based Decision Making Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.10. ISBN: 9798244538229.
- Ch7.1B: T. Nguyen, "Nvidia NIM and Colang," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.1B. ISBN: 9798244538229.
- Ch7.3: T. Nguyen, "NeMo Agent Toolkit Profiling," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.3. ISBN: 9798244538229.
- Ch7.5: T. Nguyen, "NeMo Curator, Riva Speech AI & Multimodal Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 7.5. ISBN: 9798244538229.
- Ch8.2A: T. Nguyen, "Error Rates and Reliability," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.2A. ISBN: 9798244538229.
- Ch8.2B: T. Nguyen, "NeMo Guardrails Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 8.2B. ISBN: 9798244538229.
- Ch9.2: T. Nguyen, "Action Constraints and Permission Models," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.2. ISBN: 9798244538229.
- Ch9.4: T. Nguyen, "Fairness and Bias Mitigation," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.4. ISBN: 9798244538229.
- Ch9.5: T. Nguyen, "Constitutional AI," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.5. ISBN: 9798244538229.
- Ch9.8: T. Nguyen, "Standards and Frameworks for AI Governance," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 9.8. ISBN: 9798244538229.
- Ch10.4: T. Nguyen, "Human-in-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.4. ISBN: 9798244538229.
- Ch10.5: T. Nguyen, "Human-over-the-Loop," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 10.5. ISBN: 9798244538229.
- Ref7.14: "NVIDIA Agentic AI Platform Ecosystem Integration," unpublished reference note (14-NVIDIA-Ecosystem-Integration.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref7.17: "Scaling Agentic AI Systems: Patterns and Strategies," unpublished reference note (17-Scalability-Patterns.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref7.18: "Chapter 7 Summary: NVIDIA Platform Implementation," unpublished reference note (18-Chapter-7-Summary.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref8.07: "Agent Health Checks and Diagnostics," unpublished reference note (07-Agent-Health-Checks-Diagnostics.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref9.07: "Risk Assessment and Management for AI Systems," unpublished reference note (references/Chapter 9 - Safety, Ethics, and Compliance/07-Risk-Assessment-Management.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref9.08: "Safety Incident Response for AI Systems," unpublished reference note (08-Safety-Incident-Response.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref9.09: "Compliance Automation and Tools," unpublished reference note (09-Compliance-Automation-Tools.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note
- Ref10.01: "Human-in-the-Loop Systems for Agent Interactions," unpublished reference note (01-Human-in-the-Loop-Systems.md), Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam supplementary materials, 2026. unpublished note