ETL Pipeline Configuration

Data artifactOrchestrationOrchestration & Toolsarc:ETLPipelineConfig

A centralized configuration of source connections, chunking parameters, quality filters, vector store settings and incremental-update settings for an ETL pipeline.

Responsibility. Separates pipeline tuning and connection settings from code.

Also known as: Pipeline configuration dictionary

Direct neighbourhood (hover for relationship types)

Relationships

configures structural

Ingestion Pipeline Orchestrator Ch6.3A

Design guidance

SHOULD separate configuration from code so the same pipeline deploys to development, staging and production.
SHOULD validate configuration against a schema and version configuration changes to enable rollback after quality regressions.
MUST NOT hardcode credentials; interpolate them from environment variables or a secrets manager.

Classification

Patterns: Configuration as single source of truthEnvironment promotion via configuration
Technologies: PydanticAWS Secrets ManagerHashiCorp VaultKubernetes secrets
Quality attributes: Maintainability (ISO/IEC 25010)Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)

Sources

Ch6.3A: T. Nguyen, "ETL Pipeline Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.3A. ISBN: 9798244538229.