Knowledge & Data · Software component
Data Source Connector
Software componentKnowledge & DataKnowledge & DataVariation point (abstract)arc:DataSourceConnector
An abstract extraction component that interfaces with one class of source system to pull raw records or documents in their native format for an ETL pipeline.
Responsibility. Retrieves raw data from a specific type of source system without applying transformation logic.
Also known as: Data connector, Extractor, Extract stage
Variants
| Variant | When to choose |
|---|---|
| Event Stream Consumer | Choose for real-time or semi-structured event sources when near-instant knowledge updates justify the added complexity. |
| File Store Extractor | Choose for unstructured document repositories such as shared drives, Git repositories or cloud object storage. |
| Paginated API Extractor | Choose for semi-structured sources exposed through REST or GraphQL APIs (e.g., ticketing, CRM, wiki platforms). |
| Source Record Extractor | Choose for structured sources (relational/NoSQL databases, warehouses) whose schemas and audit timestamps enable selective, batch delta queries. |
Relationships
exposes structural
reads dependency
receives data from dynamic
sends data to dynamic
is orchestrated by control
Design guidance
- SHOULD confine the Extract stage to data acquisition, leaving cleaning and transformation to later stages.
- SHOULD handle transient failures gracefully and respect source-system limits so extraction does not overwhelm source infrastructure.
- SHOULD extract only records changed since the last successful run (timestamp or version markers) once full reprocessing becomes infeasible.
- SHOULD load source credentials from environment variables or a secrets manager rather than hardcoding them.
Quantitative guidance
As stated by the sources; verify before use.
- Updating 1,000 changed documents takes minutes versus hours to reprocess 1 million total documents (Ch6.3A).
Classification
- Patterns
- ETL Extract stageIncremental (delta) extractionTimestamp-based change filtering
- Quality attributes
- Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)Maintainability (ISO/IEC 25010)Performance efficiency (ISO/IEC 25010)
- Risks mitigated
- Knowledge gaps from fragmented enterprise dataOverloading source system infrastructureStale agent knowledge
Sources
- Ch6.3A: T. Nguyen, "ETL Pipeline Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.3A. ISBN: 9798244538229.