Knowledge & Data · Software component
File Store Extractor
Software componentKnowledge & DataKnowledge & Dataarc:FileStoreExtractor
A data source connector that discovers files in file systems or object storage via path patterns and extracts those modified since the last run using file metadata.
Responsibility. Retrieves new or modified files from file systems or object storage.
Also known as: File system extractor, S3 connector, Object storage connector
Variant of Data Source Connector abstract
When to choose. Choose for unstructured document repositories such as shared drives, Git repositories or cloud object storage.
Relationships
invokes dependency
reads dependency
sends data to dynamic
Design guidance
- SHOULD isolate per-file failures (encoding, permission, deleted files) and continue so the pipeline completes when some files are inaccessible.
- SHOULD route binary formats by file type to an appropriate document parser.
- SHOULD detect file encodings before decoding legacy documents, with best-effort fallback.
- SHOULD deduplicate symlinks, hardlinks and copies using inodes or content hashes.
- SHOULD load multiple formats (PDF, TXT, Markdown), handle encoding and track source, date and author metadata (Ref7.07).
Quantitative guidance
As stated by the sources; verify before use.
- Modification-time checks yield 10-100x speedups on large documentation repositories where only a few files change daily (Ch6.3A).
Classification
- Patterns
- Glob-based file discoveryModification-time change detectionPer-file graceful error handlingEncoding detectionInode or content-hash duplicate detection
- Technologies
- Amazon S3Azure Blob StorageGoogle Cloud StorageSharePointGit repositorieschardetLangChain DirectoryLoader
- Quality attributes
- Reliability (ISO/IEC 25010 | NIST AI RMF: valid and reliable)Performance efficiency (ISO/IEC 25010)
- Risks mitigated
- Pipeline abort from individual unreadable filesDuplicate entries from symlinks or copiesEncoding decode failures on legacy documents
Sources
- Ch6.3A: T. Nguyen, "ETL Pipeline Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.3A. ISBN: 9798244538229.
- Ref7.07: E. Li, V. Bellotti, R. Kraus, and R. Kao, "Build a retrieval-augmented generation (RAG) agent with NVIDIA Nemotron," NVIDIA Technical Blog, Sep. 23, 2025. [Online]. Available: https://developer.nvidia.com/blog/build-a-rag-agent-with-nvidia-nemotron/