Orchestration · Data store

Incremental Watermark Store

Data storeOrchestrationOrchestration & Toolsarc:IncrementalWatermarkStore

A persistent record of the last successful extraction timestamp of a pipeline, used to bound the next incremental extraction window.

Responsibility. Remembers where the last successful run ended so the next run processes only the delta.

Also known as: ETL state file, High-water mark, Last extraction timestamp state

Direct neighbourhood (hover for relationship types)

Relationships

is read by dependency

is written by dependency

Ingestion Pipeline Orchestrator Ch6.3B

Design guidance

MUST update the watermark only after a run loads successfully, guaranteeing at-least-once processing.
SHOULD fall back to a configured lookback window (e.g., 24 hours) on first run when no state exists.
SHOULD write state atomically so partial writes never corrupt it.

Classification

Patterns: Watermark-based incremental processing
Technologies: JSON state fileDatabase tableApache ZooKeeper
Risks mitigated: Permanently skipped data after mid-run crashesMissed updates from guessed lookback windows

Sources

Ch6.3A: T. Nguyen, "ETL Pipeline Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.3A. ISBN: 9798244538229.
Ch6.3B: T. Nguyen, "ETL Worked Example - Load Phase & Pipeline Integration," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.3B. ISBN: 9798244538229.