Add a durable incremental OSM replication lifecycle

[Medformatik]

Problem

OpenMapX downloads atomic OSM PBF snapshots, while Nominatim and Overpass use their own daily replication settings. There is no shared durable replication lifecycle for keeping regional/planet inputs current or reporting their sequence and lag.

Current state

• The data manager supports checksummed, conditional full PBF downloads and promotion.
• Nominatim and Overpass manifests point at daily replication.
• OSM search, conflation, routing/tile inputs, and the base PBF do not share a replication state or freshness contract.

Proposed solution

Add a sequence-aware OSM replication job that can maintain an authoritative mutable snapshot safely, using Pyosmium/Osmium primitives. Support the correct source cadence for a region: Geofabrik daily diffs where paired with its extracts, or official planet replication where planet-scale operation is explicitly configured.

Acceptance criteria

• A persisted state records source URL, sequence, timestamp, base checksum, last success, lag, and current snapshot generation.
• Bootstrap derives a safe sequence from PBF headers/timestamp and refuses ambiguous source/extract combinations.
• Diffs are fetched and applied consecutively, idempotently, with bounded batches and checksum/integrity validation.
• A crash cannot corrupt or partially replace the last promoted PBF; apply occurs in staging with atomic promotion.
• Sequence gaps, expired regional history, source changes, excessive backlog, or invalid output trigger a documented full-resync path.
• Only one writer may advance a dataset; retries and cancellation preserve a recoverable checkpoint.
• Metrics/admin state expose sequence, source timestamp, wall-clock lag, batch size, duration, and resync reason.
• Consumers receive immutable generation IDs and choose an explicit debounce/rebuild strategy rather than reading a mutating file.
• A first downstream consumer proves the contract; minute-level rebuilds of every graph/index are not implied.
• Tests cover create/modify/delete, referenced nodes, repeated object versions, gaps, crash recovery, and regional boundary changes.
• Operator docs explain regional versus planet sources, retention windows, sizing, and recovery.

Alternatives considered

Frequent full extracts are operationally simple but waste bandwidth and extend freshness lag. Applying diffs directly to live files risks corruption and mixed consumer state.

Area

OSM data manager, operations, downstream data freshness.

References

• https://docs.osmcode.org/pyosmium/latest/user_manual/10-Replication-Tools/
• https://docs.osmcode.org/pyosmium/latest/user_manual/08-Working-With-Change-Files/