Initialization hangs at: "OSM file has no replication headers. Looking for newest OSM object."

[pcace]

Hi there, i am trying to initialize the overpass api with these environment vars:

 "Env": [
 "OVERPASS_RULES_LOAD=10",
 "OVERPASS_META=yes",
 "OVERPASS_MODE=init",
 "OVERPASS_PLANET_URL=https://download.geofabrik.de/europe-latest.osm.bz2",
 "OVERPASS_DIFF_URL=https://download.openstreetmap.fr/replication/europe/minute/",
 "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
 "NGINX_VERSION=1.27.1",
 "NJS_VERSION=0.8.5",
 "NJS_RELEASE=1~bookworm",
 "PKG_RELEASE=1~bookworm",
 "DYNPKG_RELEASE=2~bookworm",
 "OTEL_VERSION=0.1.0",
 "OVERPASS_USE_AREAS=true",
 "OVERPASS_ALLOW_DUPLICATE_QUERIES=no"
 ],

the machine i am running it on is a 32GB Ram / 16Core machine hostet at hetzner.de.
i am currently wondering if it somehow crashed or not, since i have the same log message for more than a day:

2025-07-06 01:52:34 DEBUG: OSM file has no replication headers. Looking for newest OSM object.

which generally makes sense since i have pulled it from geofabrik. if i look at htop i can see that there are processes working on the update:

but there is currently nothing in the diffs folder:

(base) root@machine:/big/docker/overpass_europe# ls 
changes.log cookie.jar db/ diffs/ planet.osm.bz2 
(base) root@machine:/big/docker/overpass_europe# ls diffs/
(base) root@machine:/big/docker/overpass_europe# ls db/
node_frequent_tags.bin	 nodes.bin.idx	node_tags_global.bin	 relation_frequent_tags.bin	 relation_roles.bin.idx relations_meta.bin		relation_tags_local.bin.idx user_indices.bin.idx	ways.bin ways_meta.bin.idx
node_frequent_tags.bin.idx nodes.map		node_tags_global.bin.idx relation_frequent_tags.bin.idx relations.bin		 relations_meta.bin.idx	rules			 way_frequent_tags.bin	ways.bin.idx way_tags_global.bin
node_keys.bin		 nodes.map.idx	node_tags_local.bin	 relation_keys.bin		 relations.bin.idx	 relation_tags_global.bin	user_data.bin		 way_frequent_tags.bin.idx	ways.map way_tags_global.bin.idx
node_keys.bin.idx	 nodes_meta.bin	node_tags_local.bin.idx relation_keys.bin.idx		 relations.map		 relation_tags_global.bin.idx	user_data.bin.idx	 way_keys.bin		ways.map.idx way_tags_local.bin
nodes.bin		 nodes_meta.bin.idx	osm_base_version	 relation_roles.bin		 relations.map.idx	 relation_tags_local.bin	user_indices.bin	 way_keys.bin.idx		ways_meta.bin way_tags_local.bin.idx
(base) root@machine:/big/docker/overpass_europe# 

these are the file sizes:

du -sch *
4.0K	changes.log
4.0K	cookie.jar
80G	db
4.0K	diffs
49G	planet.osm.bz2
129G	total

is there some missconfiguration going on, or do i just need to wait another day or two?
currently the docker container reports it is running since 4 days (this is what it took to process europe) and unhealthy:

4e6c2f89d791 wiktorn/overpass-api "/docker-entrypoint.…" 4 days ago Up 4 days (unhealthy) 0.0.0.0:12345->80/tcp, [::]:12345->80/tcp overpass_europe

to me it seems it is taking way too long, but i am not sure if it might be ok. should i restart the container? or just leave it?

Thanks a lot for help!

[zstadler]

I've reached a similar point after 44 hour of init. About 5.25 hours later, the latest timestamp was found, and within less than a second, the corresponding sequence ID was found.

I was under the impression that in init mode, the .osm.bz2 file is scanned for the latest timestamp by the
osmium fileinfo -e -g data.timestamp.last /db/planet.osm.bz2 command within

Overpass-API/docker-entrypoint.sh

Line 89 in 5396bf5

/app/bin/init_osm3s.sh /db/planet.osm.bz2 /db/db /app "${META}" "--version=$(osmium fileinfo -e -g data.timestamp.last /db/planet.osm.bz2) --compression-method=${OVERPASS_COMPRESSION} --map-compression-method=${OVERPASS_COMPRESSION} --flush-size=${OVERPASS_FLUSH_SIZE}" &&

As far as I can tell, that scan took about 3.5 hours.

@wiktorn,
Do you think it would be useful to capture that last timestamp and pass it to update_overpass.sh

Overpass-API/docker-entrypoint.sh

Line 94 in 5396bf5

/app/bin/update_overpass.sh -O /db/planet.osm.bz2 &&

which would add it as a --start-date flag to

Overpass-API/bin/update_overpass.sh

Line 51 in 5396bf5

/app/venv/bin/pyosmium-get-changes -vvv "$@" --cookie /db/cookie.jar --server "${OVERPASS_DIFF_URL}" -o "${DIFF_FILE}" -f /db/replicate_id

---

Some timestamps from docker compose logs --timestamps overpass, FYI.

Downloading the planet data:

overpass-1 | 2025-07-16T19:00:17.129481630Z No database directory. Initializing
overpass-1 | 2025-07-16T19:00:17.136533058Z % Total % Received % Xferd Average Speed Time Time Time Current
overpass-1 | 2025-07-16T19:00:17.136564387Z Dload Upload Total Spent Left Speed
100 341 100 341 0 0 8872 0 --:--:-- --:--:-- --:--:-- 8973
100 384 100 384 0 0 7566 0 --:--:-- --:--:-- --:--:-- 7566
100 152G 100 152G 0 0 34.6M 0 1:14:56 1:14:56 --:--:-- 39.1M1:14:50 1:12:35 0:02:15 30.5M
overpass-1 | 2025-07-16T20:15:13.379337171Z Running preprocessing command: 

End of the first scan for the latest timestamp :

overpass-1 | 2025-07-17T01:51:32.036555163Z Reading XML file ... elapsed node 1. Compute current ... ready. Flushing to database ....... done.

The database creation completion:

overpass-1 | 2025-07-18T15:05:43.002259241Z Update complete.
overpass-1 | 2025-07-18T15:05:43.181244239Z Database created. Now updating it.
overpass-1 | 2025-07-18T15:05:43.185340497Z Updating
overpass-1 | 2025-07-18T15:05:49.200599256Z 2025-07-18 15:05:49 DEBUG: OSM file has no replication headers. Looking for newest OSM object.

End of the second scan for the latest timestamp:

overpass-1 | 2025-07-18T20:21:36.106824278Z 2025-07-18 20:21:36 INFO: Using replication server at https://planet.openstreetmap.org/replication/minute/
overpass-1 | 2025-07-18T20:21:36.107181203Z 2025-07-18 20:21:36 DEBUG: Looking up sequence ID for timestamp 2025-07-06 23:59:59+00:00

Identification of the sequence ID

overpass-1 | 2025-07-18T20:21:36.375772665Z 2025-07-18 20:21:36 DEBUG: Starting download at ID 6673397 (max 100 MB)