[BUG]: Access blocked for OpenStreetMap tiles

[elpamplina]

Description

Generating a map from OpenStreetMap, some tiles are filled with the error message: "403r Access blocked. Referer is required by tile usage policý of OpenStreetMap's volunteer-run servers: osm.wiki/Blocked"

This occurs randomly about half the times, on one or more tiles.

Screenshots/Video

Steps to reproduce

Python code:

import plotly.graph_objects as go

lat = 41.580472
lon = -1.1161953

fig = go.Figure(go.Scattermap(
 lat=[lat],
 lon=[lon],
 mode='markers',
 marker=dict(symbol="circle", size=30)
))
fig.update_layout(
 map=dict(
 style="open-street-map",
 center=dict(lat=lat, lon=lon),
 zoom=16
 ),
 margin={"r": 0, "t": 0, "l": 0, "b": 0} 
)
img = fig.to_image('png')

with open('map.png', 'wb') as f:
 f.write(img)

Generated with:
plotly 6.6.0
kaleido 1.2.0

[StyXman]

According to OSM's tile policy https://operations.osmfoundation.org/policies/tiles/ you have to either set "a valid HTTP User-Agent [header]" or "a valid HTTP Referer header", otherwise you might get throttled as above.

Plotly should allow to provide either of these so OSM people are happy.

[camdecoster]

I just tried running your example and didn't see an issue. Is the problem still happening for you?

[camdecoster]

The image I produced:

[NorthLake]

I just tried running your example and didn't see an issue. Is the problem still happening for you?

For us the issue happens every 50% of times a map is rendered, I'd say. Only some of the tiles in view would be replaced by the error. Perhaps it's a usage thing that regular testing wouldn't bring up?
Either way, I took a look at the obfuscated frontend code, and it seems like the referrer is passed along when creating the Request object, but the browser ignores the argument and sets it to about:client. At least, that's the case for us when rendering it in a new iframe (using Viktor). Why this is the case I do not know, but that's why OpenStreetMap is saying the referer header is missing.

[elpamplina]

I just tried running your example and didn't see an issue. Is the problem still happening for you?

The problem occurs randomly, about once in two.

[camdecoster]

Okay, I was able to reproduce it. That's clearly an issue somewhere in the rendering chain. Our team won't be able to dig into this any time soon, but we can prioritize a PR review if you want to investigate and fix the problem.

[StyXman]

I took a look at the source code, but I couldn't find where do you either have a function that generates that image, or call a function that does it. @elpamplina said that at some point there's a browser involved?

[elpamplina]

I have found that the JS core does try to get the referrer, but the worker created by the Python wrapper cannot give it to it (probably because being a background instance created by the wrapper it has no real reference).
You can find the referrer in the line 176742 in the unobfuscated core library:
https://github.com/plotly/plotly.js/blob/a6663c92620c222b0fd9b01495c724bbde4ab5d3/dist/plotly.js

 var getReferrer = isWorker() ? function() {
 return self.worker && self.worker.referrer;
 } : function() {
 return (window$1.location.protocol === "blob:" ? window$1.parent : window$1).location.href;
 };

We would have to look for the reason why some tiles are loaded and others not because different OSM servers will have different referrer policies, and it randomly coincides that at some point they all accept it.
I think the best solution is for the wrapper to support a referrer attribute, so that it can be passed to the core JS and thus comply with the policy.
For instance:

map=dict(
 style="open-street-map",
 center=dict(lat=lat, lon=lon),
 zoom=16,
 referrer="https://example.com"
 ),

I do not have the precise knowledge of how wrapper works to do it myself.

[StyXman]

We would have to look for the reason why some tiles are loaded and others not because different OSM servers will have different referrer policies, and it randomly coincides that at some point they all accept it.

It's something that the OSM tile server does, it only replaces some of the tiles with the one with the error, so users can find the issue by themselves and act accordingly without flatly refusing to serve tiles.

[camdecoster]

The calls to the tile server go through the maplibre library, which is where that code reference comes from in the bundle. That library does try to add the referrer properly, but something is breaking between that point and the rendering of the image. The image rendering is done with Kaleido, which uses a headless version of Chrome. It's possible the headers are being lost during that process.

[JoGorska]

I was wondering if this could be connected with openmaptiles returning phishing error

[AliaHa3]

any solution?