Kaleido alters the working directory which prevents the use of relative file paths in figures

[filiagees]

Hi all,

I'm facing a problem when invoking write_image()¹ to save a hello-world-simple choropleth image (using plotly.express).
It returns the error:

ValueError: Transform failed with error code 525: Unexpected error while fetching from test/sample.json

I'm using:

• kaleido 0.0.2
• plotly 4.9.0

I've shrunk the source to the simplest form, it's available here:
https://github.com/filiagees/laesp

---

Trying to troubleshooting the issue, I found out I can trigger the error running the following command:

echo -n '{"format": "svg", "width": 700, "height": 500, "scale": 1, "data": {"data": [{"colorscale": [[0.0, "rgb(254,232,200)"], [1.0, "rgb(254,232,200)"]], "featureidkey": "properties.prop0", "geo": "geo", "geojson": "test/sample.json", "hovertemplate": "color=1<br>prop0=%{location}<extra></extra>", "locations": ["square"], "name": "1", "showlegend": true, "showscale": false, "z": [1], "type": "choropleth"}], "layout": {"geo": {"domain": {"x": [0.0, 1.0], "y": [0.0, 1.0]}, "projection": {"type": "mercator"}, "center": {}}, "template": {"data": {"histogram2dcontour": [{"type": "histogram2dcontour", "colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}], "choropleth": [{"type": "choropleth", "colorbar": {"outlinewidth": 0, "ticks": ""}}], "histogram2d": [{"type": "histogram2d", "colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}], "heatmap": [{"type": "heatmap", "colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}], "heatmapgl": [{"type": "heatmapgl", "colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}], "contourcarpet": [{"type": "contourcarpet", "colorbar": {"outlinewidth": 0, "ticks": ""}}], "contour": [{"type": "contour", "colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}], "surface": [{"type": "surface", "colorbar": {"outlinewidth": 0, "ticks": ""}, "colorscale": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]]}], "mesh3d": [{"type": "mesh3d", "colorbar": {"outlinewidth": 0, "ticks": ""}}], "scatter": [{"type": "scatter", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "parcoords": [{"type": "parcoords", "line": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "scatterpolargl": [{"type": "scatterpolargl", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "bar": [{"error_x": {"color": "#2a3f5f"}, "error_y": {"color": "#2a3f5f"}, "marker": {"line": {"color": "#E5ECF6", "width": 0.5}}, "type": "bar"}], "scattergeo": [{"type": "scattergeo", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "scatterpolar": [{"type": "scatterpolar", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "histogram": [{"type": "histogram", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "scattergl": [{"type": "scattergl", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "scatter3d": [{"type": "scatter3d", "line": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "scattermapbox": [{"type": "scattermapbox", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "scatterternary": [{"type": "scatterternary", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "scattercarpet": [{"type": "scattercarpet", "marker": {"colorbar": {"outlinewidth": 0, "ticks": ""}}}], "carpet": [{"aaxis": {"endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f"}, "baxis": {"endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f"}, "type": "carpet"}], "table": [{"cells": {"fill": {"color": "#EBF0F8"}, "line": {"color": "white"}}, "header": {"fill": {"color": "#C8D4E3"}, "line": {"color": "white"}}, "type": "table"}], "barpolar": [{"marker": {"line": {"color": "#E5ECF6", "width": 0.5}}, "type": "barpolar"}], "pie": [{"automargin": true, "type": "pie"}]}, "layout": {"colorway": ["#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52"], "font": {"color": "#2a3f5f"}, "hovermode": "closest", "hoverlabel": {"align": "left"}, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": {"bgcolor": "#E5ECF6", "angularaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "radialaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}}, "ternary": {"bgcolor": "#E5ECF6", "aaxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "baxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}, "caxis": {"gridcolor": "white", "linecolor": "white", "ticks": ""}}, "coloraxis": {"colorbar": {"outlinewidth": 0, "ticks": ""}}, "colorscale": {"sequential": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "sequentialminus": [[0.0, "#0d0887"], [0.1111111111111111, "#46039f"], [0.2222222222222222, "#7201a8"], [0.3333333333333333, "#9c179e"], [0.4444444444444444, "#bd3786"], [0.5555555555555556, "#d8576b"], [0.6666666666666666, "#ed7953"], [0.7777777777777778, "#fb9f3a"], [0.8888888888888888, "#fdca26"], [1.0, "#f0f921"]], "diverging": [[0, "#8e0152"], [0.1, "#c51b7d"], [0.2, "#de77ae"], [0.3, "#f1b6da"], [0.4, "#fde0ef"], [0.5, "#f7f7f7"], [0.6, "#e6f5d0"], [0.7, "#b8e186"], [0.8, "#7fbc41"], [0.9, "#4d9221"], [1, "#276419"]]}, "xaxis": {"gridcolor": "white", "linecolor": "white", "ticks": "", "title": {"standoff": 15}, "zerolinecolor": "white", "automargin": true, "zerolinewidth": 2}, "yaxis": {"gridcolor": "white", "linecolor": "white", "ticks": "", "title": {"standoff": 15}, "zerolinecolor": "white", "automargin": true, "zerolinewidth": 2}, "scene": {"xaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white", "gridwidth": 2}, "yaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white", "gridwidth": 2}, "zaxis": {"backgroundcolor": "#E5ECF6", "gridcolor": "white", "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white", "gridwidth": 2}}, "shapedefaults": {"line": {"color": "#2a3f5f"}}, "annotationdefaults": {"arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1}, "geo": {"bgcolor": "white", "landcolor": "#E5ECF6", "subunitcolor": "white", "showland": true, "showlakes": true, "lakecolor": "white"}, "title": {"x": 0.05}, "mapbox": {"style": "light"}}}, "legend": {"title": {"text": "color"}, "tracegroupgap": 0}, "margin": {"t": 60}}}}' | \
/_/git/laesp/venv/lib/python3.8/site-packages/kaleido/executable/kaleido 'plotly'  --disable-gpu --plotlyjs='/_/git/laesp/venv/lib/python3.8/site-packages/plotly/package_data/plotly.min.js' --mathjax='https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.5/MathJax.js'

(considering the files inside the folder /_/git/laesp/, see run_kaleido.sh)

I have no more idea how to analyze the binary, so I'm cannot be more helpful :(

Am I doing something wrong?

Any help would be apreciated!
:)

[jonmmease]

Hi @filiagees, thanks for the report and debugging the issue a bit.

As the error mentions, it looks like the issue is with finding/loading the test/sample.json file. Is this exact plot working for you when you display it in another context (e.g. the notebook or a standalone html file)? If so, I'm thinking this might be an issue where kaleido doesn't preserve the current working directory and so relative paths aren't working the way you would expect.

Here are two things to try:

• Replace test/sample.json with the absolute path to your sample.json file. e.g. /path/to/test/sample.json
• Replace test/sample.json with a file:// url. e.g. file:///path/to/test/sample.json

Thanks!

[filiagees]

Thanks for your fast reply @jonmmease

It's really interesting your thoughts about the different contexts!

The result was:

• full path -> the map is not displayed in jupyter cell (the legend is), but it's correctly written as file :)
• full URL path -> same behavior as full path!

On the other hand:

• relative path -> map is displayed ok, but it's not saved.

With geojson as dict it's displayed and saved properly.

Evidences:

click to see prints

full path:

URL full path:

[jonmmease]

Hi @filiagees,

Thanks for reporting on those experiments. Here's what I think is going on.

1. The script that launches the kaleido executable sets the working directory to the directory of the executable so relative paths are interpreted as relative to the executable instead of relative to the Python working directory.
2. For security reasons, browser generally block access to arbitrary files on the user's local file system and only allow access to data files with relative paths below the webserver's working directory.
3. Kaleido disables this restriction to allow for the use of absolute paths.

All that said, I think the way forward here is to avoid (1) and not have Kaleido change the working directory. I did this initially to make it easy for the C++ code to access it's own resource files using relative paths. We can avoid this by either having the C++ executable calculate the path to itself, or have the wrapper script pass this directory as an argument.

I'm going to change the name of this issue to describe what needs to be done. Thanks again for the report and helping out with the investigation!

[filiagees]

Thanks for your guidance @jonmmease ,

Based in your previous comment, I used strace to search for some failures similar to access(..."sample.json"...) = -1 ENOENT (No such file or directory).

The interesting thing I found out is that kaleido binary is trying the read the file in /tmp/[1].
Once I placed my geojson file there, Kaleido could save the file without 525 error! 🎉 🎉

What are the next steps?
Would be to add this bug in the project's backlog?

Appreciate for your patience!

(evidence)

[1]

[jonmmease]

Hi @filiagees, thanks for the investigation. I actually have no idea off hand why plotly.js would be looking for sample.json in the /tmp/ directory.

As I've thought about it more, I'm not sure that it makes sense for plotly to support local file paths for resources like geojson. The problem is that these figures won't be portable when exported to HTML, or when viewed in places like nbviewer. The most compatible way to do this is to either load the geojson from a CDN, or include it as a dict in the figure specification.

Are there use-cases you have in mind that aren't covered by these options?

[filiagees]

Your thought about exported html and nbviewer is really interesting, @jonmmease !

--
An use-case that is my concern is, under situations where a network connection is not available, therefore, no CDN is available.
I can imagine some corporate on-premisse application hosted inside a LAN, with no access to internet.

oh well... I can configure a http server to act as a local CDN, true fact; but it would become more complex setup than the original python & plotly script 😅

Another worry: due to poor network, the plotly scripts would take too much time to download the geojson[1] and render the output.
(some similar to this happened with a friend of mine: while presenting his web product during a conference event with weak wifi signal, the browser barely could download the assets)

[1] for instance, the original geojson file I was working on, is 27Mb big

feel free to ask for clarification... idk if I made myself clear :)

[jonmmease]

Thanks for that @filiagees. Do you have concerns with including the GeoJSON bundle as a dict in the figure specification? It definitely makes the figure specification larger, but what ends up being stored in memory and displayed in the browser (at least for a single figured) would be the same.

[filiagees]

No, I don't have any real concern, @jonmmease 👍

The only think that called my attention, as mentioned in the documentation, was the huge difference in time taken by px.choropleth() with a geojson vs dict¹.
But it's not a issue, indeed.

--

¹ In my scenario, coarsely measured, there was a ~270x difference (using ibge's geojson with 27Mb, and default_timer(); inside a loop)

[Saravanac5a]

Hi @filiagees, thanks for the report and debugging the issue a bit.

As the error mentions, it looks like the issue is with finding/loading the test/sample.json file. Is this exact plot working for you when you display it in another context (e.g. the notebook or a standalone html file)? If so, I'm thinking this might be an issue where kaleido doesn't preserve the current working directory and so relative paths aren't working the way you would expect.

Here are two things to try:

• Replace test/sample.json with the absolute path to your sample.json file. e.g. /path/to/test/sample.json
• Replace test/sample.json with a file:// url. e.g. file:///path/to/test/sample.json

Thanks!

Yes, this worked for me. It was a url but it somehow was not able to read it. I downloaded the safe file and entered the full path and it worked like a charm!

Thanks!

[gvwilson]

Thanks for your interest in Kaleido. We are currently working on an overhaul that might address your issue - we hope to have news in a few weeks and will post an update then. Thanks - @gvwilson