[Python] Make pyarrow installable in Jupyterlite

[MarcSkovMadsen]

Describe the enhancement requested

I'm trying to use pyarrow with Panel in Panelite. Panelite is a custom build of Jupyterlite that works with Panel.

The issue is that piplite cannot install pyarrow. It would be great if pyarrow also worked with Pyodide and in Jupyterlite/ Panelite.

Thanks.

---

import piplite
await piplite.install(['panel', 'xlsxwriter', 'pyarrow'])

---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[16], line 2
 1 import piplite
----> 2 await piplite.install(['panel', 'xlsxwriter', 'pyarrow'])

File /lib/python3.10/asyncio/futures.py:284, in Future.__await__(self)
 282 if not self.done():
 283 self._asyncio_future_blocking = True
--> 284 yield self # This tells Task to wait for completion.
 285 if not self.done():
 286 raise RuntimeError("await wasn't used with future")

File /lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future)
 302 def __wakeup(self, future):
 303 try:
--> 304 future.result()
 305 except BaseException as exc:
 306 # This may also be a cancellation.
 307 self.__step(exc)

File /lib/python3.10/asyncio/futures.py:201, in Future.result(self)
 199 self.__log_traceback = False
 200 if self._exception is not None:
--> 201 raise self._exception
 202 return self._result

File /lib/python3.10/asyncio/tasks.py:234, in Task.__step(***failed resolving arguments***)
 232 result = coro.send(None)
 233 else:
--> 234 result = coro.throw(exc)
 235 except StopIteration as exc:
 236 if self._must_cancel:
 237 # Task is cancelled right before coro stops.

File /lib/python3.10/site-packages/piplite/piplite.py:102, in _install(requirements, keep_going, deps, credentials, pre)
 100 """Invoke micripip.install with a patch to get data from local indexes"""
 101 with patch("micropip._micropip._get_pypi_json", _get_pypi_json):
--> 102 return await _micropip.install(
 103 requirements=requirements,
 104 keep_going=keep_going,
 105 deps=deps,
 106 credentials=credentials,
 107 pre=pre,
 108 )

File /lib/python3.10/site-packages/micropip/_micropip.py:573, in install(requirements, keep_going, deps, credentials, pre)
 563 wheel_base = Path(getsitepackages()[0])
 565 transaction = Transaction(
 566 ctx=ctx,
 567 ctx_extras=[],
 (...)
 571 fetch_kwargs=fetch_kwargs,
 572 )
--> 573 await transaction.gather_requirements(requirements)
 575 if transaction.failed:
 576 failed_requirements = ", ".join([f"'{req}'" for req in transaction.failed])

File /lib/python3.10/site-packages/micropip/_micropip.py:333, in Transaction.gather_requirements(self, requirements)
 330 for requirement in requirements:
 331 requirement_promises.append(self.add_requirement(requirement))
--> 333 await gather(*requirement_promises)

File /lib/python3.10/asyncio/futures.py:284, in Future.__await__(self)
 282 if not self.done():
 283 self._asyncio_future_blocking = True
--> 284 yield self # This tells Task to wait for completion.
 285 if not self.done():
 286 raise RuntimeError("await wasn't used with future")

File /lib/python3.10/asyncio/tasks.py:304, in Task.__wakeup(self, future)
 302 def __wakeup(self, future):
 303 try:
--> 304 future.result()
 305 except BaseException as exc:
 306 # This may also be a cancellation.
 307 self.__step(exc)

File /lib/python3.10/asyncio/futures.py:201, in Future.result(self)
 199 self.__log_traceback = False
 200 if self._exception is not None:
--> 201 raise self._exception
 202 return self._result

File /lib/python3.10/asyncio/tasks.py:232, in Task.__step(***failed resolving arguments***)
 228 try:
 229 if exc is None:
 230 # We use the `send` method directly, because coroutines
 231 # don't have `__iter__` and `__next__` methods.
--> 232 result = coro.send(None)
 233 else:
 234 result = coro.throw(exc)

File /lib/python3.10/site-packages/micropip/_micropip.py:340, in Transaction.add_requirement(self, req)
 337 return await self.add_requirement_inner(req)
 339 if not urlparse(req).path.endswith(".whl"):
--> 340 return await self.add_requirement_inner(Requirement(req))
 342 # custom download location
 343 wheel = WheelInfo.from_url(req)

File /lib/python3.10/site-packages/micropip/_micropip.py:435, in Transaction.add_requirement_inner(self, req)
 432 metadata = await _get_pypi_json(req.name, self.fetch_kwargs)
 434 try:
--> 435 wheel = find_wheel(metadata, req)
 436 except ValueError:
 437 self.failed.append(req)

File /lib/python3.10/site-packages/micropip/_micropip.py:303, in find_wheel(metadata, req)
 300 if best_wheel is not None:
 301 return wheel
--> 303 raise ValueError(
 304 f"Can't find a pure Python 3 wheel for '{req}'.\n"
 305 f"See: {FAQ_URLS['cant_find_wheel']}\n"
 306 "You can use `micropip.install(..., keep_going=True)`"
 307 "to get a list of all packages with missing wheels."
 308 )

ValueError: Can't find a pure Python 3 wheel for 'pyarrow'.
See: https://pyodide.org/en/stable/usage/faq.html#micropip-can-t-find-a-pure-python-wheel
You can use `micropip.install(..., keep_going=True)`to get a list of all packages with missing wheels.

Component(s)

Python

[MarcSkovMadsen]

Related to pyodide/pyodide#2933

[eliasdabbas]

+1

pyarrow is becoming more crucial with pandas 2.0 as well. Would be great to have it supported.

[westonpace]

I'm not super familiar with pyodide. Is the only solution to have a pure python wheel? There have been other efforts to get arrow-c++ to be usable with emscripten. If this were done would the C++ then be usable with pyodide?

[joemarshall]

I am 90% there on arrow in pyodide. I have a running build of pyarrow here, but it needs a little work to be generally buildable by anyone. And a little more work so that it loads straight into pyodide nicely.

This PR:
#35471
is the bulk of the work, because for pyodide / browsers we require a build of arrow cpp that works okay without threads.

Once that is merged, there are a few minor build file changes to add support and cmake presets for emscripten & pyodide - this isn't that hard because we've done a bunch of work in pyodide-build which simplifies stuff. Oh and that all needs adding to the github CI here or else people will break it over time.

Then finally, once there is a sensible build process for emscripten pyarrow, it will probably make sense to add a recipe to pyodide so that it is distributed with core pyodide. That is pretty straightforward (I've done a few of them before) - basically pretty much a matter of pointing it at the git repository tag and telling it the build command to make arrow-cpp and pyarrow.

[elehcimd]

Are there any updates on this? Thanks for your incredible work

[kylebarron]

It'll be included as an available package in the next pyodide release: pyodide/pyodide#4950. Or you can use wheels from https://github.com/joemarshall/pyarrow-pyodide/releases/tag/0.26.2 if you'd like to try it sooner.