Adds support for all Jupyter kernels to PythonTeX

[nathancarter]

I'm pretty excited about this! Hope you find it useful. A few notes:

1. Should silently do nothing if you don't have Jupyter installed.
2. If you have Jupyter installed, then when you run PythonTeX, you'll see it print the list of Jupyter kernels installed, and how it chose to name them.
3. I'm sure there are many features and tweaks that will be needed, but it does actually work in this rudimentary form.

[gpoore]

This is a great way to add Jupyter support while working around the limitations of the current code execution system!

I may have a couple minor improvements for interfacing with PythonTeX. My only substantive comment is about using Jupyter kernels. Something like \begin{jupyterjavascriptcode} works fine for a kernel like javascript that has a name composed only of [a-z]+. However, there are also kernels like python2 and python3. This presents a problem because LaTeX macro names by default can only take the form [A-Za-z]+. One option would be to convert numbers to words, so python3 is used via \begin{jupyterpythonthreecode}. Another option, which should be easy with a little macro programming on the LaTeX side, would be something like \begin{jupytercode}{python3}. Basically, when Jupyter is in use, the environment is named Jupyter, and there's an extra mandatory argument that specifies the kernel. Do you have any preferences between those options, or other suggestions?

We can discuss additional refinements and features later, but just so I can be thinking about future directions, do you eventually want automatic inclusion of rich output like LaTeX representations of output or images?

[nathancarter]

Right now (see this line) I just drop all offending characters from a kernel name. This will, unfortunately, make python2 and python3 into "jupyterpython" which is a collision. I love your idea of supporting them by recognizing \begin{jupytercode}{X} as a reference to the CodeEngine with name "jupyter"+X.

Eventually, I'd love to support lots of kinds of rich output from Jupyter, but I just wanted to see if I could get a simple proof of concept working at first. If this passes your tests smoothly, then we can talk about adding things like image support and so forth afterwards.

[asinghvi17]

Has there been any progress on this on the LaTeX side?

[nathancarter]

I didn't realize so much time had elapsed with nothing being done on this. Any word? Any way I can help?

[gpoore]

@nathancarter Sorry for the lack of progress on this. What you've provided on the Python side looks like it would provide basic functionality for stdout and stderr. I think the main thing remaining there is image support for rich output. I have some experience with that from Codebraid and don't see that as particularly difficult, at least for basic cases.

The real issue is the LaTeX side. My time for open-source software has been somewhat limited for an extended time, most recently largely due to the pandemic. I find LaTeX programming something like 10-1000x less productive than languages like Python, so when my time is limited it's difficult to justify spending it on LaTeX. Getting Jupyter kernels working sooner rather than later will probably mean that someone besides me will have to implement the LaTeX side. That might not be too difficult to just get stdout and stderr working, but getting support for rich output (images) will need a new set of features that provides a \richoutputpythontex or something similar that is analogous to the existing \stdoutpythontex and \stderrpythontex. The existing \stdoutpythontex won't really work for Jupyter stdout mixed with images, because stdout might not be valid LaTeX and thus might need to be displayed verbatim. However, if stdout is mixed with \includegraphics commands for images, showing it verbatim will disable image inclusion.

[nathancarter]

@gpoore Thanks for the update! Let's make the important distinction between the three very different benefits there are to having Jupyter supported in PythonTeX.

1. The community immediately gets an absolute boatload of new PythonTeX languages supported right away for free.
2. You can stop inventing new PythonTeX kernels, because Jupyter has already solved that problem and it makes sense to just re-use their (let's be honest--industry standard) work.
3. Jupyter supports rich output and it would be possible to upgrade PythonTeX to support it.

The reason it's important to split these out is because 1+2 require almost no work at all and no LaTeX programming and can be done right away and bring those benefits now, while item 3 is the one that requires tons of work, including LaTeX programming. (I totally sympathize, by the way--I hope to never write a real program in LaTeX ever.)

The takeaway: I wouldn't want the existence of item 3 as a possible later upgrade to block the immediate potential benefits of items 1+2. Item 3 should be classified as a possible later upgrade, not a blocker for items 1+2.

[gpoore]

@nathancarter Those distinctions are helpful. For the things I do, rich output is really the only reason to ever use a Jupyter kernel outside of a Jupyter notebook.

To support Jupyter kernels only with stdout and stderr output, here's (at least most of) what's needed.

1. An interface from the LaTeX side. Ideally, I'd probably want something like \begin{jupyter<type>}[<session>][<fancyvrb>]{<kernel>}, where jupyter<type> would be jupytercode, jupyterblock, and jupyterverbatim similar to the existing pycode, pyblock, etc. Only code, block, and verbatim environments will be possible, at least at first (no sub). Without rich output, it's not worthwhile (or maybe really possible) to make a Jupyter equivalent of \py. Inline commands corresponding to the environments should be possible, something like \jupyc, \jupyv, and \jupyb. Inline could also be omitted for a first implementation. This will require some LaTeX programming and possibly a few changes on the Python side. LaTeX programming could be almost entirely eliminated by using a syntax like \begin{jupyterpythoncode} (kernel isn't a separate argument). That would make implementation easy, but also wouldn't work with kernel names containing anything but [a-zA-Z]. Also, at some point these long environment names become awkward. It would be possible to start with something like \begin{jupyterpythoncode} and then add an alternative syntax later.
2. Documentation.
3. Ideally, there needs to be at least one very short example document under test/ for checking functionality.
4. There will probably need to be a few small modifications to the JupyterCodeEngine implementation and related code. For example, setting kernel to python should get python3 kernel, and setting kernel to julia should find a Julia kernel even if the kernel name ends with a version number (and if there are multiple kernels like that, presumably choose the most recent).

[dimpase]

In case, sphinx has extensions which use Jupyter (kernels) to typeset Python docs. So that's not something very unique here.