Fix matplotlib eventloops

[ianthomas23]

This fixes the use of GUI toolkit eventloops for Matplotlib plots in downstream libraries such as jupyterlab, qtconsole and jupyter-console which was broken in the 7.0.0 release. Fixes#1454. Here is a video of it in action:

mpl-qt.mp4

The primary change is to ensure that the event loops look for jupyter messages on the correct stream, which isn't necessarily the shell_stream. But I have also removed various functions (do_one_iteration, process_one, dispatch_queue etc) that passed messages to a shell via a message queue and ensured that only one was processed at a time. This functionality is now achieved by the asyncio_lock in shell_main and the zmq sockets queue the messages for this without specific extra queues. This in-process kernel tests that were using the do_one_iteration code path now use the asyncio_lock route, the same as the subshell-supporting full kernels. AFAICT no downstream libraries explicitly use this old code anyway.

I've implemented the fix for qt, tk (twice as Windows is different), wx and macosx eventloops. I've ignored the gtk eventloops as gtk3 has been deprecated here for a year and a half, and gtk doesn't work anyway as the correct name in Matplotlib is gtk4 so I assume that nobody has been using this otherwise they would have complained that it is not working.

In some situations there are warnings about resources being closed or not being closed when event loops are stopped. These may have been around before, but I don't know. Given that there is work underway in jupyter_client (jupyter/jupyter_client#1080) it is sensible to wait for that to be finished before looking at similar improvements here and in downstream libraries like qtconsole.

[ianthomas23]

Confirmed by OP in #1454 that this fixes the issue.

[minrk]

Awesome work, thanks for tackling this!

[jeandet]

@ianthomas23 , we are using that 'do_one_iteration' function inside our app , if you have any hint on what we should use instead (we need an in process kernel), that would be awesome.

[pauldmccarthy]

FWIW I have also been using do_one_iteration in FSLeyes for an in-process kernel. A little forewarning (via e.g. deprecation warnings at runtime) would be appreciated when you intend to remove functions from the API...

[pauldmccarthy]

@jeandet and anybody looking for an alternative - it appears that calling shell_stream.flush(limit=1) can be used in place of do_one_iteration(). @ianthomas23 does this sound reasonable?

[ianthomas23]

@jeandet and anybody looking for an alternative - it appears that calling shell_stream.flush(limit=1) can be used in place of do_one_iteration(). @ianthomas23 does this sound reasonable?

No, that is not always the correct stream to flush. Using eventloops.pyget_shell_stream is probably the way to go. But I'll need to check and confirm.

[pauldmccarthy]

@ianthomas23 Thanks - I don't envisage ever using sub-shells in my application, so for me it looks like eventloops.get_shell_stream(kernel) is equivalent to kernel.shell_stream. Still testing, but it seems to work just fine for me.

[jeandet]

@ianthomas23 , @pauldmccarthy , thank you for your help, get_shell_stream(self.kernel).flush(limit=1) seems to work so far.