Simplying Tkinter's event loop

[mdehoon]

BPO	1252236
Nosy	@loewis, @terryjreedy, @asvetlov
Files	_tkinter.c.diff: Patch for _tkinter.c

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = None
created_at = <Date 2005-08-04.21:51:48.000>
labels = ['type-feature', 'expert-tkinter']
title = "Simplying Tkinter's event loop"
updated_at = <Date 2019-04-26.18:38:42.954>
user = 'https://bugs.python.org/mdehoon'

bugs.python.org fields:

activity = <Date 2019-04-26.18:38:42.954>
actor = 'BreamoreBoy'
assignee = 'loewis'
closed = False
closed_date = None
closer = None
components = ['Tkinter']
creation = <Date 2005-08-04.21:51:48.000>
creator = 'mdehoon'
dependencies = []
files = ['6749']
hgrepos = []
issue_num = 1252236
keywords = ['patch']
message_count = 9.0
messages = ['48640', '48641', '48642', '48643', '48644', '89363', '114542', '127967', '222645']
nosy_count = 7.0
nosy_names = ['loewis', 'terry.reedy', 'jimjjewett', 'mdehoon', 'gpolo', 'asvetlov', 'PythonInTheGrass']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1252236'
versions = ['Python 3.5']

[mdehoon]

This patch is a simplification of Tkinter's event loop
by moving the event loop to Python itself.

Tkinter needs an event loop to be able to handle both
Tcl/Tk events and keyboard input by the user. Usually,
an event loop looks like this:

while true:
 wait for an event (Tcl/Tk events or keyboard input)
 handle the event

Tkinter's loop is set up differently. If Tkinter is
imported, it sets the PyOS_InputHook pointer to the
EventHook function in _tkinter.c. The EventHook
function handles Tcl/Tk events only. In essence,
Tkinter's event loop looks like this:

while true:
 handle all Tcl/Tk events that need to be handled
 if keyboard input is waiting: break
 sleep for 20 milliseconds

Note that the event loop exits only if the user presses
a key.
This causes the following problems:

1. This event loop is available only for Tkinter. There
   is no way to handle events other than the Tcl/Tk events.
2. If another extension module needs events to be
   handled, it can do so by using a similar event loop.
   However, it is not possible to handle both Tcl/Tk
   events and other events, (except for keyboard input).
3. Chaining the two event loops will fail. As soon as
   Tkinter's event loop starts running, it will break out
   of this loop only in the case of keyboard input. So any
   other events will not be processed until a key is pressed.
4. Even if Tkinter's event loop is the only event loop
   that is needed, it can fail. If a Python thread is
   waiting for a mutex variable to change, it cannot run
   Tkinter's event loop while waiting, since Tkinter's
   event loop will only return after a key is pressed (it
   doesn't pay attention to the mutex variable). This is
   the reason that running Tkinter in an IDLE shell will fail.
5. On some platforms (e.g. Cygwin), the event loop is
   unable to check for keyboard input. Hence, one needs to
   call mainloop() in order to see Tkinter's widgets.

The reason that the event loop was set up in this
peculiar way may be due to a bug with PyOS_InputHook
in Python. If Python is compiled with readline support,
PyOS_InputHook will be called ten times per second,
while Python is waiting for keyboard input. (As shown
below, this allows us to fix this event loop problem.)
However, if Python is compiled without readline support
(notably on Windows), PyOS_InputHook is called only
once (just before a user starts entering a new command
on the keyboard). This necessitates Tkinter to run an
event loop, and not return from it until keyboard input
is ready.

If PyOS_InputHook is called ten times per second, we
can set up the event loop as follows:

Set PyOS_InputHook to Tkinter's EventHook

while true:
 call PyOS_InputHook
 handle keyboard input, if present
 sleep for 100 milliseconds

Here, the key point is that Tkinter's EventHook returns
immediately if there are no more Tcl/Tk events to be
handled. Effectively, we have moved the event loop from
Tkinter to Python itself.

Hence, there are two problems to be solved:

1. PyOS_InputHook should be called ten times per
   second, regardless of whether Python is compiled with
   or without readline support.
2. Tkinter's EventHook should return immediately after
   handling all Tcl/Tk events.

Problem 1) is solved in patch bpo-1049855; problem 2) is
solved in this patch. This patch is a considerable
simplication of _tkinter.c, as it removes all lines
that are no longer needed (and adds almost no code).

This patch changes the nature of the function to which
PyOS_InputHook points. Before, PyOS_InputHook is used
to start a Tcl/Tk event loop. Now, PyOS_InputHook is
called from an existing event loop to handle Tcl/Tk events.

With these two patches, Python has a functioning
PyOS_InputHook, which can be used both by Tkinter and
by other extension modules. We can handle Tcl/Tk events
by calling Tkinter's EventHook function if we're
waiting for something else other than keyboard input,
like a mutex variable. We can chain event-handling
functions from different extension modules (e.g.
Tkinter and some other module), by first calling
Tkinter's EventHook and then the other extension
module's event hook function.
Finally, on Cygwin, we can now create Tkinter widgets
without having to call mainloop. So this will now work
on Cygwin:

>> from Tkinter import *
>> Label()
>> # label appears, without having to call mainloop.

This patch has been tested on WIndows and Cygwin.

[jimjjewett]

Logged In: YES
user_id=764593

(1) Why was the behavior different before? Is 10 times
per second not responsive enough? Does a busy-wait of
10 times per second cause too much thrashing?

(2) It seems like the problem isn't really about Tkinter so
much as it is about event loops vs threading. The event
loop is infinite, so nothing else *in that thread* will happen
after it. This isn't solvable with a single-threaded python.
(On the other hand, single-threaded python should never
have the mutex problem you mentioned.)

(3) With multi-threaded python, is there any reason not to
start the event loop in a fresh thread? (And let that new
thread block waiting for events.) This would also reduce
contention with other frameworks that want to treat the
"main" thread differently.

[mdehoon]

Logged In: YES
user_id=488897

(1) Why was the behavior different before?
Actually, the behavior is not much different from before;
this patch together with patch bpo-1049855 essentially move the
event loop from Tkinter to Python core, so that it is
available to other extension modules also. But effectively
the program goes through the same steps as before.
If you want to know why the design of Tkinter's event loop
is the way it is: I am not quite sure, but it may just be a
quick solution to get Tkinter working (and it does work fine
as long as you're interested in Tkinter only). Since Tcl/Tk
already has an event loop, it is easy to run that event loop
and let it check for keyboard input (via
Tcl_CreateFileHandler in EventHook in _tkinter.c). Writing
such a loop for Python is not extremely difficult but also
not straightforward (see my woes with patch bpo-1049855).

Is 10 times per second not responsive enough?
With the current Python, the loop sleeps every 20 ms before
checking for keyboard input and handling Tcl/Tk events. With
the patch, Tcl/Tk events are handled every 100 ms; keyboard
input is handled much faster (depending on how quickly
"select" on Unix or "WaitForSingleObject" on Windows
respond to keyboard input). Of course, these delays can be
set to some other value in the code.

Does a busy-wait of 10 times per second cause too much
thrashing?
I am not sure if I understand this question correctly. The
"select" function on Unix and "WaitForSingleObject" function
on Windows do not do a busy-wait.

(2) It seems like the problem isn't really about Tkinter so
much as it is about event loops vs threading. The event
loop is infinite, so nothing else *in that thread* will
happen
after it. This isn't solvable with a single-threaded
python.

Sure it's solvable. Even the current implementation of
Tkinter can handle Tcl/Tk events as well as listen for
keyboard input. If you import Tkinter and create a Tk
widget, you can still issue Python commands and move the Tk
widget around, even though they are running in the same
thread. The problem with the current implementation is that
it works for Tkinter only, and secondly, that it doesn't
allow chaining of hook functions. Patch bpo-1049855 solves this
by calling select on Unix (WaitForSingleObject on Windows)
to find out if keyboard input is available, and if not, to
handle Tk/Tcl events. No need for a separate thread for that.

(On the other hand, single-threaded python should never
have the mutex problem you mentioned.)

(3) With multi-threaded python, is there any reason not to
start the event loop in a fresh thread? (And let that new
thread block waiting for events.) This would also reduce
contention with other frameworks that want to treat the
"main" thread differently.

Yes, this is possible on multi-threaded python. However, an
extension module (such as Tkinter) cannot rely on the
assumption that Python is multi-threaded. Personally, I am
interested in PyOS_InputHook for scientific visualization
software, which is likely to be used on all kinds of
outlandish systems. I don't know if I can expect a
multi-threaded Python to be available on all those systems.
Secondly, given that Python has the PyOS_InputHook
functionality, why not make sure it is implemented
correctly? Meaning that its behavior should not depend on
whether readline is installed or not, and that its usage in
Tkinter does not block other extension modules from using it.

I am not sure if I interpreted all of your questions
correctly. Please let me know if I didn't.

[jimjjewett]

Logged In: YES
user_id=764593

Clarifying (1a) -- Why (pre-patch) were Windows and
Unix intentionally set to act differently? Is there something
in the default runtimes or libraries which makes
(expected?) performance very different on the platforms?

Clarifying (1b) -- Is ten times per second enough? 0.1
seconds is long enough that people can notice it. If the
pre-patch version cycles 50 times/second, then going to
only 10 times/second might make the interface seem
sluggish. I'm not sure I'm qualified to make this
judgement myself, but it is a concern.

Clarifying (1c) -- My (possibly wrong) understanding is
that before this pair of patches, unix effectively did an
active check for input, but the windows version waited for
notification from the OS that keyboard input was available
-- and after this, both would actively check N times/
second. If both are just passively waiting, then I'm not
sure what the 20ms/100ms timeout actually does.

If python is running as a batch process, then forcing it
back into the "is there input" section several times a
second (even though there is never any keyboard input)
will cause the program to take more clocktime.

Clarifying (2) -- The pre-patch version can certainly take
events (including keyboard events) during the event loop.
What you can't do is:

"""
(define/run a bunch of stuff)
...
start the event loop
...
(define/run a bunch more stuff)
"""

You need to set up all the definitions and event handlers
before the loop starts, or else do them as a result of
events. Roughly, calling mainloop has to be the *last*
thing you do in a given thread. Which leads to (3)

Clarifying (3) -- Why not just assume threads? The
problem you are trying to solve can't exist without threads.
Assuming threads won't make anything fail any harder
than it does now. If you default to dummy-threads and
ensure that the event-loop the *last* pseudo-thread, you'll
even clear up some bugs in carelessly written single-
threaded code.

[mdehoon]

Logged In: YES
user_id=488897

Clarifying (1a) -- Why (pre-patch) were Windows and
Unix intentionally set to act differently? Is there
something
in the default runtimes or libraries which makes
(expected?) performance very different on the platforms?

The difference is not Windows versus Unix, but
Unix-with-readline versus Unix-without-readline and
Windows-without-readline. I wasn't there when PyOS_InputHook
was first added to Python, but I doubt that the difference
in behavior was intentional. Essentially, I think that the
behavior of Tkinter (on Python-without-readline) is a design
flaw.

Clarifying (1b) -- Is ten times per second enough? 0.1
seconds is long enough that people can notice it. If the
pre-patch version cycles 50 times/second, then going to
only 10 times/second might make the interface seem
sluggish. I'm not sure I'm qualified to make this
judgement myself, but it is a concern.

Good point. The 10 times/second is the default used in
readline, and hence this is the frequency at which
PyOS_InputHook is called in Python-with-readline. I can
modify the patch to increase this to 50 times/second.

Clarifying (1c) -- My (possibly wrong) understanding is
that before this pair of patches, unix effectively did an
active check for input, but the windows version waited for
notification from the OS that keyboard input was available
-- and after this, both would actively check N times/
second. If both are just passively waiting, then I'm not
sure what the 20ms/100ms timeout actually does.

That is correct. Pre-patch, both Unix and Windows sit in a
loop inside Tkinter's EventHook function that calls
Tcl_DoOneEvent and check for keyboard input, and then sleep
for 20 ms. The check for keyboard input is done via a call
to _kbhit (on Windows) or select (on Unix, hiding inside
Tcl/Tk's Tcl_DoOneEvent function). Post-patch, this loop is
located inside the my_fgets function. It works essentially
the same, except that I'm using WaitForSingleObject (on
Windows) / select (on Unix) for the timeout.

If python is running as a batch process, then forcing it
back into the "is there input" section several times a
second (even though there is never any keyboard input)
will cause the program to take more clocktime.

Also a good point. I will modify the patch such that it will
skip to fgets immediately if PyOS_InputHook==NULL. In a
batch process, there is no reason to load a GUI extension
module such as Tkinter, so PyOS_InputHook should be NULL.

Clarifying (2) -- The pre-patch version can certainly take
events (including keyboard events) during the event loop.
What you can't do is:

"""
(define/run a bunch of stuff)
...
start the event loop
...
(define/run a bunch more stuff)
"""

Sure you can. I do it all the time. Try this, for example:
>>> from Tkinter import *
>>> Label(text="Label1").pack()
# ... Label1 appears
>>> print "more stuff I want to do"
more stuff I want to do

Here, Label1 appears and responds to events, even though I
did not run mainloop. Note that pre-patch, this does not
work on Cygwin; post-patch, it works on Cygwin too.

You need to set up all the definitions and event handlers
before the loop starts, or else do them as a result of
events. Roughly, calling mainloop has to be the *last*
thing you do in a given thread. Which leads to (3)

You don't need to call mainloop at all. This is quite useful
for scientific visualization:
>>> do_some_calculation()
>>> plot_the_result()
>>> if dont_like_the_plot: do_some_more_calculations()
>>> plot_the_new_result()
Here, you never need to start or stop the event loop. It
continues running as long as PyOS_InputHook is set
appropriately by the extension module that does the plotting.

Clarifying (3) -- Why not just assume threads? The
problem you are trying to solve can't exist without threads.
Assuming threads won't make anything fail any harder
than it does now. If you default to dummy-threads and
ensure that the event-loop the *last* pseudo-thread, you'll
even clear up some bugs in carelessly written single-
threaded code.

Whether or not Tkinter should have been coded with threads
is a separate question. Tkinter is written for a single
thread, using an event loop to handle window events and
keyboard input. While it is possible to rewrite Tkinter to
use threads, that would constitute a separate patch (and
would probably not have a good chance of getting accepted,
since Tkinter already works well as a single thread).

[gpolo]

Michiel, the patch on bpo-1049855 has been rejected so there is no longer a
fix for the first problem to be solved.
Also, the fourth problem you described is not entirely true, it is
possible to run tkinter apps on IDLE without calling the mainloop
function (see bpo-989712). It is possible that this wasn't the case 4 years
ago, I don't know, just pointing out the current situation.

I also don't understand how just moving event handling to Python will
solve the problems. But maybe it could be good to add an option that
would prevent calling EnableEventHook in _tkinter after loading tk, so
the user could handle tk events and what not the way he whises.

Is there anyone still interested on this ? Please ?

[BreamoreBoy]

No reply to msg89363.

[terryjreedy]

See 11077 for motivation for reopening.

[BreamoreBoy]

I've just asked for an update on bpo-11077 .