Fix context in protocol callbacks by fantix · Pull Request #348 · MagicStack/uvloop

fantix · 2020-05-25T23:02:34Z

A quick recap of uvloop core:

Loop is the uvloop-version of an asyncio event loop, exposing APIs to create UVHandles like TCP transports.
UVHandle is the base class of uvloop wrappers of the libuv uv_handle_t structs, see the full family below.
Each UVHandle references ~~one~~ zero or more Handle instances that encapsulate the actual callback, its arguments, and a PEP-567 context.
There's at least one cdef function per UVHandle that is registered to the libuv uv_handle_t struct. This function usually just triggers running the Handle.
[UPDATE] Not all UVHandle uses Handle for callbacks, and we don't want to change that.

UVHandle (handle)
+- UVAsync (async_)
+- UVCheck (check)
+- UVIdle (idle)
+- UVPoll (poll)
+- UVProcess (process)
| +- UVProcessTransport (process)
+- UVSocketHandle (handle)
| +- UVBaseTransport (basetransport)
| | +- UDPTransport (udp)
| | +- UVStream (stream)
| | +- TCPTransport (tcp)
| | +- UnixTransport (pipe)
| | +- ReadUnixTransport (pipe)
| | +- WriteUnixTransport (pipe)
| +- UVStreamServer (streamserver)
| +- TCPServer (tcp)
| +- UnixServer (pipe)
+- UVTimer (timer)

What is missing is e.g. the listen_handle of some UVHandle subclasses like the UVStreamServer. For now, it calls the actual callback (_on_listen) directly without PEP-567 context. So this PR should add those missing Handles.

The tricky part is, the current Handle supports only fixed parameters provided at initialization, what is needed for callbacks like _on_listen or __uv_stream_on_read is a partial-like Handle._run_with_param().

List of UVHandles to check:

Principals

Each UVHandle instance (including transports, servers, etc) sticks to one context where it was created from and/or started to take effect.
All user code triggered by UVHandle (mostly protocol callbacks, but also protocol factory) runs in the same context.
Affiliated UVHandle instances share the same context - for example, protocol callbacks are always called in the same context even though the transport is upgraded by start_tls() in a different context (even multiple times, a.k.a. SSL over SSL).
By "the same context", it's not necessarily the same Context instance - it can be a copied instance. Therefore, changes to a ContextVar are not always carried between different protocol callbacks. But all callbacks could see the same inherited value from the context where the UVHandle was created or started.
When in need, we lean towards using Context.copy() or copy_context() in user-triggered events, and reuse existing Context instances for uvloop-triggered events, so as to avoid re-entering the same context twice.
The above copy_context() is only applied to direct method calls, callbacks through e.g. loop.call_soon() are not considered necessary to copy the context.
Don't over-optimize. The context copy operation is actually fast (underlying context data is not copied because of the copy-on-write design), it is okay to copy when unnecessary if it means to avoid complication.

Example 1: Client Protocol

importasynciofromcontextvarsimportContextVarimportuvloopuvloop.install()
cvar=ContextVar("cvar", default="in initial context")
classProtocol(asyncio.Protocol):
def__init__(self):
self.pipe=asyncio.Queue()
defconnection_made(self, transport):
self.pipe.put_nowait("connection_made() "+cvar.get())
defdata_received(self, data):
self.pipe.put_nowait("data_received() "+cvar.get())
defconnection_lost(self, exc):
self.pipe.put_nowait("connection_lost() "+cvar.get())
asyncdefmain():
cvar.set("in context of create_connection()")
trans, proto=awaitasyncio.get_event_loop().create_connection(
Protocol, "google.com", 80
)
print(awaitproto.pipe.get())
cvar.set("in context of write()")
trans.write(b"GET / HTTP.1.1\r\nHost: google.com\r\n\r\n")
print(awaitproto.pipe.get())
cvar.set("in context of close()")
trans.close()
print(awaitproto.pipe.get())
asyncio.run(main())

Expected result:

connection_made() in context of create_connection()
data_received() in context of create_connection()
connection_lost() in context of create_connection()

Example 2: Server Protocol

importasyncioimportsocketfromcontextvarsimportContextVarimportuvloopuvloop.install()
cvar=ContextVar("cvar", default="in initial context")
classProtocol(asyncio.Protocol):
def__init__(self, pipe):
self.pipe=pipeself.pipe.put_nowait("Protocol() "+cvar.get())
defconnection_made(self, transport):
self.pipe.put_nowait("connection_made() "+cvar.get())
defdata_received(self, data):
self.pipe.put_nowait("data_received() "+cvar.get())
defconnection_lost(self, exc):
self.pipe.put_nowait("connection_lost() "+cvar.get())
asyncdefmain():
loop=asyncio.get_event_loop()
run=lambda*args: loop.run_in_executor(None, *args)
pipe=asyncio.Queue()
cvar.set("in context of create_server()")
server=awaitloop.create_server(
lambda: Protocol(pipe), "127.0.0.1", 0
)
s=socket.socket()
awaitrun(s.connect, server.sockets[0].getsockname())
print(awaitpipe.get())
print(awaitpipe.get())
awaitrun(s.send, b'data')
print(awaitpipe.get())
awaitrun(s.close)
print(awaitpipe.get())
asyncio.run(main())

Expected result:

Protocol() in context of create_server()
connection_made() in context of create_server()
data_received() in context of create_server()
connection_lost() in context of create_server()

Example 3: `start_serving`

Similar to example 2, but start serving in a different context:

asyncdefmain():
...
cvar.set("in context of create_server()")
server=awaitloop.create_server(
lambda: Protocol(pipe), "127.0.0.1", 0, start_serving=False,
)
cvar.set("in context of start_serving()")
awaitserver.start_serving()
...

Expected result:

Protocol() in context of start_serving()
connection_made() in context of start_serving()
data_received() in context of start_serving()
connection_lost() in context of start_serving()

Example 4: TLS Upgrade

importasyncioimportsocketimportsslfromcontextvarsimportContextVarimportuvloopuvloop.install()
cvar=ContextVar("cvar", default="in initial context")
classProtocol(asyncio.Protocol):
def__init__(self, pipe):
self.pipe=pipeself.pipe.put_nowait("Protocol() "+cvar.get())
defconnection_made(self, transport):
self.pipe.put_nowait((transport, self))
self.pipe.put_nowait("connection_made() "+cvar.get())
defdata_received(self, data):
self.pipe.put_nowait("data_received() "+cvar.get())
defconnection_lost(self, exc):
self.pipe.put_nowait("connection_lost() "+cvar.get())
asyncdefmain():
loop=asyncio.get_event_loop()
run=lambda*args: loop.run_in_executor(None, *args)
pipe=asyncio.Queue()
cvar.set("in context of create_server()")
server=awaitloop.create_server(
lambda: Protocol(pipe),
"127.0.0.1",
0,
start_serving=False,
)
cvar.set("in context of start_serving()")
awaitserver.start_serving()
s=socket.socket()
awaitrun(s.connect, server.sockets[0].getsockname())
print(awaitpipe.get())
trans, proto=awaitpipe.get()
print(awaitpipe.get())
awaitrun(s.send, b"data")
print(awaitpipe.get())
sslctx=ssl.SSLContext()
sslctx.load_cert_chain("tests/certs/ssl_cert.pem", "tests/certs/ssl_key.pem")
client_sslctx=ssl.create_default_context()
client_sslctx.check_hostname=Falseclient_sslctx.verify_mode=ssl.CERT_NONEs=run(client_sslctx.wrap_socket, s)
cvar.set("in context of start_tls()")
awaitloop.start_tls(trans, proto, sslctx, server_side=True)
s=awaitsawaitrun(s.send, b"data")
print(awaitpipe.get())
awaitrun(s.close)
print(awaitpipe.get())
asyncio.run(main())

Expected result:

Protocol() in context of start_serving()
connection_made() in context of start_serving()
data_received() in context of start_serving()
data_received() in context of start_serving()
connection_lost() in context of start_serving()

1st1 · 2020-05-25T23:08:33Z

My memory here is a little rusty, but can we instead change

uvloop/uvloop/cbhandles.pyx

Lines 338 to 421 in 465717f

 cdef new_Handle(Loop loop, object callback, object args, object context): 
 cdef Handle handle 
 handle = Handle.__new__(Handle) 
 handle._set_loop(loop) 
 handle._set_context(context) 
 handle.cb_type =1
 handle.arg1 = callback 
 handle.arg2 = args 
return handle 
 cdef new_MethodHandle(Loop loop, str name, method_t callback, object ctx): 
 cdef Handle handle 
 handle = Handle.__new__(Handle) 
 handle._set_loop(loop) 
 handle._set_context(None) 
 handle.cb_type =2
 handle.meth_name = name 
 handle.callback =<void*> callback 
 handle.arg1 = ctx 
return handle 
 cdef new_MethodHandle1(Loop loop, str name, method1_t callback, 
object ctx, object arg): 
 cdef Handle handle 
 handle = Handle.__new__(Handle) 
 handle._set_loop(loop) 
 handle._set_context(None) 
 handle.cb_type =3
 handle.meth_name = name 
 handle.callback =<void*> callback 
 handle.arg1 = ctx 
 handle.arg2 = arg 
return handle 
 cdef new_MethodHandle2(Loop loop, str name, method2_t callback, object ctx, 
object arg1, object arg2): 
 cdef Handle handle 
 handle = Handle.__new__(Handle) 
 handle._set_loop(loop) 
 handle._set_context(None) 
 handle.cb_type =4
 handle.meth_name = name 
 handle.callback =<void*> callback 
 handle.arg1 = ctx 
 handle.arg2 = arg1 
 handle.arg3 = arg2 
return handle 
 cdef new_MethodHandle3(Loop loop, str name, method3_t callback, object ctx, 
object arg1, object arg2, object arg3): 
 cdef Handle handle 
 handle = Handle.__new__(Handle) 
 handle._set_loop(loop) 
 handle._set_context(None) 
 handle.cb_type =5
 handle.meth_name = name 
 handle.callback =<void*> callback 
 handle.arg1 = ctx 
 handle.arg2 = arg1 
 handle.arg3 = arg2 
 handle.arg4 = arg3 
return handle 

to automatically capture the current context?

fantix · 2020-05-26T00:32:59Z

Maybe yes - they're intended to bridge the native callbacks anyways, right? Submitted a commit to try this approach. I had to hack in the arg2 value before calling _run().

1st1 · 2020-05-26T01:49:35Z

Maybe yes - they're intended to bridge the native callbacks anyways, right? Submitted a commit to try this approach. I had to hack in the arg2 value before calling _run().

I think we need parallel APIs like new_MethodHandleNWithContext.

1st1 · 2020-05-26T01:51:40Z

Or add a trailing argument to all new_MethodHandleN methods which can be either a context, or a boolean with True/False, with True meaning "copy the current context", and False meaning run in empty context. As I said my knowledge of the code base is a bit rusty, but I believe that this is a way better approach than to handle context in UVHandles.

fantix · 2020-05-26T03:19:08Z

Got it, that makes sense - I'll then try to propose something more complete.

fantix · 2020-12-13T17:17:26Z

/me is trying to fix this one now.

1st1 · 2020-12-15T18:53:33Z

/me is trying to fix this one now.

Nice, what's the latest? ;)

1st1 · 2021-01-21T19:58:09Z

+ | +- UnixServer (pipe)
+ +- UVTimer (timer)
+"""
+


This is awesome, thanks for adding it

1st1

looks good so far!

1st1

LGTM!

This is a combined fix to correct contexts from which protocal callbacks are invoked. In short, callbacks like data_received() should always be invoked from consistent contexts which are copied from the context where the underlying UVHandle is created or started. The new test case covers also asyncio, but skipping the failing ones.

fantix force-pushed the uvhandle-context branch from 5f36d64 to 32979f8Compare January 21, 2021 19:27

1st1 reviewed Jan 21, 2021
View reviewed changes

fantix force-pushed the uvhandle-context branch 4 times, most recently from 70b6ae4 to 81d3900Compare January 26, 2021 19:54

fantix changed the title ~~[WIP] Uvhandle context~~[WIP] UVHandle contextJan 27, 2021

fantix force-pushed the uvhandle-context branch 7 times, most recently from 6f41c33 to e9b4056Compare February 3, 2021 16:08

1st1 reviewed Feb 3, 2021
View reviewed changes

fantix force-pushed the uvhandle-context branch from ab49e7a to b41a50dCompare February 5, 2021 15:28

fantix changed the title ~~[WIP] UVHandle context~~Fix context in protocol callbacksFeb 5, 2021

fantix marked this pull request as ready for review February 5, 2021 19:10

1st1 approved these changes Feb 5, 2021
View reviewed changes

versusvoidand others added 2 commits February 5, 2021 15:14

Restore context on listen in UVStreamServer.FixMagicStack#305
ed6561d

fantix force-pushed the uvhandle-context branch from 6c2d403 to 41844b6Compare February 5, 2021 20:22

fantix merged commit f691212 into MagicStack:masterFeb 5, 2021

fantix deleted the uvhandle-context branch February 5, 2021 20:38

This was referenced Feb 5, 2021
Restore context on listen in UVStreamServer. Fix #305 #306
Closed
Remove import tests for testinstalled #395
Merged

samypr100 mentioned this pull request Apr 4, 2026
fix(tests): explicit yield in test_create_ssl_server_manual_connection_lost #743
Merged

Navigation Menu

Uh oh!

Fix context in protocol callbacks - #348

Fix context in protocol callbacks#348
fantix merged 2 commits into
MagicStack:masterfrom
fantix:uvhandle-context

fantix commented May 25, 2020•
edited
Loading

Uh oh!

1st1 commented May 25, 2020

Uh oh!

fantix commented May 26, 2020

Uh oh!

1st1 commented May 26, 2020

Uh oh!

1st1 commented May 26, 2020

Uh oh!

fantix commented May 26, 2020

Uh oh!

fantix commented Dec 13, 2020

Uh oh!

1st1 commented Dec 15, 2020

Uh oh!

1st1Jan 21, 2021

Uh oh!

Uh oh!

Uh oh!

1st1 left a comment

Uh oh!

1st1 left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

fantix commented May 25, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Principals

Example 1: Client Protocol

Example 2: Server Protocol

Example 3: start_serving

Example 4: TLS Upgrade

Uh oh!

1st1 commented May 25, 2020

Uh oh!

fantix commented May 26, 2020

Uh oh!

1st1 commented May 26, 2020

Uh oh!

1st1 commented May 26, 2020

Uh oh!

fantix commented May 26, 2020

Uh oh!

fantix commented Dec 13, 2020

Uh oh!

1st1 commented Dec 15, 2020

Uh oh!

1st1Jan 21, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

1st1 left a comment

Choose a reason for hiding this comment

Uh oh!

1st1 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fantix commented May 25, 2020•
edited
Loading

Example 3: `start_serving`