Proposal: Event loop redesign

[kprotty]

The current event loop is not ready yet (relatively slow, windows unfinished, bugs/races) and many wish for it to be. From the discord communities at least, there seems to be enough interest to warrant addressing this, with some interested in helping out. My question then is: what do people really want out of the standard libraries event loop? Toying with some designs locally produced incomplete implementations from lack of a direction. In addition to what people want from an event loop, how do they also want it implemented? Here are some example properties:

• Multi-threaded by default?
  • Threading is customizable (NUMA, affinities, priorities)?
  • Work-stealing or work-requesting scheduler? configurable?
• Globally configured?
  • Other standard library stuff subscribing to the event loop automatically/case-by-case?
  • Multiple event loops in the same program?
• Performance goals?
  • Prioritize simple implementation over absolute efficiency?
  • What sort of sacrifices should be made here to accommodate other metrics?
  • Is there a definite goal it should strive to maintain (e.g. faster than X in Y)?
• What features should it provide API wise and how would those look like?
  • Ability to lock execution to a specific OS thread?
  • Influence scheduling (hard/soft priorities, QoS classes, fairness)?
  • Scheduling patterns (batch, switchto, scheduling cancellation)?
• What should sync primitives (i.e. Mutex, Channel) and event sources (i.e. I/O, timers) look like?
  • Support cancellation?
  • Are allowed to dynamically allocate (i.e. heap alloc) internally?
  • Require user-provided, intrusive memory to operate?
• How async should it be?
  • Usable as a C api?
  • Usable without zig features (i.e. comptime, threadlocal, async) for easier bootstrap?
• Should we even have an event loop at all?
  • Only provide thread pool / io-poller / timer datastructures separately?
  • GCD/libdispatch approach? (user creates distinct/specific work-queues and pushes work into them per app)
  • Global thread-pool/executor approach? (think windows ThreadPool or windows UMS)

I'm open to adding more considerations if others have some. Just want to understand the requirements before making any sort of PR.

[kristoff-it]

Assuming we continue with the idea of having an "event loop" abstraction, I would like to see from it some form of consistency across different OSs, at least when it comes to the big 3.

More specifically, I believe one important goal should be to ensure, as much as possible, that if an application doesn't deadlock on OS Y it also doesn't deadlock on OS X (hah pun kinda intended). I am not sure what is the current state of things, but one example that comes to mind is file I/O which only simulates evented I/O by using a thread pool.

Somewhat related to the above goal, there should be an API (or a clear pattern) for dealing with blocking I/O (and CPU intensive operations). As a concrete example, in bork I used to steal one thread from the event loop when using BearSSL and, depending on the CPU configuration of the host (and other implementation details of the event loop that fortunately in this case didn't pose a problem) it could simply deadlock.

If instead the event loop gets broken down into discrete components, then this question IMO still stands and would need to be re adapted for the new context.

[haze]

I want one that prints "bruh" asynchronously

[lithdew]

I'll put my high-level thoughts down in points here since my thoughts are a little all over the place when it comes to Zig's async story. The perspective I'm coming from with my thoughts here is after spending the last few months implementing a few resource-constrained high-performance distributed systems and async-driven sensor applications in Zig, and spending the last month researching how async works in other programming languages.

---

1. It's really, really hard to design for a general-purpose, all-in-one event loop that everyone can be happy with. I don't think we should aim towards such an amalgamation, and that we should instead aim towards creating production-ready modules (I/O reactors, I/O proactors, thread pool executors, synchronization primitives) that a developer may then combine into an event loop design that they are happy with.

I personally have a lot of frustrations over some of the design decisions in event loop amalgamations featured in the libraries and runtimes for Go/Rust/Nim/Pony/etc. At the very least, aiming towards modularizing the event loop is work that can be done with Zig today that would undoubtedly be reusable should the time come that we eventually come to a consensus on an all-in-one event loop design.

2. There should be a focus in the standard library on supporting the lowest-common-denominator subset of cross-platform features that can be driven asynchronously by an I/O poller. To paint a picture, there should be a single io module in the standard library that presents first-class async support for sockets, timers, pipes, events (i.e. eventfd in Linux), etc.

Given that async file I/O really is something that has only been recently been made available in Linux via. io_uring, the io module should only contain support for blocking file I/O. An experimental module could optionally be included inside the standard library which provides Linux support for async file I/O.

There should also be cross-platform support for async Ctrl+C handling in io, with a separate, optional experimental module for async signal handling support.

3. I feel that the cross-platform abstraction for I/O pollers in the Zig standard library should follow the proactor pattern, and that all async I/O modules have completion-based APIs (callback-based). This would allow for easy C interoperability (pass a callback), and also only requires minimal boilerplate for being driven by async/await in Zig by wrapping the callback into an async frame. A number of battle-tested cross-platform async I/O libraries like libuv, libevent, and asio follow the proactor pattern as well.

This comes from my experience that abstracting a proactor-based I/O poller (e.g. IOCP) to follow a reactor-based design requires quite a bit of hackery and workarounds vs. abstracting a reactor-based I/O poller to follow a proactor-based design.

Having such a design where either a callback may be provided, or an async frame may be created to wrap under a callback, allows for users to have flexibility in how they want to allocate the memory necessary to asynchronously drive an I/O primitive (i.e. on the heap, or on the stack).

4. All async I/O and synchronization primitives should support cancellation and composition by design. The memory and CPU overhead for not supporting cancellation or composition is negligible for most applications.

With a production-ready I/O poller and thread pool abstraction, either way, it is most likely the case that you wouldn't settle with general-purpose implementations of async I/O or synchronization primitives that are easy to use if you are in a situation where the smallest amount of overhead present in supporting cancellation and compositional operations matter to you.

5. Much like the rest of the standard library, I feel that allocations should always be left up to the user. Therefore, where possible, memory should be intrusive.

6. The thread pool should support being configured with callbacks that are to be run in certain places (e.g. when a thread is spawned). A common use case for this is e.g. initializing thread-local state (tracy encourages users to have all user-spawned threads marked with names via. it's C API), and logging events.

7. I personally dislike global flags like the io_mode flag, and would rather have individual modules be configured to operate in a blocking/non-blocking manner. Hence, unlike status-quo Zig today, my suggestion is the instantiation of an I/O poller and thread pool and more should be made explicit in a user's program.

8. Apart from the overhead of supporting cancellation and async chaining operations (i.e. wait all ops to finish, race for either one of many ops to finish, etc.), all modules should be optimized for performance. For something as low-level as asynchronous I/O and thread pooling, performance and a good user-experience in coordinating async operations should be the only two things people really care about when it comes to a standard library.

---

Regarding a few of the points above, I didn't dig that deep into the specifics as I'd love to see other people's thoughts and feedback first given my one-sided point of view (i.e. how module-level configuration should look like, how async operations are to be facilitated by the standard library to be composed together, how I/O poller-related state should be separated and contained, etc.).

That being said, let me know if you spotted any glaring holes in my thoughts, or if you'd like me to explain more of my rationale on any one of the points above.

[kprotty]

Any further opinions on this?

[jorangreef]

My question then is: what do people really want out of the standard libraries event loop?

Thanks @kprotty for setting this up.

Multi-threaded by default?

No, and yes, depending on the plane: control plane or data plane. Having a clear distinction between these helps to inform the design decisions for each separately because the safety and performance profiles are different.

For example, given Zig's approach to safety and performance, I would really want a single-threaded control plane, with this single-threaded control plane outsourcing to the kernel thread pool for I/O through io_uring (or kqueue or IOCP for other platforms) to reduce the performance cost of syscalls, context switches and locks, and to side-step or at least diminish the gap for data races for the most part. Backwards compatibility for older Linux platforms can be shimmed with an I/O threadpool, but io_uring should drive the design because it reflects where hardware performance is at today.

Influence scheduling (hard/soft priorities, QoS classes, fairness)?

QoS is important. In addition to the single-threaded control plane (with the kernel threadpool for I/O) I really want additional multi-threaded data plane(s) for CPU-intensive work [xor] long-running work, with clear QoS, i.e. one data plane for user-defined CPU-intensive tasks (e.g. bulk crypto like checksumming 256 MiB chunks, or erasure coding, or deduplicating them), and another separate data plane for long-running background tasks (stuff like DNS requests that are not CPU-intensive, that could take minutes and that could be context-switched without a performance hit).

So the data plane for CPU-intensive tasks should be sized according to the number of cores to reduce context switching. On the other hand, the data plane for long-running background tasks would be okay with context switching (being on a different order compared to the task time), and would be sized larger than the number of cores to support concurrency.

Support cancellation?

This could solve alot of pain for people. Cancellation can also be racy and difficult to get right. Then again, it's hard at every layer and solving this lower down can help higher up. My gut feel is it's hard to do well. On the other hand, there could be cancellation for everything, but with a completion event for when the cancellation succeeds, so that this could shim over underlying abstractions that don't support cancellation, while not holding back the abstractions that do.

Are allowed to dynamically allocate (i.e. heap alloc) internally?

No. Although there might be a good argument to made for something like registered buffers, with a user-provided allocator of course. The focus overall though should be on zero-copy into user provided buffers. For TigerBeetle, everything is static allocation (even disk) and zero-copy. We would never use something that allocated dynamically, unless this was at initialization, where we have some kind of control over the parameters, and again for something like registered buffers. Allocation for stuff like internal queues etc. is also okay but it needs to be subject to the user's restrictions and at initialization only.

How async should it be?

Zig's async is awesome of course. However, here's something surprising from our experience. We tried out Zig's async plus io_uring for TigerBeetle (https://github.com/coilhq/tigerbeetle/blob/main/src/io_async.zig) and then actually went back to explicit callbacks plus io_uring in the end (https://github.com/coilhq/tigerbeetle/blob/main/src/io.zig). The reason being that we were doing this for a distributed consensus protocol where we wanted to make sure our message handlers run to completion with the same system state, whereas coroutines increase dimensionality while a function is still running. We wanted clear jumping off points to I/O just because getting the consensus protocol right was hard enough. This is specific to our use-case for TigerBeetle only, it might not be relevant here, but wanted to share the anecdote if it helps.

Should we even have an event loop at all?

The money question. We could start with the primitives. For example, @ifreund and I worked hard to get an event loop primitive going for io_uring that could mimic epoll's "poll for some amount of time": https://github.com/coilhq/tigerbeetle/blob/main/src/io.zig#L52-L83

io_uring

io_uring from the 5.6 kernel on up has almost everything we need for the data plane intensive aspects of storage and network I/O, and even the 5.8 kernel is gaining support. io_uring is a beast and it's only accelerating from here. It's something that's hard to ignore.

While we could plan according to current support for the 5.8 Linux kernel right now, perhaps a better approach would be to plot a course taking Zig's 1.0.0 trajectory into account.

With that in mind, I really want to see the IO component of the event loop taking full advantage of io_uring as a first-class citizen, because that's where the puck is already at, just not evenly distributed.

NVME devices have become so fast that the cost of a context switch is on the same order as an I/O operation, so io_uring makes sense to support the next generation of lockless thread-per-core designs, which is what we're doing for TigerBeetle to support a million financial transactions a second.

It's also a perfect time for Zig to take advantage of this shift in API design. I would really want to start with io_uring and work backwards rather than the other way round. A worker threadpool or traditional approaches to event loops should be last-resort, not something driving our design.

The bottom line: Zig should be "High-Definition" or "Dolby Atmos" where the machinery supports it.

[kprotty]

Thanks @kprotty for setting this up.

Thanks @jorangreef for the detailed response! This is an area where I strongly believe zig should make some good decisions after having interacted with other async/IO environements so any input/ideas are welcome.

control plane or data plane

Could you elaborate on the differences between the two?

In regards to performance, particularly throughput, it sounds difficult to reach its optimum if there's only one thread which handles control flow (if this is what you're implying). Having multiple threads performing application logic and without locking sounds ideal for throughput in the absence of effective static work distribution.

I do agree that io_uring should somewhat drive the design for the I/O subsection of the Event Loop. I say "somewhat" as SQEs are allocated internally on flush (at least currently) and it may not be what is desired for io_uring compatibility layers.

My gut feel is it's hard to do well

This has been my experience as well

Allocation for stuff like internal queues

This was my primary intent when posing the question. While I have working thread pools which only utilize intrusive memory, I have been interested if using dynamically growing but contiguous task queues would help in latency. This sort of memory allocation sounds contrary to your idea though.

We wanted clear jumping off points to I/O

By "clear", would this be a reference to a function call doing I/O without the callers awareness? or some other scenario?

an event loop primitive going for io_uring

This is pretty neat! Like the idea of intrusive SQE/CQE structs and the flushing/polling strategy. This would solve the "io_uring SQE allocation" issue at the abstraction level.

io_uring is a beast and it's only accelerating from here.

This was my initial experience too. Although having micro-benchmarked it against epoll, the decrease in syscalls and use of kernel thread-pool that sounds more efficient on the surface ended up being slower. Do you (or anyone else really) have any ideas why this might be the case?

I really want to see the IO component of the event loop taking full advantage of io_uring as a first-class citizen,

Definitely. Although it's a bit rough trying to envision how this would manifest itself higher-level API-wise. Have personally been looking into libdispatch for scheduling and queuing insight/inspirations recently.

[jorangreef]

Could you elaborate on the differences between the two?

Sure! In the abstract, the control plane is just anything that's not in the critical data path, it's safety critical but not performance critical, whereas the data plane has the inverse profile. For example, it would be fine to have (and one would want) plenty of assertions in the control plane for safety, whereas the data plane is in the critical path with huge volumes flowing through it. Here in the data plane one would want to optimize for performance: cache misses, context switches, branch mispredicts etc. The data plane would be like a water mains or oil pipeline, there's nothing inside to obstruct flow. The control plane would be all the safety checks you do outside the pipeline as an operator, the little control box that adjusts pressure and controls the pipeline. Having this clean split in the design provides both safety and performance without compromising either. It also obviates in large part the need for a borrow checker, although data races are still possible.

As a concrete example of this technique, in TigerBeetle, we have around 10,000 financial transactions in a batch. Our single-threaded control plane is responsible for switching each of these batches through the consensus protocol, and we amortize all runtime bounds checks, assertions, syscalls and I/O across the batch, so that these become almost free, yet we have literally hundreds of assertions, and we're doing O_DSYNC on every write etc.

In regards to performance, particularly throughput, it sounds difficult to reach its optimum if there's only one thread which handles control flow (if this is what you're implying). Having multiple threads performing application logic and without locking sounds ideal for throughput in the absence of effective static work distribution.

I think there's also a third option. For example, in TigerBeetle, if we want to do crypto for a batch (we have a use-case for this for Interledger), we would then drop that into a multi-threaded data plane with QoS for CPU-intensive work, by dividing the batch up across threads. We wouldn't want to solve that kind of performance problem with a multi-threaded control plane, because that would introduce too much complexity into a safety-critical domain, and also be conflating the control plane and data plane. In terms of this approach, the performance-critical work should happen in the multi-threaded data plane because that's been optimized for it. If we do find any scalability limit, then we would simply move to multiple event loop abstractions per domain, in our case across networking, storage, and state machine, all connected by ring buffers, and all the control planes still single-threaded.

This is also not new to TigerBeetle, we based this design on the work done by LMAX, a high-performance trading platform, which has single-threaded control planes for all event loops, and which has multiple event loops connected by ring buffers. They found the single-threaded control plane is also faster because it lets the CPU run unimpeded by context switches, like a sprinter doing the 100m in a straight line without zigzags. Martin Thompson calls this "mechanical sympathy" or "the single writer principle" (great blog post). He's done some awesome talks on this that we've documented here: https://github.com/coilhq/tigerbeetle/blob/main/docs/DESIGN.md#references

Redpanda is another new database, from people who worked on ScyllaDB, also doing thread-per-core.

Allocation for stuff like internal queues
This was my primary intent when posing the question

Just speaking from our point of view, it would be totally fine to allocate this at initialization, which coincidentally is also how io_uring does it for the most part. Contiguous task queues would definitely be great for latency to reduce cache misses and avoid pointer-chasing. We would really like fixed-size bounded queues with an error for overflow and back pressure (in our IO code we still need to set our upper bounds for SQ overflow, we're yet to profile some of our upper bounds). Explicit limiting is a good thing, especially for any kind of distributed system. This is probably also an area in event loops where developers have been used to "hidden allocation" and unbounded IO depth but I think we could be more explicit about this.

By "clear", would this be a reference to a function call doing I/O without the callers awareness? or some other scenario?

Yes, this would be where the function call ends up doing I/O that suspends so that when the caller is resumed, the rest of the system has moved on to a new state. For example, in the Viewstamped Replication consensus protocol that we use, it's critical that distributed messages are only processed when a replica process' status is .normal and where the replica process' view number (or term or epoch as with other consensus protocols) is exactly the same. We wouldn't want state like this changing during the lifetime of a function. Again, these examples are specific to TigerBeetle and Zig's async/await still has plenty of good use cases. Our team just decided on the explicit pain of colored function call sites for safety. Here's an example of a footgun, just for us, where the use of async I/O made this kind of bug much more likely (we're still weaning this code off async and onto callbacks): https://github.com/coilhq/tigerbeetle/blob/main/src/vr/journal.zig#L522-L536

This is pretty neat! Like the idea of intrusive SQE/CQE structs and the flushing/polling strategy. This would solve the "io_uring SQE allocation" issue at the abstraction level.

Thanks! We learned alot from the event loop gists you did!

Although having micro-benchmarked it against epoll, the decrease in syscalls and use of kernel thread-pool that sounds more efficient on the surface ended up being slower. Do you (or anyone else really) have any ideas why this might be the case?

Yes, there was a known networking regression in 5.7.16 that has since been patched and @axboe has also been optimizing against epoll significantly since then so that io_uring should be ahead on most benchmarks. In terms of storage, io_uring performance on NVME can be more than double compared to blocking syscalls and more than quadruple compared to an I/O threadpool like libuv. Overall, what excites me is that io_uring offers a single unified interface for networking and storage and potentially more and more syscalls down the line. There are some rough edges, like what we had to do to implement IO.run_for_ns() but there is already better support for that in io_uring_enter() in newer kernels. It's also early days and there are incremental improvements all the time.

Have personally been looking into libdispatch for scheduling and queuing insight/inspirations recently.

Thanks for the link! I'm definitely looking up to you when it comes to anything like this, and look forward to seeing your work on this.