Arm request timeouts on an event loop

[pavel-ptashyts]

Problem

Request and read timeouts are armed on the client's HashedWheelTimer. That has two
properties that only show up on short deadlines:

• A wheel quantizes. It fires on the first tick at or after the deadline, so a
  deadline near or below hashedWheelTimerTickDuration is rounded up to it.
• One thread carries every expiry for the whole client, and HashedWheelTimer's
  default taskExecutor is ImmediateExecutor, so each expiry runs inline on the wheel
  thread - including future.completeExceptionally(...) and therefore whatever the caller
  chained onto the response future.

On a one-second budget the first costs 0.3% and nobody notices. On a budget of tens of
milliseconds a tick is a large fraction of it, and a burst of expiries has no headroom to
absorb before the wheel starts running late.

Measured

2000 timeouts armed as one burst on Netty 4.2.16, JDK 17, tasks doing nothing but
recording their own lag. This is the floor; real work on the firing thread only adds to it.

instrument	deadline	mean	p50	p99	max
wheel, tick 5 ms	20 ms	+2.7	+2	+5	+5
wheel, tick 1 ms	20 ms	+1.3	+1	+2	+2
EventLoop.schedule	20 ms	+0.0	+0	+0	+0
wheel, tick 5 ms	1000 ms	+3.0	+3	+3	+3
wheel, tick 1 ms	1000 ms	+1.9	+2	+2	+2
EventLoop.schedule	1000 ms	+1.6	+2	+2	+2

An event loop shows zero overshoot because it schedules by deadline and derives its own
select() timeout from the nearest one. There is no quantum to round to.

This was a throwaway probe rather than JMH - client/src/jmh/java is not currently wired
into the build, so its benchmarks do not compile. Happy to add a proper benchmark if that
is fixed first, or as part of this.

Change

AsyncHttpClientConfig#isUseEventLoopTimeouts(), off by default, arms the request and
read timeouts on an event loop instead of the timer.

• Always the channel's own loop. On the pooled path the channel is already in hand, so
  its loop is used and the timeout expires on the thread that would have to close it. On the
  connect path there is no channel yet - deliberately, so that the timeout also bounds
  address resolution and the connect - so it is armed on the timer and moved onto the loop
  once the connect succeeds. No other loop is ever used; see the review round below for why
  that matters.
• Arming allocates nothing extra. The cancellation handle lives on the task rather than
  in a wrapper, and the existing done flag stands in for the scheduler's already-expired
  flag, which the two schedulers spell differently.
• Shutdown race closed.isShuttingDown() can return false and schedule reject
  immediately after. Netty answers a rejected timeout with a logged warning rather than an
  exception, which would leave the exchange with nothing to end it, so a rejection falls
  back to the timer.
• The connection-pool cleaner stays on the timer either way.

Off by default because the expiry - and so whatever the caller chained onto the future -
then runs on an I/O thread, and blocking one stalls every connection it serves. The javadoc
says so and points callers at handleAsync.

Why not a wheel per event loop

That is how the Aerospike client solves the same problem: EventLoopBase owns a
HashedWheelTimer that is a Runnable the loop ticks itself. Deliberately not copied here.
A wheel arms in O(1) against O(log n) for a deadline queue, but at a few thousand timeouts
per loop that is a dozen comparisons, while the quantization it reintroduces costs
milliseconds on a 20 ms budget - the third row above is the whole point. A wheel also has to
be ticked forever, waking every loop even with nothing armed. Aerospike wrote its own because
its EventLoop abstracts over NIO, Netty and direct NIO and needed one timer; AHC is
Netty-only and gets a per-loop deadline queue for free.

Review round 1

Most of the substance of this PR changed in review, so the sections above describe the
current shape rather than what was first pushed. Two things are worth calling out here
because they were design errors, not polish:

The loop is now only ever the channel's own. It used to come from
EventLoopGroup#next(), which is almost never the loop the channel ends up on:
initAndRegister draws from the same chooser, so the two agreed about one time in N. Every
completion then cancelled an entry on a foreign loop, and until the original deadline that
entry sat in a queue whose loop it would wake for a request that had long finished. Drawing
from the chooser also shifted which loops connections land on. The pooled path has the
channel in hand; the connect path arms on the timer, as before this branch, and
NettyConnectListener moves the timeouts onto the loop once the connect succeeds, next to
the attachChannel that publishes it on the future. That listener already runs on the
channel's loop, so the move costs a same-thread schedule and no wakeup.

Arming left the TimeoutsHolder constructor. The task holds the holder and can run the
moment it is armed, and an event loop does not round a short deadline up to a tick, so the
expiry could reach a holder whose fields were not yet frozen, a future that had not been
handed the holder, and on the pooled path a future with no channel attached - which aborted
with null and left the pooled socket open. The caller now publishes the holder, attaches
the channel, and calls start() last.

Also from the review: arm re-checks cancelled after recording its handle, so an exchange
that finishes mid-arming cannot leave behind an entry nobody will cancel; cancelArmed
catches the RejectedExecutionException that Netty's off-loop cancellation path can raise on
a closing client, which had never escaped ListenableFuture#cancel before; the cancellation
handle is two typed fields rather than an Object and instanceof; requestTimeoutArmed is
gone in favour of a null test on the task; the rationale lives on
isUseEventLoopTimeouts() alone; and the new option sits in the // timeouts group
everywhere rather than splitting the two failedIpCooldown entries.

API compatibility

One revapi difference, with an ignore entry scoped to the single method:
java.method.exception.checkedRemoved on TimeoutTimerTask::run(Timeout).

TimerTask#run declares throws Exception and Runnable#run does not, so the Runnable
entry point either catches an exception no subclass throws - a handler that can only log,
and that defends against a subclass which cannot exist, the only constructor being package
private - or the declaration is narrowed. Review asked for the narrowing and it is the better
code, so the difference is accepted rather than worked around. It is binary compatible; the
only source-level effect would be on code catching a narrower checked exception around a
direct run(Timeout) call, which nothing outside the library does.

For completeness, the alternative shapes were tried. Letting each subclass own run() and
leaving the inherited signature alone trades this for java.method.abstractMethodAdded,
which is the worse difference of the two: it would break a subclass rather than an exotic
catch. Only keeping the dead handler avoids both. Everything else here is additive - the
existing TimeoutsHolder constructor is kept and delegates, and nothing is removed.

Tests

Four cases in EventLoopTimeoutTest, asserting where an expiry is delivered from rather than
what it does. They hand the config their own Timer and EventLoopGroup so the assertions
are against those objects and not against thread names, which a pool name containing timer
or a configured thread factory would have broken with no bug present:

• the timer default, by identity against the timer's own thread;
• a connecting exchange, on the loop of the channel onTcpConnectSuccess reported;
• an exchange on a pooled channel, on the loop of the channel onConnectionPooled reported,
  after waiting for onConnectionOffer so it cannot quietly open a connection of its own;
• a read timeout under the new mode, which is armed after the request is written and so
  exercises a different arming path.

The connecting case gets a one second budget on purpose: a deadline it could reach before
connecting would be delivered from the timer quite correctly, there being no channel to
deliver it from, and would prove nothing either way.

One gap, called out rather than papered over: the RejectedExecutionException fallback in
arm has no test. Reaching it needs a loop that answers isShuttingDown() with false and
then rejects the schedule, and the executor comes from the channel, so there is no way in
through the config. A test double for EventExecutor would do it if that is acceptable.

Verification

mvnw clean verify - BUILD SUCCESS, 1468 tests, 0 failures, 0 errors, 21 skipped. Error
Prone, NullAway clean; revapi clean with the one scoped entry above.

Caveat on the testing gate: AGENTS.md requires the build to run on JDK 11 and no JDK 11 is
installed on this machine, so it was run on JDK 17 (also in the CI matrix). The JDK 11
leg of CI on this PR is the real gate.

Claude Code on behalf of @pavel-ptashyts

🤖 Generated with Claude Code

[hyperxpro]

Round 1

[hyperxpro]

arm() catches RejectedExecutionException, this doesn't. Netty's ScheduledFutureTask.cancel goes through removeScheduled, which for an off-loop caller lazyExecutes a removal task, and offerTask rejects once the loop is shut down. So after a client.close() any late future.cancel(true) or abort() throws out of ListenableFuture.cancel(), which never threw before. The same try/catch arm() already has would settle it.

[pavel-ptashyts]

Right, and the path checks out: ScheduledFutureTask.cancel goes through AbstractScheduledEventExecutor.removeScheduled, which for an off-loop caller has to enqueue the removal. cancelArmed now catches RejectedExecutionException and reports it as not-cancelled, so cancel() and abort() keep the contract they had before this branch. Debug rather than warn: the entry dies with the loop anyway, and a caller cancelling a request on a closing client has no use for the news.

[hyperxpro]

This arms a task holding this from inside the constructor, so before requestTimeoutTask and requestTimeoutArmed are frozen and before the future has been handed the holder. At the pooled call site it is also before attachChannel. The wheel's 100 ms tick hid that window, an event loop won't: with a small enough request timeout the task can run first, find channel null in expire() and leave the pooled socket open, and setTimeoutsHolder then installs a holder that has already fired. Arming after the constructor returns closes it.

[pavel-ptashyts]

Agreed, and this is the one that actually bites. Arming has moved out of the constructor into TimeoutsHolder#start(), which scheduleRequestTimeout calls only after setTimeoutsHolder, so the task can no longer reach a half-built holder or a future that has not been handed one.

The pooled path also had attachChannel after the arming, and those two are now swapped: an expiry that lands immediately finds a channel on the future and closes it, instead of aborting with null and leaving the socket open.

[hyperxpro]

This is almost never the loop the channel ends up on. initAndRegister calls next() again, so for an N loop group the affinity the javadoc above claims holds about one time in N. Every completion then cancels a scheduled entry on a foreign loop, and that entry sits in its queue waking it at the original request timeout deadline long after the request finished. Read timeout re-arms cross loops too. NettyConnectListener.onSuccess already has both the channel and the holder, it sets the resolved address there, so re-homing at that point would make the claim true on the connect path as well.

[pavel-ptashyts]

You are right, and the javadoc was claiming something the code did not do. timeoutExecutor no longer calls next(): it returns the channel's loop when there is a channel and null otherwise, which leaves the connect path on the timer exactly as it was. NettyConnectListener.onSuccess then moves it across, right after attachChannel as you suggested, through a new TimeoutsHolder#rehomeOn. That listener already runs on the channel's own loop, so the move is a same-thread schedule, and the read timeout armed later re-arms on that loop too. The holder keeps the switch rather than every caller consulting the config, which also keeps the listener from needing the request sender it has no other use for at that point.

The invariant is now simply: the loop that owns the exchange's channel, or the timer for as long as there is no channel.

[hyperxpro]

Separately: calling next() only to pick a timeout thread advances the group's round robin counter, and that is the same counter that assigns channels to loops. With a power of two group and a fixed number of next() calls per request, registrations can settle onto a subset of the loops. Whether that helps or hurts depends on whether an address resolver group is configured, so it is action at a distance either way. Picking with ThreadLocalRandom over the group's executors, or re-homing once the channel exists, leaves the chooser alone.

[pavel-ptashyts]

The same change covers this one: nothing draws from the group's chooser any more, so the counter is left to initAndRegister alone. That seemed better than picking with ThreadLocalRandom, which would still have been the wrong loop, only less predictably so.

[hyperxpro]

The handle is recorded after schedule() returns. If the exchange completes in that gap, cancel() wins the CAS, release() cancels the previous handle, and then armedOn writes the live one with nobody left to cancel it. cancelled is one shot, so nobody ever will. No wrong abort, since the task no-ops on done, but the entry stays in that loop's queue until the full deadline. Re-checking cancelled after arming and cancelling there closes it.

[pavel-ptashyts]

Agreed. arm re-reads cancelled after recording the handle and releases the task there, so either the arming thread sees the flag or cancel() sees the handle.

[hyperxpro]

This rationale is now spelled out four times: here, the TimeoutsHolder class doc, timeoutExecutor's Javadoc, and the class doc on TimeoutTimerTask. Keeping this as the canonical explanation and linking to it from the other three says it once and leaves a single place to update.

[pavel-ptashyts]

Agreed. isUseEventLoopTimeouts() is the only place that carries the reasoning; the other three say what they do and link here.

I did add one paragraph to it, about the timer carrying the timeout until a channel exists and the exchange moving it afterwards. After the change discussed on the next() thread that is part of the contract rather than an implementation detail, so it belongs in the canonical place.

[hyperxpro]

This block is alphabetical, so it belongs just before defaultUseInsecureTrustManager, not between the two failedIpCooldown entries. The same split repeats in the field list, the constructor parameters and assignments, the Builder field and copy constructor, the build() call, AsyncHttpClientConfigDefaults, and ahc-default.properties, and the getter lands between isFailedIpCooldownEnabled and getFailedIpCooldownPeriod. The constructor is positional with a lot of adjacent booleans, so order is worth more here than tidiness.

[pavel-ptashyts]

Agreed, and splitting that pair was careless. The whole set has moved into the // timeouts group next to requestTimeout and readTimeout, which is where it belongs on its own merits: field, constructor parameter and assignment, getter, builder field, copy constructor, setter, the build() argument, AsyncHttpClientConfigDefaults and ahc-default.properties. The interface method moved next to getRequestTimeout for the same reason, and the import is alphabetical, before defaultUseInsecureTrustManager.

[hyperxpro]

The subclass this defends against cannot exist. The only constructor is package private, so nothing outside this package can extend TimeoutTimerTask, and both subclasses are in it and neither throws. Which also means run(Timeout) could just drop its throws clause and run() could call it straight, no catch needed.

[pavel-ptashyts]

You are right, that constructor makes the subclass impossible. run(Timeout) is redeclared without throws, run() calls it directly, and the catch is gone.

One thing worth flagging: revapi objects to it, java.method.exception.checkedRemoved, so this needed an ignore entry scoped to that one method. I tried it without: keeping the inherited signature and letting each subclass own run() instead trades it for abstractMethodAdded, which is the worse of the two, since that one would break a subclass rather than a caller catching a narrower checked exception around a direct call. Only leaving the dead catch in place avoids both. Say the word if you would rather have the catch than the entry and I will put it back.

[hyperxpro]

Both tests use a fresh client's first request, so timeoutExecutor always takes the channel null branch. The channel branch, the whole reason scheduleRequestTimeout grew a third parameter and the only place affinity actually exists, is never exercised: drop the channel argument at the pooled call site and both tests stay green. The RejectedExecutionException fallback in arm() has no test either, and neither does a read timeout under the new mode, which is where isClaimed replaced isExpired.

[pavel-ptashyts]

Agreed, and the tests were covering only the branch that has since been removed. There are four now: the timer default, a connecting exchange, a pooled one, and a read timeout under the new mode.

The pooled case takes the channel from onConnectionPooled and waits for onConnectionOffer before firing the second request, so it cannot quietly open a connection of its own and test nothing. The connecting case takes the channel from onTcpConnectSuccess and gets a one second budget: a deadline it could reach before connecting would be delivered from the timer quite correctly, there being no channel to deliver it from, and would prove nothing either way.

The RejectedExecutionException fallback in arm still has no test, and I would rather say so than leave you to find out. To reach it I need a loop that answers isShuttingDown() with false and then rejects the schedule, and the executor comes from the channel, so there is no way in through the config. If a test double for EventExecutor is acceptable to you I will add one.

[hyperxpro]

Both assertions rest on this substring. The timer factory is the pool name plus a timer suffix and the I/O factory is the pool name alone, so a pool name containing timer makes the first test pass for the wrong reason and the second fail, and setting a thread factory on the config drops the suffix entirely and breaks the default case with no bug present. Asserting on the executor rather than the thread name would be steadier. Minor: each test leaves the echo handler sleeping 5s while the server closes, so the class pays that stop timeout twice.

[pavel-ptashyts]

Agreed on both. The tests now build their own Timer and EventLoopGroup and hand them to the config, so the default case asserts assertSame against the timer's own thread and the event-loop cases assert channel.eventLoop().inEventLoop(...). No name matching left, and setting a thread factory can no longer break them.

The 5 s delay is gone as well: the endpoint blocks on a latch that @AfterEach releases before closing the server, so teardown pays nothing for it.

[pavel-ptashyts]

Round 1 is addressed; thanks, the two structural ones changed the shape of this for the better.

What moved:

• Timeouts are only ever armed on the loop that owns the exchange's channel. next() is gone, so the group's chooser is left to initAndRegister, and the connect path arms on the timer and is moved onto the loop by NettyConnectListener once the connect succeeds.
• Arming left the TimeoutsHolder constructor for a start() the caller invokes after publishing the holder, and the pooled path attaches the channel before arming rather than after.
• arm re-checks cancelled after recording the handle; cancelArmed no longer throws out of cancel() during shutdown, and holds two typed handles instead of an Object.
• requestTimeoutArmed and the throws on run(Timeout) are gone, the rationale lives in isUseEventLoopTimeouts() alone, and the new option sits in the // timeouts group everywhere instead of splitting the failedIpCooldown pair.
• The tests assert against the timer and the loops themselves rather than thread names, and cover the pooled branch and a read timeout. The one gap left is the RejectedExecutionException fallback in arm, noted inline.

Dropping the throws on run(Timeout) needed a revapi entry, java.method.exception.checkedRemoved, scoped to that method; the justification is in the pom and the reasoning is on the inline thread. It is the only way to keep the dead catch out, and the alternative shapes trade it for a worse difference.

./mvnw clean verify is green. Not on JDK 11 as the contract asks, only 17 - no 11 on this machine.