Add pooled ByteBuffer allocator with size classes

[arturobernalg]

This is the first step towards a pluggable pooled ByteBuffer allocator.
The patch adds PooledByteBufferAllocator (power-of-two size buckets, global pool + per-thread caches) and switches the HTTP/2 FrameFactory and the benchmark to use ByteBufferAllocator. Behaviour is unchanged except for using pooled buffers for small control frames.
If this direction looks reasonable, I’ll follow up by threading the allocator through IOSession/SSLIOSession and the async codecs and add minimal metrics; otherwise I’ll keep it local to HTTP/2.

ByteBuffer allocator throughput (JMH)

Benchmark	bufferSize	iterations	Mode	Cnt	Score (ops/ms)	Error (ops/ms)
pooled_allocator_shared	1024	100	thrpt	10	1644.982	14.006
pooled_allocator_shared	8192	100	thrpt	10	533.638	34.307
pooled_allocator_shared	65536	100	thrpt	10	59.422	0.937
pooled_allocator_thread_local	1024	100	thrpt	10	539.612	5.518
pooled_allocator_thread_local	8192	100	thrpt	10	201.345	4.451
pooled_allocator_thread_local	65536	100	thrpt	10	19.603	0.501
simple_allocator_shared	1024	100	thrpt	10	172.750	4.893
simple_allocator_shared	8192	100	thrpt	10	23.083	0.199
simple_allocator_shared	65536	100	thrpt	10	2.883	0.037
simple_allocator_thread_local	1024	100	thrpt	10	129.873	1.075
simple_allocator_thread_local	8192	100	thrpt	10	21.075	0.088
simple_allocator_thread_local	65536	100	thrpt	10	2.401	0.062

@ok2c WDYT?

[ok2c]

@arturobernalg I may be wrong but I was under impression that many (if not all) Java frameworks arrived at the same conclusion that memory pooling was counterproductive as of Java 8 given the efficiency of modern garbage collection algorithms. I will run the micro-benchmark locally and look at the results, but it may take me a certain while.

Generally I see no problem with providing pluggable allocators as long as the simple one remains default and you are willing to maintain more complex ones.

@rschmitt do you happen to have an opinion on this matter?

[rschmitt]

@ok2c I'm going to ask one or two more qualified people for an opinion and get back to you. My understanding is that object pooling can outperform garbage collection, but it's harder to do than you'd think. (There's also the question of what "outperform" means. What are we measuring, tail latencies? CPU overhead? Heap footprint?) Pooled buffers also come with a lot of risks, like increased potential for memory leaks, or security vulnerabilities such as buffer over-reads.

The Javadoc says that the PooledByteBufferAllocator is inspired by Netty's pooled buffer allocator, but which one? In Netty 4.2, they changed the default allocator from the pooled allocator to the AdaptiveByteBufAllocator. What does that mean, exactly? ¯\_(ツ)_/¯ Evidently it may have something to do with virtual threads.

I guess the main concern I have here is the effectiveness of adding buffer pooling retroactively, compared with the cost in code churn. Typically what I see is frameworks or applications that are designed from the ground up to be garbage-free or zero-copy or what have you. I think this proposal would be more persuasive if I knew what we were measuring and what our performance target is, and what the hotspots currently are for ephemeral garbage. Can they be addressed with a minimum of API churn? (I find it's very difficult to thread new parameters deep into HttpComponents; if we implemented pooling, I'd prefer to make it a purely internal optimization, and an implementation detail. We should be more hesitant to increase our API surface area.)

Finally, I think it's a little late in the development cycle for httpcore 5.4 to be considering such a change. Any usage of pooling in the HTTP/2 or TLS or IOReactor implementation should probably be gated behind a system property and considered experimental.

[arturobernalg]

@olegk@rschmitt I’ve added a small JMH benchmark that exercises the old and new pool under mixed routes with slow discard / expiry.
On my machine the segmented pool removes the cross-route stall and slightly improves tail latency while keeping throughput comparable.
Happy to adjust the scenario or parameters if you’d like to capture other access patterns.

To clarify the Netty reference: the allocator is conceptually closest to Netty 4.1s

Allocator	Kind	Buffer	Throughput (ops/ms)	Error
pooled_allocator_shared	HEAP	1024	517.697	±7.829
pooled_allocator_shared	DIRECT	1024	527.269	±20.476
pooled_allocator_shared	HEAP	8192	194.948	±1.124
pooled_allocator_shared	DIRECT	8192	222.407	±2.573
pooled_allocator_shared	HEAP	65536	19.387	±0.297
pooled_allocator_shared	DIRECT	65536	18.704	±1.621
pooled_allocator_thread_local	HEAP	1024	519.383	±9.957
pooled_allocator_thread_local	DIRECT	1024	544.220	±11.254
pooled_allocator_thread_local	HEAP	8192	205.072	±2.435
pooled_allocator_thread_local	DIRECT	8192	222.178	±7.711
pooled_allocator_thread_local	HEAP	65536	18.960	±0.172
pooled_allocator_thread_local	DIRECT	65536	18.286	±1.217
simple_allocator_shared	HEAP	1024	150.141	±6.162
simple_allocator_shared	DIRECT	1024	8.553	±5.767
simple_allocator_shared	HEAP	8192	24.545	±0.880
simple_allocator_shared	DIRECT	8192	5.835	±2.174
simple_allocator_shared	HEAP	65536	2.767	±0.162
simple_allocator_shared	DIRECT	65536	2.351	±0.244
simple_allocator_thread_local	HEAP	1024	149.243	±5.933
simple_allocator_thread_local	DIRECT	1024	8.373	±5.096
simple_allocator_thread_local	HEAP	8192	25.226	±1.756
simple_allocator_thread_local	DIRECT	8192	5.665	±2.431
simple_allocator_thread_local	HEAP	65536	2.700	±0.248
simple_allocator_thread_local	DIRECT	65536	2.274	±0.182

Allocator	Kind	Buffer	gc.alloc.rate.norm (B/op)	gc.count	gc.time (ms)
pooled_allocator_shared	HEAP	1024	0.013	≈0	-
pooled_allocator_shared	DIRECT	1024	0.013	≈0	-
pooled_allocator_shared	HEAP	8192	0.035	≈0	-
pooled_allocator_shared	DIRECT	8192	0.031	≈0	-
pooled_allocator_shared	HEAP	65536	0.356	≈0	-
pooled_allocator_shared	DIRECT	65536	0.370	≈0	-
pooled_allocator_thread_local	HEAP	1024	0.013	≈0	-
pooled_allocator_thread_local	DIRECT	1024	0.013	≈0	-
pooled_allocator_thread_local	HEAP	8192	0.034	≈0	-
pooled_allocator_thread_local	DIRECT	8192	0.031	≈0	-
pooled_allocator_thread_local	HEAP	65536	0.364	≈0	-
pooled_allocator_thread_local	DIRECT	65536	0.378	≈0	-
simple_allocator_shared	HEAP	1024	104000.046	100.000	94.000
simple_allocator_shared	DIRECT	1024	13600.926	8.000	4147.000
simple_allocator_shared	HEAP	8192	820800.283	89.000	84.000
simple_allocator_shared	DIRECT	8192	13601.245	20.000	2020.000
simple_allocator_shared	HEAP	65536	6555202.508	81.000	77.000
simple_allocator_shared	DIRECT	65536	13602.957	29.000	252.000
simple_allocator_thread_local	HEAP	1024	104000.046	94.000	90.000
simple_allocator_thread_local	DIRECT	1024	13600.875	8.000	3920.000
simple_allocator_thread_local	HEAP	8192	820800.276	86.000	91.000
simple_allocator_thread_local	DIRECT	8192	13601.321	19.000	1827.000
simple_allocator_thread_local	HEAP	65536	6555202.575	86.000	81.000
simple_allocator_thread_local	DIRECT	65536	13603.057	27.000	272.000

[rschmitt]

I asked Aleksey Shipilëv for his thoughts:

Depends. In a pure allocation benchmark, allocation would likely be on par with reuse. But once you get far from that ideal, awkward things start to happen.

1. When there is any non-trivial live set in the heap, GC would have to at least visit it every so often; that "so often" is driven by GC frequency, which is driven by allocation rate. Pure allocation speed and pure reclamation cost becomes much less relevant in this scenario -- what else is happenning dominates hard. Generational GCs win you some, but they really only prolong the inevitable.
2. When objects are allocated, they are nominally zeroed. Under high allocation rate, that is easily the slowest part, think ~10 GB/sec per thread. Re-use often comes with avoiding these cleanups, often at the cost of weaker security posture (leaking data between reused buffers).
3. For smaller objects, the metadata management (headers, all that fluff) dominates the allocation path performance, and is often logically intermixed with the real work. E.g. you rarely allocate 10M objects just because, there is likely some compute in between. But allocating new byte[BUF_SIZE] (BUF_SIZE=1M defined in another file) is very easy. So hitting (1) and (2) is much easier the larger the object in questions get.
4. For smaller objects, the pooling overheads become on par with the size of the objects themselves. The calculation for total memory footprint can push the scale in either direction.
5. For some awkward classes like DirectByteBuffers that have separate cleanup schedule, unbounded allocation is a recipe for a meltdown.

So answer is somewhat along the lines of: Pooling common (small) objects? Nah, too much hassle for too little gain. Pooling large buffers? Yes, that is a common perf optimization. Pooling large buffers with special lifecycle? YES, do not even think about not doing the pooling. For everything in between the answer is somewhere in between.

Here, "special lifecycle" refers to things like finalizers, Cleaners, weak references, etc.; nothing that would apply to a simple byte buffer.

Another interesting point that came up is that if you use heap (non-direct) byte buffers, and if the pool doesn't hold on to byte buffer references while they are leased out, then there is no risk of a memory leak: returning the buffer to the pool is purely an optimization. Since HEAP and DIRECT have near-identical performance, maybe we should just hardcode a pooled heap buffer allocator into key hotspots.

[ok2c]

@rschmitt Thank you so much for such an informative summary. Please convey my gratitude to Aleksey.

One thing bugs me is how big is big? How big should be byte buffers to justify pooling? If it is a couple of MB, then memory pooling may be useful in our case.

Here, "special lifecycle" refers to things like finalizers, Cleaners, weak references, etc.; nothing that would apply to a simple byte buffer.

I think we have objects with "special lifecycle" in the HttpClient Caching module only but they are backed by files and not byte buffers. There is nothing else I can think of.

However, I imagine the classic on async facade may actually qualify as a potential beneficiary of the pooled memory allocator, so I am leaning towards approving this change-set and letting @arturobernalg proceed with further experiments.

What do you think?

[arturobernalg]

@olegk@rschmitt How should I proceed here?

[rschmitt]

@olegk@rschmitt How should I proceed here?

The advantage of pooling (or of more aggressive buffer reuse) is that we can reduce ephemeral garbage from the client. The risks are:

1. We could leak memory (can be mitigated by following Aleksey's suggestion, i.e. transfer ownership of a buffer to and from the pool instead of the pool holding a reference to leased-out buffers)
2. We could leak data by leasing out previously used buffers (can be mitigated by zeroing out buffers upon return to the pool)
3. Buffers could be aliased internally, or references to buffers could leak outside of the client, resulting in corruption or potentially a security vulnerability (e.g. disclosure of plaintext, disclosure of request or response data across HTTP exchanges)

The third risk has no comprehensive mitigation in Java, we simply have to be able to tell by inspecting the code that there are obviously no issues like this.

Before we proceed with any code changes, I think we should profile the client first to see what is actually allocating the most ephemeral garbage internally. If nobody has done this in a while, then there are probably some easy wins available. A simple example is EntityUtils, which has several fixed new byte[] and new char[] allocations that could trivially be replaced with thread-local buffers, no object pooling necessary; EntityUtils::toString shouldn't need to allocate any memory other than the String it returns.

[ok2c]

@arturobernalg I agree with @rschmitt here. There is no point adding this complex feature just in case. We need to have a strong case where it actually improves performance. I think it could really help with classic over async facade, but this needs to be carefully measured and benchmarked. I would propose you start by profiling the classic over async facade classes.

[arturobernalg]

Hi, I took a first pass at profiling the classic-over-async HTTP/1 facade using the ProfileClassicOverAsyncHttp1Main harness (IntelliJ allocation profiler). I may be interpreting this wrong (or the harness may still have artifacts), but the results suggest the dominant allocation hot spot is in the classic bridge buffering path.

In the baseline run, the largest allocation volume appears on the response path:

ClientHttp1StreamDuplexer.consumeData(ByteBuffer)
→ ClassicToAsyncResponseConsumer.consume(ByteBuffer)
→ SharedInputBuffer.fill(ByteBuffer)
→ ExpandableBuffer.ensureAdjustedCapacity(int)

In a run with my allocator/buffer changes enabled, allocations attributed to that path drop from ~6.33 GB to ~4.68 MB (diff view shows ~-99.9% on that stack). In addition, overall allocation volume in the run dropped from ~8.49 GB to 5.08 GB (-40%), and java.nio.ByteBuffer.allocate(int) volume dropped substantially (in an earlier run: 8.66 GB → 716 MB, ~-92%).

I’m aware these numbers are very sensitive to the profiling setup (threads selected, test wrappers, etc.), so I don’t want to over-claim anything. If you think this harness is not representative, or if there’s a preferred way you’d like this benchmarked), I’m happy to adjust and rerun with whatever methodology you consider reliable.

Thanks for the guidance.

[rschmitt]

Why the classic-over-async facade?

[ok2c]

Why the classic-over-async facade?

@rschmitt It was my suggestion. See my comment above. If you have a better idea where this feature can be applied please do suggest.

[ok2c]

@arturobernalg I am aware that performance of the classic over async facade is bad for larger messages. So, thank you for doing this work. I still would like to run the same test through my benchmark https://github.com/ok2c/httpclient-benchmark and compare the results. I need to update the benchmark first and do lots of other things beside, so bear with me.

[ok2c]

@arturobernalg This is what I think. Micro-benchmarks are nice and they can be useful, but ultimately what matters if a performance optimization feature makes a real and verifiable impact on request execution. This is the benchmark I have been using while working on HttpClient 5.x performance optimization. I can still be useful to collect total execution time / request per seconds metrics of different agents or different message types. It is not very scientific but if it is usually good enough for better / worse performance comparison.

If you can find a scenario where pooled ByteBuffer allocator makes a noticeable and verifiable impact, say, of 5% improvement, then it is useful. If not, I am sorry to say I do not think it should be added to the project code.

I think there is a chance of making classic over async with very large messages to perform better with this pooled ByteBuffer allocator but I would like to see hard evidence of that.

[1] https://github.com/ok2c/httpclient-benchmark

[arturobernalg]

Hi @ok2c ,

I've been benchmarking different ByteBufferAllocator strategies for the classic-over-async facade with 1MB responses at concurrency 50.

Key finding: A simple ThreadLocalByteBufferAllocator that caches the single largest released buffer per thread significantly outperforms both the baseline and the bucketed PooledByteBufferAllocator.

JMH microbenchmark results (HEAP, ops/ms, higher is better):

┌─────────────┬───────────────────┬────────┬─────────────┐
│ Buffer Size │ Simple (baseline) │ Pooled │ ThreadLocal │
├─────────────┼───────────────────┼────────┼─────────────┤
│ 1KB │ 174 │ 260 │ 367 │
├─────────────┼───────────────────┼────────┼─────────────┤
│ 8KB │ 22 │ 88 │ 90 │
├─────────────┼───────────────────┼────────┼─────────────┤
│ 64KB │ 2.6 │ 11.5 │ 11.6 │
└─────────────┴───────────────────┴────────┴─────────────┘

Both Pooled and ThreadLocal show zero GC allocations on the hot path (gc.count ≈ 0).

End-to-end benchmark (50K GET requests, 1MB response, c=50, 4 rounds with alternating order):

┌─────────────┬─────────────┬─────────────┐
│ Agent │ Avg req/sec │ vs Baseline │
├─────────────┼─────────────┼─────────────┤
│ Baseline │ 1422 │ — │
├─────────────┼─────────────┼─────────────┤
│ Pooled │ 1442 │ +1.4% │
├─────────────┼─────────────┼─────────────┤
│ ThreadLocal │ 1567 │ +10.2% │
└─────────────┴─────────────┴─────────────┘

Why ThreadLocal wins: The IO reactor uses a small fixed thread pool. After the first request each thread's buffer stabilizes at the final size. Subsequent requests get a direct cache hit — zero expandCapacity()
chains, zero copies, zero GC. The PooledByteBufferAllocator achieves the same reuse but pays for bucket lookups and CAS operations that the ThreadLocal approach avoids entirely.

The implementation is ~30 lines: one ThreadLocal, keep-largest-on-release policy, no locks, no contention.

I'm going to prepare the PR with the ThreadLocalByteBufferAllocator and the full benchmark results.

[ok2c]

@arturobernalgThreadLocals usually make me uncomfortable, ThreadLocals that can hold MBs of data make me very, very uncomfortable. It is nice it shows improvement in the benchmark results but it needs to be absolutely leak-proof for real life scenarios.

[rschmitt]

ThreadLocals that can hold MBs of data make me very, very uncomfortable

Two things worth keeping in mind here:

1. Virtual threads make it economical to have on the order of 10,000 threads.
2. Java is currently in the hot seat due to the DRAM shortage. A basic aspect of the JVM's design is to use the largest heap possible, because that results in more efficient garbage collection, but just over the last few months this is starting to become expensive. Reducing ephemeral garbage is still a good idea as far as it goes, but now would be a particularly bad time to increase memory overhead, i.e. the baseline amount of tenured heap memory we need to run.

[arturobernalg]

Hi @ok2c ,

I've spent considerable time benchmarking the classic-over-async facade trying to reach the 5% improvement threshold. Here's a summary of what I found.

Benchmark setup: 50,000 requests, concurrency 200, 1MB response bodies, HTTP/1.1, 6 rounds per configuration, A/B comparison in the same session where possible.

What I tried:

1. PooledByteBufferAllocator wired through ClassicToAsyncAdaptor → SharedInputBuffer: Result within noise (±2%). during buffer expansion (2KB→4KB→...→2MB), only intermediate
   buffers are recycled. The final ~2MB buffer cannot be safely returned to the pool because releaseResources() races with async framework callbacks (use-after-free). So every request still allocates a fresh final
   buffer.
2. Content-Length pre-sizing (read Content-Length header and pre-allocate SharedInputBuffer to the right size): Actually 16% slower. With 200 concurrent connections, pre-allocating 200×1MB buffers up front
   overwhelms the system and disrupts the flow control dynamics (capacity channel reports 1MB available instead of 2KB).
3. SharedInputBuffer micro-optimizations (signalAll() → signal(), AtomicInteger → plain int for capacity increment since it's always accessed under the lock): Correct improvements semantically, but not
   measurable — within noise.

Conclusion: At concurrency 200 with 1MB responses, the system is transferring ~2.5GB/sec of content. The bottleneck is memory bandwidth and network I/O, not buffer allocation or lock contention in
SharedInputBuffer. The buffer management overhead is a rounding error compared to the cost of moving data through the network stack.

The PooledByteBufferAllocator itself performs well in isolation (JMH: 471 ops/ms vs 194 for SimpleByteBufferAllocator at 1KB HEAP), but the classic-over-async facade's one-way buffer growth pattern prevents the pool from being effective. The allocator would likely show better results in code paths where buffers are allocated and released at the same size repeatedly, like async framework internals or H2 frame
handling.

I'll continue testing other code paths this week to find where the allocator can demonstrate measurable improvement.