OtlpRecordable: about 5 heap allocations per attribute, then a deep copy at export. Arena backed recordable or adopt into the request?

[ygkat]

Summary

On main the OTLP trace recordable costs about 5 heap allocations per span attribute on the recording thread, roughly 2.5x what the generic SpanData recordable costs for the same attribute, and the span it builds is then deep copied into the Arena allocated request at export time because the recordable's Span message is not Arena allocated. We measured this with the committed otlp_recordable_benchmark and span_data_benchmark on main, and in a large HAProxy based edge deployment where OTLP protobuf construction on the request thread is the largest single block of the tracing overhead. This is a design question before we invest in a PR, since the two plausible fixes have very different API and ABI consequences.

Measured in a large HAProxy based edge deployment

HAProxy with an OpenTelemetry tracing filter, OTLP over HTTP with content_type: binary and a BatchSpanProcessor, 7 spans and 19 attributes per request, 4 request threads pinned to 4 CPUs. Allocations were attributed with an LD_PRELOAD shim that records a backtrace per allocation and aggregates by thread and stack, run at 1k rps with no sampling, alongside separate perf record and futex tracepoint passes at 10k rps.

Tracing costs +90.3 us per request (35.9 us to 126.2 us), of which the libc allocator is +27.7 us. Allocations go from 2.44 to 307.8 per request, which gives a measured 0.091 us per allocation lifecycle (the malloc plus the matching free, wherever it happens). Attributed to OTLP protobuf construction on the request thread:

group	allocs/req	per span or attribute	~us/req
SetAttribute into PopulateAttribute (DefaultConstruct<KeyValue>, <AnyValue>, ArenaStringPtr::Set for key and value, InternalExtend)	87.15	4.59 per attribute	7.93
SetIdentity (trace_id, span_id, parent_span_id, trace_state)	35.70	5.1 per span	3.25
SetName	10.50	1.5 per span	0.96
the recordable object itself (MakeRecordable)	7.35	1.05 per span	0.67
total on the request thread	140.7		12.8

That is 46 percent of all allocations in the request path, on the latency critical thread. The batch thread spends another 9.4 us per request in the allocator, mostly freeing blocks the request threads allocated, and 73 percent of the futex calls we recorded are glibc arena mutexes taken by that cross thread free.

Measured with the committed benchmarks on main

Built from main at 58d4197, Release, GCC 11.5.0, protobuf 35.0.0 shared, abseil 20250512, C++17, run with taskset -c 0-3 and 5 repetitions on an Intel Xeon Gold 5120 at 2.20 GHz. The committed baselines in the file headers come from a 32 x 5700 MHz machine, so absolute times here are about 2.5 to 3x slower and the ratios are the point.

benchmark (mean)	committed baseline	this machine
RecordMinimalSpan	362 ns	1039 ns
RecordSpanWithAttributes/attribute_count:1	414 ns	1021 ns
RecordSpanWithAttributes/attribute_count:10	880 ns	2478 ns
BM_OtlpPopulateRequest/span_count:1	0.387 us	1.57 us
BM_OtlpPopulateRequest/span_count:512	23.6 us	102 us

same run against span_data_benchmark from #4203	OtlpRecordable	SpanData	ratio
RecordMinimalSpan	1039 ns	825 ns	1.26x
RecordNominalSpan	1726 ns	1448 ns	1.19x
RecordSpanWithAttributes/attribute_count:10	2478 ns	1944 ns	1.27x
marginal ns per attribute (1 to 10)	161.9 ns	105.4 ns	1.54x
RecordSpanWithAttributes/attribute_count:128	24306 ns	25689 ns	0.95x

The same 1.5x marginal ratio is present in the committed baselines (51.8 ns versus 35.0 ns). At 128 attributes the two cross over, because SpanData pays for its std::unordered_map at that size.

Allocation counts from the same two binaries, counted with a minimal LD_PRELOAD counter over malloc/calloc/realloc at a fixed iteration count (--benchmark_min_time=50000x --benchmark_repetitions=1) and divided by iterations. The loop body is start span, set attributes, End(), ForceFlush():

case	OtlpRecordable	SpanData
RecordMinimalSpan	9.05 allocs/iter	4.01
attribute_count:1	14.05	7.01
attribute_count:10	62.05	25.01
attribute_count:128	656.05	265.01
marginal allocations per attribute (1 to 10)	5.33	2.00
marginal allocations per attribute (10 to 128)	5.03	2.03

The count is flat in the number of attributes and matches the per field breakdown, one repeated field element for the KeyValue, one for the AnyValue, and one std::string object each for key and value plus the value's character buffer, and it reproduces the 4.59 per attribute measured above. For BM_OtlpPopulateRequest the marginal cost of one more span is 1.02 allocations and about 0.20 us, low on allocations because the destination is an Arena, and a lower bound on the real copy because MakeSpanBatch builds spans with no attributes, events or links.

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DWITH_OTLP_HTTP=ON -DWITH_OTLP_GRPC=OFF -DBUILD_TESTING=ON -DWITH_BENCHMARK=ON
cmake --build build --target otlp_recordable_benchmark span_data_benchmark
taskset -c 0-3 ./build/exporters/otlp/otlp_recordable_benchmark --benchmark_repetitions=5 --benchmark_report_aggregates_only=true
taskset -c 0-3 ./build/sdk/test/trace/span_data_benchmark --benchmark_repetitions=5 --benchmark_report_aggregates_only=true

Where it comes from

• exporters/otlp/include/opentelemetry/exporters/otlp/otlp_recordable.h:119 holds proto::trace::v1::Span span_ by value, so the message and every submessage and string under it are heap allocated with no Arena, and otlp_recordable.h:62-63 expose span() returning a reference to that member, which is part of the public surface of the exporter library.
• exporters/otlp/src/otlp_recordable.cc:148-151 SetAttribute calls span_.add_attributes() and then OtlpPopulateAttributeUtils::PopulateAttribute. :42-55 SetIdentity sets trace_id, span_id, parent_span_id and trace_state, four or five string allocations per span, and :218-221 SetName adds one more.
• exporters/otlp/src/otlp_populate_attribute_utils.cc:356 and :384 do attribute->set_key(...), and :104, :111, :224, :231 do proto_value_->set_string_value(...). On a non Arena message each of those ArenaStringPtr::Set calls allocates the std::string object, and the character buffer too when the value exceeds SSO.
• exporters/otlp/src/otlp_recordable_utils.cc:133-134 is the copy, carrying a comment added in [CODE HEALTH] fix nondeterministic pointer iteration order warnings #4035 that already states the cause: // The recordable span can only be copied here since the request message is Arena allocated. above scope_spans->add_spans()->CopyFrom(otlp_recordable->span());
• The request Arena is created per export in otlp_http_exporter.cc:326-338, otlp_grpc_exporter.cc:123-133 and otlp_file_exporter.cc:72-82, while MakeRecordable at otlp_http_exporter.cc:302-307 creates the recordable with std::make_unique and no Arena.

Prior discussion

• Optimize OTLP exporter #302 "Optimize OTLP exporter" (2020) reported the same copy and measured roughly 1000 ns versus 500 ns for dense spans using Swap, and was closed as inactive by the stale bot in 2022 with no change.
• Span Limit Configuration #4046 states the problem directly: "there is a hard to avoid deep copy from each OtlpRecordable protobuf Span to the otlp exporter request messages due the how the memory management works with protobuf Arenas. OtlpRecordable creates a protobuf Span on the heap (no Arena). The resulting proto Span message serialized in the hot path cannot be moved to the Arena allocated request message. This deep copy along with hot path serialization on the heap are areas to look at. Both expensive." The same comment asks for a comparison of OtlpRecordable recording performance against the common SpanData recordable, which is what the second section provides.
• [BENCHMARK] add span otlp recordable benchmark #4165 added otlp_recordable_benchmark and [BENCHMARK] add SpanData recordable benchmark and unify common utils #4203 added span_data_benchmark with unified test utilities, which is what made this measurable without patching anything.
• [ADMIN] 2025 WISH LIST #3256 "[ADMIN] 2025 WISH LIST" carries the entry "Add more control over dynamic memory allocation and move heap allocation to initialization of the sdk components/extensions where possible. The otlp exporters may be a good place to focus."

Proposal

Option A. OtlpRecordable owns a google::protobuf::Arena and creates its Span on it. Every set_key, set_string_value, add_attributes and mutable_value becomes an Arena bump instead of a malloc, which replaces the 5 allocations per attribute and the 5 per span from SetIdentity with one or two block allocations per recordable, and removes the cross thread free of hundreds of small blocks. Open questions:

• Handover to the request. Protobuf will not move a message between two Arenas without copying, so either the recordable's Arena becomes the request's Arena, or the exporter takes ownership of the recordable's Arena and keeps it alive until the request completes, or the request is built without an Arena. Is there a preferred shape, and should the per export Arena in the three exporters stay?
• ABI. span() can keep its signature if the member becomes a pointer, but the size and layout of OtlpRecordable change. Is that acceptable in a minor release?
• Arena reuse. Recordables are pooled and reused by the batch processor, so a block can be retained across spans, and we would benchmark Reset() against reconstruction.

Option B. Keep the heap recordable and remove only the second copy. OtlpRecordableUtils::PopulateRequest adopts the recordable's Span through a release plus AddAllocated style transfer instead of CopyFrom, which requires the request not to be Arena allocated or both sides to share one Arena. It is a much smaller change and is essentially what #302 asked for in 2020. It does nothing for the roughly 5 allocations per attribute on the recording thread, which in our deployment is the larger half of the cost.

Our measurements point at Option A, because the recording thread is where the latency is and where 46 percent of the request path allocations are. But Option A touches public API and ABI, and the interaction with the per export Arena is a decision for maintainers, not for us. Which direction would you prefer? If there is agreement on one, we are happy to prepare the PR with otlp_recordable_benchmark and span_data_benchmark numbers and allocation counts before and after on the same machine, and to add an allocation counting benchmark fixture if that would be useful to the project independently of this change.

[dbarker]

Thanks for the report and analysis. Your analysis is correct that the recording path for the OTLP Recordables has higher latency than the recording path for the common c++ types (SpanData and ReadWriteLogRecord).

The spec has some requirements that the OTLP Recordables will struggle to meet without additional performance cost and complexity:

1. Enforcing unique attribute keys (ideally while also keeping within attribute limits while recording)

   • https://opentelemetry.io/docs/specs/otel/common/#attribute-collections

     Implementation MUST by default enforce that the exported attribute collections contain only unique keys.

   • See [OTLP EXPORTER] fix OTLP recordable attribute spec compliance #4166 for some prototypes and costs added with OtlpRecordable

2. Readable interfaces

   • https://opentelemetry.io/docs/specs/otel/trace/sdk/#additional-span-interfaces

     Readable span: A function receiving this as argument MUST be able to access all information that was added to the span, as listed in the API spec for Span.

There is a significant architecture challenge to achieve spec compliance and optimal performance when recording directly into protobuf types given that the system must support a simple processor, a batch processor, and configurations with multiple processors. Recording into a single arena implies some complexity around managing that arena's lifetime and scope. The thread that created the span manages the span lifetime, while the thread that creates/exports the otlp request message manages its lifetime. Recording directly to the heap without an arena likely won't offer better performance in recording or export. Neither address the spec compliance concerns.

I propose we move towards an optimized common spec-compliant recordable type that can be used by all built-in exporters.

If you consider the case of instantiating multiple processors (see #4454 - also relevant to advanced span processor use cases), then there is a strong argument to unify the recordable type so the recording and export memory allocations are not duplicated for each processor. A common recordable type would also eliminate the dynamic allocations of the MultiRecordable itself and from the MultiSpanProcessor.

What should we do next?

I think your Option A (but limited to just adding an arena to OtlpRecordable and OtlpLogRecordable) will be a quick performance improvement to the recording path that we can implement now. But we also need a better long term solution to recording for OTLP export that can be spec compliant with better performance than today.

Standardizing on SpanData is an option and will give better recording performance than OtlpRecordable while meeting the above spec requirements, but I think it is worth considering alternatives that could be allocated on initialization.

[dbarker]

Related: #4043

For single processor use cases the dynamic memory allocations of the MultiProcessor should be avoided.

[lalitb]

@ThomsonTan and I have been looking into a similar issue reported by one of our internal customers, so the allocation overhead and latency on the application thread are particularly relevant to us.

I agree with starting with a per-recordable arena while retaining CopyFrom into the existing request arena. That lets us measure the recording improvement while keeping request ownership straightforward.

One correction to the proposal: BatchSpanProcessor does not currently pool recordables. It creates fresh ones and destroys them after export, so reuse would be separate work.

Could the PR measure allocations and latency on the application thread for StartSpan, SetAttribute, and End, before and after the change? Please also include export CPU and peak memory to show the overall tradeoff. Minimal spans, realistic populated spans, long strings, and a full batch queue would help cover the different cases. Arena-backed strings can still allocate their character buffers on the heap.

Also, Option B does not necessarily require removing the request arena: protobuf's AddAllocated() can adopt a separately heap-allocated message without copying. The embedded span_ cannot be transferred directly, but swapping its contents into a standalone heap message before adoption is another option we could benchmark separately.