Span Limit Configuration

[dbarker]

[Blocked - pending benchmarking of span limits]

SpanLimits are currently only supported by the Otlp HTTP and gRPC exporters. This issue is to support span-limit configuration via the TracerProvider to match the specification.

Currently the TracerProviderConfiguration contains the limits but the interfaces to set limits in the spans/recordables is not implemented.

The SDK builder will need to apply those limits through a common interface.

See spec-complianc-matrix

[github-actions]

This issue is available for anyone to work on. Make sure to reference this issue in your pull request.
✨ Thank you for your contribution! ✨

[lalitb]

Thanks for opening this.

This area already had a detailed discussion in #3303, which was closed by #3816.

In otel-cpp, SDK Span does not centrally store attributes, events, and links. It mostly forwards them to Recordable, and the actual storage is exporter-specific. Because of that, adding generic SDK-level span limits would add counters/configuration/checks around the SDK hot path, or require a Recordable wrapper.

I am still concerned about that cost. Even if limits are disabled by default, we would still add more code paths, configuration surface, tests, and maintenance. If implemented in Span, it can also increase per-span memory or add CPU checks on calls like SetAttribute, AddEvent, and AddLink. These are hot-path APIs.

The previous conclusion was to enforce limits where the data is actually stored. #3816 added collection limits to the OTLP exporter/recordable, with dropped counts tracked there. That keeps the core SDK span path unchanged and avoids overhead for users/exporters that do not need this. For end-user/custom exporters, if they store span attributes/events/links and need protection from unbounded growth, they can add similar limits in their own Recordable implementation, similar to the OTLP exporter. That is the right layer because this is where the memory allocation/storage happens.

Looking at the spec, I do not think generic SDK-level event/link limits are a hard MUST. The spec says:

• span attributes must follow common attribute limit rules
• SDK spans may discard links/events beyond configured limits
• if an SDK implements those limits, they must be configurable
• dropped counts must be available to exporters when data is dropped

Also, the spec compliance matrix marks SpanLimits as optional. So I do not think we should add SDK-level support only to satisfy the matrix if it adds CPU or memory overhead to the common path.

So I think we should clarify what is still missing here:

• If this is about OTLP exporter collection limits, that was already handled by [EXPORTER] Add support for otlp exporter collection limits #3816.
• If this is about generic SDK-level SpanLimits, I would prefer not to add it unless we have a design that does not add hot path CPU or per-span memory overhead.
• If the compliance matrix is the reason for this issue, we should first confirm whether exporter-level support is enough, or whether the matrix expects SDK-level support specifically.

My preference is to keep the SDK Span path minimal and put limits in the exporter/recordable layer unless there is a strong spec reason that requires otherwise.

[dbarker]

Thanks for the clarification @lalitb. I looked at the sdk/span.cc and just missed the implementation (and the linked issue/pr) for otlp recordable.

The otlp grpc/http exporters independently implement the span limits in their options classes and will apply them on span calls to SetAttribute, AddEvent, and AddLink, in the hot path. This seems to be what is desired to keep any one span from growing beyond the limits and using too much memory. Nothing major seems missing here (just optional attribute value limits and log messages to let users know data is being dropped).

There are some tradeoffs/gaps here for configuration. Currently the spec places the span limits with the Tracer Provider to apply to all spans. Configuration from environment variables and yaml are done at that level independent of processor/exporter. The TracerProviderConfiguration in otel-cpp has the limits but not the path to set them on exporters. The SDK builder does not link directly to the otlp component so a general interface is needed. Supporting the environment variables for span limits on the otlp component would be straight forward though.

Given that the limits are currently in the otlp span hot path, I think its worth a discussion about moving them to the sdk Span and propagating the configured limits from the tracer provider so they apply to any processor/exporter. We are missing Span (backed by OtlpRecordable) benchmarks and that would help ground it. That could be a starting point. What do you think?

For the spec compliance matrix, maybe we hold off on checking the box for span limits until configuration is resolved?

Some more references:

• https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#span-limits
• https://github.com/open-telemetry/opentelemetry-configuration/blob/7ea099ae1dd9c4cce271444c46404c8ac9d0ed51/schema/tracer_provider.yaml#L10

[lalitb]

Given that the limits are currently in the otlp span hot path, I think its worth a discussion about moving them to the sdk Span and propagating the configured limits from the tracer provider so they apply to any processor/exporter. We are missing Span (backed by OtlpRecordable) benchmarks and that would help ground it. That could be a starting point. What do you think?

Yes, I agree the concrete gap here is configuration flow from TracerProviderConfiguration to the OTLP exporter options.

The part I am more cautious about is moving enforcement into SDK Span. The current C++ design intentionally does not keep full span storage in SDK Span; it forwards to Recordable, where the exporter owns storage. That helps keep the SDK hot path smaller.

So I think we can separate the two issues:

1. Propagating configured span limits to OTLP exporters sounds reasonable and worth fixing.
2. Moving generic enforcement into SDK Span needs more discussion and benchmark data, because that changes the current performance/layering tradeoff and adds checks/counters to SetAttribute, AddEvent, and AddLink for all exporters, for all exporters, including lock-free/no-batching exporter paths like Windows ETW and Linux user-events exporter.

Holding the compliance matrix entry until the configuration path is fixed makes sense to me.

[dbarker]

The part I am more cautious about is moving enforcement into SDK Span. The current C++ design intentionally does not keep full span storage in SDK Span; it forwards to Recordable, where the exporter owns storage. That helps keep the SDK hot path smaller.

I've been digging into this a bit recently. One key finding from #4035 is that there is a hard to avoid deep copy from each OtlpRecordable protobuf Span to the otlp exporter request messages due the how the memory management works with protobuf Arenas. OtlpRecordable creates a protobuf Span on the heap (no Arena). The resulting proto Span message serialized in the hot path cannot be moved to the Arena allocated request message. This deep copy along with hot path serialization on the heap are areas to look at. Both expensive.

Comparing hot path (recording) performance of OtlpRecordable to the common SpanData recordable will also help with discussion.

So I think we can separate the two issues:

1. Propagating configured span limits to OTLP exporters sounds reasonable and worth fixing.
2. Moving generic enforcement into SDK Span needs more discussion and benchmark data, because that changes the current performance/layering tradeoff and adds checks/counters to SetAttribute, AddEvent, and AddLink for all exporters, for all exporters, including lock-free/no-batching exporter paths like Windows ETW and Linux user-events exporter.

Sounds good. We should hold off on span limit configuration interfaces to exporters until otlp recordable / sdk span / span data benchmarks are in and discussed. I'm working on those benchmarks.