Using Sentry breadcrumbs and tags alongside OpenTelemetry isn't working correctly

[seeARMS]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using? If you use the CDN bundles, please specify the exact bundle (e.g. bundle.tracing.min.js) in your SDK setup.

@sentry/node

SDK Version

7.43.0

Framework Version

No response

Link to Sentry event

https://paragraph.sentry.io/performance/node:08b586a3e77243548a96ac02db296d0f/?project=6606495&query=&statsPeriod=5m&transaction=POST+%2Fmetrics%2Fpost-track

SDK Setup

Sentry.init({
 dsn: "MY_DSN",

 enabled: isProduction,

 instrumenter: "otel",

 release: process.env.COMMIT,
 environment: process.env.FLY_APP_NAME || "development",


 tracesSampler: (samplingContext) => {
 // Drop transactions for routes we don't care about
 for (const route of SENTRY_IGNORED_ROUTES) {
 if (samplingContext?.request?.url?.endsWith(route)) return 0
 }

 // Drop transactions for non-production environments. (We still care about errors
 // on non-production environments, but we don't need to see all the transactions.)
 if (process.env.FLY_APP_NAME !== "paragraph-master") return 0

 return TRACE_SAMPLE_PERCENTAGE
 },

 profilesSampleRate: 1.0, // Profiling sample rate is relative to tracesSampleRate

 beforeSend: (event) => {
 if (!event.request?.data) return event

 const size = JSON.stringify(event.request?.data || "").length

 // Arbitrarily cut off massive payloads. Sentry silently drops events that are too large,
 // so trim them.
 // See https://github.com/getsentry/sentry-javascript/issues/3036#issuecomment-1066491190
 if (size > 500000) {
 event.request = event.request?.data?.slice(0, 500000)
 }

 return event
 },

 integrations: [new ProfilingIntegration()],
})

// Only load OpenTelemetry in production,
// so we aren't spamming Sentry with test data.
if (isProduction) {
 const sdk = new opentelemetry.NodeSDK({
 /*
 sampler: new opentelemetry.tracing.TraceIdRatioBasedSampler(
 TRACE_SAMPLE_PERCENTAGE
 ),
 */
 // Existing config
 traceExporter: new OTLPTraceExporter(),
 instrumentations: [
 new HttpInstrumentation({
 ignoreIncomingRequestHook(req) {
 for (const route of SENTRY_IGNORED_ROUTES) {
 if (req.url.endsWith(route)) {
 return true
 }
 }
 return false
 },
 }),

 new ExpressInstrumentation(),
 new GrpcInstrumentation({
 ignoreGrpcMethods: [
 "google.logging.v2.LoggingServiceV2/WriteLogEntries",
 new RegExp(".*google.logging.*"),
 new RegExp(".*WriteLogEntries*"),
 (name) => {
 return name.startsWith("google.logging")
 },
 ],
 }),
 ],

 // Sentry config
 spanProcessor: new SentrySpanProcessor(),
 })

 otelApi.propagation.setGlobalPropagator(new SentryPropagator())

 sdk.start()
}

Steps to Reproduce

I've been happily using Sentry for some time. using @sentry/node with Express.

Today we switched from using Sentry tracing to OpenTelemetry-based tracing, using @sentry/opentelemetry-node. Transactions are getting logged fine, but it appears that breadcrumbs and tags are not correctly added to the transaction.

Viewing any Sentry event (eg the one linked above) has tags from what seems like other requests. Additionally, the breadcrumbs also don't correspond to this request.

There's a chance I'm doing something incorrect in Sentry.init() above. For example, I don't need to use any Sentry tracing functionality now that I have OpenTelemtry tracing working, right? The documentation here is a bit light so it's unclear to me.

Additionally, I'm getting an exception on startup (see logs below).

Expected Result

I'd expect to be able to use Sentry.addBreadcrumb and Sentry.addTag throughout the codebase, and it would add the breadcrumb and tag to the current transaction/span created by Opentelemetry.

Actual Result

I get this non-fatal exception on startup:

2023-03-20T19:21:52Z app[53d5f709] iad [info]Error: @opentelemetry/api: Attempted duplicate registration of API: propagation
2023-03-20T19:21:52Z app[53d5f709] iad [info] at registerGlobal (/workspace/node_modules/@opentelemetry/api/build/src/internal/global-utils.js:32:21)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at PropagationAPI.setGlobalPropagator (/workspace/node_modules/@opentelemetry/api/build/src/api/propagation.js:52:50)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at NodeTracerProvider.register (/workspace/node_modules/@opentelemetry/sdk-trace-base/build/src/BasicTracerProvider.js:100:31)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at NodeTracerProvider.register (/workspace/node_modules/@opentelemetry/sdk-trace-node/build/src/NodeTracerProvider.js:43:15)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at NodeSDK.start (/workspace/node_modules/@opentelemetry/sdk-node/build/src/sdk.js:158:24)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at Object.<anonymous> (/workspace/packages/api/dist/trace.cjs:127:7)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at Module._compile (node:internal/modules/cjs/loader:1159:14)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at Module.load (node:internal/modules/cjs/loader:1037:32)
2023-03-20T19:21:52Z app[53d5f709] iad [info] at Module._load (node:internal/modules/cjs/loader:878:12)

If I replace the NodeSDK init with the following, the error goes away, but I don't know if it's actually using SentryPropagator now:

 const sdk = new opentelemetry.NodeSDK({
 textMapPropagator: new SentryPropagator(),
... 
})

[AbhiPrasad]

Hey @seeARMS, thanks for writing in!

To isolate your breadcrumbs/tags, you need to use our express request handler!

As per the docs,

// The request handler must be the first middleware on the app
app.use(Sentry.Handlers.requestHandler());

This should make sure sentry data is isolated.

I get this non-fatal exception on startup:

Yes, we need to adjust our docs for this, using dependency injected propagator is more correct than the global propagation API. Will fix!

[AbhiPrasad]

Created getsentry/sentry-docs#6501 and #7548 to update the docs!

[seeARMS]

Thanks for the help!

I gave that a try and now no breadcrumbs or tags are getting logged.

Transaction after the change, with no tags/breadcrumbs: https://paragraph.sentry.io/performance/node:7cb35584c76c4a598319153e8ad6b3aa/?project=6606495&query=&referrer=performance-transaction-summary&statsPeriod=2m&transaction=GET+%2Fblogs%2FxeXAY7C8qmb1h6Cp8zTa%2Fcommunities&unselectedSeries=p100%28%29

Same transaction but different event which occurred yesterday, with the erroneous tags/breadcrumbs as described above: https://paragraph.sentry.io/performance/node:a87a3b8398ee42e39c33dc5b9b1955f5/?project=6606495&query=&referrer=performance-transaction-summary&showTransactions=recent&statsPeriod=24h&transaction=GET+%2Fblogs%2FxeXAY7C8qmb1h6Cp8zTa%2Fcommunities&unselectedSeries=p100%28%29

Do I need to add the tracing integration (new Tracing.Integrations.Express({ app }) to Sentry.init as well as the tracing handler middleware (app.use(Sentry.Handlers.tracingHandler()))? Or is this not needed since I'm now using OpenTelemetry?

[AbhiPrasad]

You shouldn't need to use the Express integration nor the tracingHandler, since we are relying completely on OpenTelemetry to generate the spans.

The Sentry.Handlers.requestHandler is needed, because it's isolating the context between your requests. I don't understand why this is not working though 🤔. Is the requestHandler the first express middleware getting registered?

Need to reproduce this, will try tomorrow! If you could provide a quick repro, would be a huge help as well!

Also 👋 to fellow uwaterloo grad ^^

[seeARMS]

Here's a repro: https://github.com/seeARMS/sentry-javascript-7538-reproduction

Clone it, run npm install, then npm run start. It successfully instantiates OpenTelemetry and Sentry, and the transaction gets created successfully when the / route is hit, but the single breadcrumb never gets displayed in Sentry.

Here's an event in question that was logged using the repro (no breadcrumb present): https://paragraph.sentry.io/performance/node:f116399485744a8ab2452906b6d39fee/?environment=sentry-reproduction&project=6606495&query=&referrer=performance-transaction-summary&statsPeriod=2m&transaction=GET+%2F&unselectedSeries=p100%28%29

And hello! Always great to see other uWaterloo grads in the wild 😄

[AbhiPrasad]

Hey @seeARMS thank you so much for the reproduction! Took me a while to debug, but the issue is due to inconsistency between how we isolate requests and how opentelemetry isolates requests. I opened a PR to fix this: #7577

Thank you for your patience while we figured it out - this will be part of the next release of the SDK (going out end of week).