[CON-284] Distributed Tracing

[jonaylor89]

Description

This PR adds OpenTelemetry sdk and W3C spec to the content node codebase, which provides a rich API for adding tracing into the application. This includes global requestIDs (traceID), parentIDs which tell you where a request came from, and spanIDs which refers to the current request on a specific node. OpenTelemetry docs go into much more detail.

Along with requests, the queues we use will need extra parameters to carry span context

Along with Content Node, I also added basic instrumentation to Identity and Discovery so we can see when these services communicate with each other.

This PR also opens the doors for further exploration into trace visualization tools like Jaegar, which accepts the OpenTelemetry format, to visualize and debug traces and service dependencies.

Tests

Tested locally using A seed create-user and A seed upload-track to kick off some flows. The trace data was collected using Jaeger and by manually looking at the logs.

Below is a screenshot of Jaeger as the trace visualization tool that I used to make sure everything was stitched together correctly

Monitoring - How will this change be monitored? Are there sufficient logs / alerts?

With just this PR, these changes will only be monitored using Loggly to look at traceid and spanid in our Bunyan logs. Later down the line we can invest in using a tool like Jaeger to visualize our traces.

[dmanjunath]

okay did a first pass through this. generally i feel pretty good about this since it's add only and no functionality changed. i left a few small comments and the only other thing i was going to ask is why use anonymous and arrow functions? maybe there's no particular reason.

but for now i feel pretty good about this change and if all tests and mad dog pass we can merge and take it to stage. i am a bit worried about regressions given the size of the change so i want to make sure we test it very thorougly

[SidSethi]

whats diff btwn calledWith and calledWithExactly? i checked briefly and didn't find anything

[jonaylor89]

Closing this PR in favor of turning this into several small PRs