feat(spanner): implement custom tracer_provider injection for opentelemetry traces

[odeke-em]

An important feature for observability is to allow the injection of a custom tracer_provider instead of always using the global tracer_provider.

[harshachinta]

The current implementation creates a tight coupling between observability_options and session, even though they are unrelated. While this approach achieves the goal and simplifies the code (since it assumes a session is available in every method), and assigning observability_options to the session can make retrieval easier, it may lead to confusion for future developers.

I propose following the approach used in other languages. We should set observability_options at the transaction level whenever a transaction is created in database.py. Specifically, observability_options would be accessible in database.py, where customers call methods like snapshot(), batch(), and run_in_transaction(). These methods have base classes, such as _SnapshotBase, _BatchBase, etc. The observability_options would be set at this class level, and methods within these classes would access the options directly from the class whenever trace_call is invoked.

[odeke-em]

Awesome and thank you very much @harshachinta for the insightful review! Roger that, I have made the updates to just infer observability_options from Instance itself. Please take another look. Thank you!

[odeke-em]

@harshachinta so a compromise/meet-in-the-middle would being able to just infer it directly from the Client instead of plumbing all within Transaction creator. I mention this because having made the proposed change within database for where transactions are createdd, it is firstly much more code but also requires much more plumbing inside for example database.run_in_transaction for which then has to also modify session.run_in_transaction instead of simply just inside trace_call retrieving it from session._database.observability_options.

[odeke-em]

Thank you for the review feedback @harshachinta! Kindly help me refresh the bots and take a look again, we should be go to go.