PAL does not work well with multiple log managers

[bliptec]

The PAL maintains a lot of global state and assumes there is a single log manager. On older versions of OneDS, creating and then destroying multiple log managers would cause a double-delete when trying to free the worker thread. This is no longer the case since the worker thread refactor, but if one log manager is shut down before the other - it appears it could cause significant issues with the other log manager.

This area should have tests.

[maxgolov]

Jason Bray (@bliptec) - have you tried the most recent version of SDK from the master, just to confirm - is this still an issue, or there's no issue - and the ask here is to specifically add more functional tests? If the ask is to add more tests, then I'd suggest we remove the 'bug' label and add 'build infra', and prioritize accordingly to late December or early [Mn], as this won't be product-impacting.

[maxgolov]

Just to clarify: there was a ref-count on PAL, but it wasn't effective because LogManagerImpl was in some cases calling into PAL :: shutdown twice on one instance :) ... in essence, breaking it for the second instance. I added a stress-test to verify the fix.

[maxgolov]

I resolved the issue with ref count double-decrement at LogManager-level. Also added the two log managers stress-test / functional test. The issue resolved on latest. Please reopen if you experience it again on latest.