Level preventing server from shutting down

[wellnesscookie]

Expected behavior

We expect server not to freeze when disabling.

Observed/Actual behavior

We came to a conclusion that the server freezes while stopping, when Level addon is being disabled.

Steps/models to reproduce

Initiate a stop or a server restart via a command.
Sadly, I do not have any more steps in order to reproduce this.

BentoBox version

[13:59:01 INFO]: Running PAPER 1.16.5.
[13:59:01 INFO]: BentoBox version: 1.16.2
[13:59:01 INFO]: Database: MYSQL
[13:59:01 INFO]: Loaded Game Worlds:
[13:59:01 INFO]: skyblock (skyblock): Overworld, [nether_color]Nether, [end_color]End
[13:59:01 INFO]: Loaded Addons:
[13:59:01 INFO]: BentoBox-InvSwitcher 0.0.4 (ENABLED)
[13:59:01 INFO]: Biomes 1.14.0-SNAPSHOT-b198 (ENABLED)
[13:59:01 INFO]: Border 3.1.1 (ENABLED)
[13:59:01 INFO]: BSkyBlock 1.14.4 (ENABLED)
[13:59:01 INFO]: Challenges 0.8.4-SNAPSHOT-b438 (ENABLED)
[13:59:01 INFO]: Chat 1.1.3 (ENABLED)
[13:59:01 INFO]: ControlPanel 1.13.0-SNAPSHOT-b45 (ENABLED)
[13:59:01 INFO]: DimensionalTrees 1.6.1-SNAPSHOT-b82 (ENABLED)
[13:59:01 INFO]: IslandUpgrade 1.0 (ENABLED)
[13:59:01 INFO]: Level 2.6.3 (ENABLED)
[13:59:01 INFO]: Limits 1.15.5 (ENABLED)
[13:59:01 INFO]: SerbCraftAddon 2.5 (ENABLED)
[13:59:01 INFO]: Warps 1.11.0-SNAPSHOT-b283 (ENABLED)

Plugin list

If relevant, will provide additionally.

Other

We are using a bit older versions because as I read, the newest ones support only 1.17+.
We are settled to stay on 1.16.5 for 3-4 more months.

[tastybento]

Does this always happen? Or just sometimes?

[tastybento]

Try this build. It just adds some debug to the onDisable(). I want to try and find out where exactly it is hanging.

Level-2.6.3-debug-SNAPSHOT-LOCAL.jar.zip

[wellnesscookie]

Yes, it does happen everytime. We will put the snapshot of Level addon you gave us to load on our next restart in 5 hours.
We will let you know as soon as possible.

[wellnesscookie]

It freezed again, and this is the log of it.

We have scheduled server restarts for everyday in 05:00h AM. Our script has a timeout of 5 minutes for servers to shutdown, otherwise it kills the server process. Sadly, since it's prevented to fully stop the server, it kills the process everytime.

[BONNe]

Please, could you try this version?
https://ci.codemc.io/job/BentoBoxWorld/job/Biomes/210/

[wellnesscookie]

Yes, of course. As previously said, our next server restart is scheduled for 05:00 AM the following day. We will send you feedback as soon as we wake up.

[tastybento]

I think @BONNe may be onto something. The Level addon seems to be shutting down okay and the queues are tiny, so I can't think why it would take any time at all. It may be Biomes.

[wellnesscookie]

I am sorry for a late reply, we've been busy these days.
Sadly, I must inform you that the problem wasn't solved even with that Biomes build from jenkins.

Please let us know if we can assist you in investigating this problem.
EDIT: Also have a look on the screenshot above, where I posted the debugging logs from Level addon. The queues indeed seem tiny, but the timestamps of those two are suspicious. They have a delay between of almost 2 minutes.

[tastybento]

Yes, in the debug it should have listed Finished saving and it didn't. It looks like the saving of the top ten is not finishing.