HDDS-6176. Ozone service WebUI is not accessible with 404 error.

[ArafatKhan2198]

What changes were proposed in this pull request?

A lot of customers were facing issues Ozone service-related WebUI not opening with a 404 error. The issue can be fixed by restarting, but during the restart, a jetty-0_0_0_0-*/webapp folder is created, and the customer noticed that such a folder does not exist on the host with access issues. The customer performed manual restarts on each node and observed that the problem was resolved by creating a new jetty-0_0_0_0-*/webapp folder.

After investigating we can confirm that the ozone code is not responsible for the deletion of the contents in the tmp directory. It appears that an external program is responsible for the deletion.

The main culprit for deletion is the Linux program called systemd-tmpfiles-clean.timer . This program is designed to clean up temporary files that are generated by the system. By default, it runs daily and removes files and directories that are older than 3 days.

To avoid this issue, it is recommended to configure Jetty to use a dedicated directory for its temporary files and directories, rather than the system's temporary directory. We can do this by changing the ozone.http.basedir property in the Ozone Service Advanced Configuration Snippet for the ozone-conf/ozone-site.xml file. It is suggested that this property should be set to a directory such as /data/ozone that already exists and has sufficient permissions in HDFS.

In case if this property is not set, then fall back to the default property.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-6176

How was this patch tested?

Added Unit-Tests for directory creation and also checked it manually in a local docker cluster

[ArafatKhan2198]

@sadanand48@ChenSammi@jojochuang can you please take a look!

[kerneltime]

One downside of this approach is that the directory is only cleaned up on clean exits (and if persistTempDirectory is false).
Another alternative might be to set a base dir for jetty, which is cleaned up on start and populated again.

[jojochuang]

Just a very minor comment otherwise it looks all good.

[jojochuang]

Review comments are addressed and test checks all came back green. I'll merging this PR now.