Hyperopt->Raytune for Hyperparameter Optimization and Different Objective Function

[stompsjo]

A refactor of hyperparameter optimization codes for ML models to use ray[tune] instead of hyperopt. See the raytune docs for more information. The main feature additions, and the motivation for the chance, are better compatibility for parallel computation and other options for hyperparameter optimization algorithms. For example, ray[tune] has a wrapper function for the same hyperopt algorithm used before, but also has other algorithms like Bayesian Optimization HyperBand which could be used in the future. Raytune also allows a user to specify the number of trials to run simultaneously, which greatly increases the speed or capabilities for testing different hyperparameters on powerful computers.

This should be reviewed after the original ML/SSML models have been reviewed and merged, since this is a feature change from the original exploratory work and is a refactor of existing code. Note: hyperparameter optimization now maximizes a score rather than minimizing a loss as before (even though both are reported to raytune. The new objective function/score is:

score = balanced_accuracy + alpha*recall + beta*precision

This allows a user to reward increases in precision or recall asymmetrically. That is, if a user cares more about reducing false positives over false negatives, or vice versa, they can change alpha/beta in optimization. This may take longer to optimization, since the state space is increasing, but provides more flexibility in model evaluation than purely based on balanced accuracy.

To address:

• Complete code coverage with unit tests.
• Consider how to make unit tests faster with lightweight models.
• Consider making it possible to use more than just HyperOptSearch for optimization.
• Consider the ability to use a Scheduler or checkpointing during training loop(s) with tune.session.report().

[coveralls]

Pull Request Test Coverage Report for Build 3388298322

• 523 of 567(92.24%) changed or added relevant lines in 6 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-5.6%) to 94.416%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
models/SSML/ShadowNN.py	75	89	84.27%
models/SSML/CoTraining.py	81	96	84.38%
models/SSML/ShadowCNN.py	132	147	89.8%

Totals
Change from base Build 3348444582:	-5.6%
Covered Lines:	744
Relevant Lines:	788

---

💛 - Coveralls

[stompsjo]

I will push this branch stompsjo/RadClass:raytune to cnerg:RadClass:raytune for posterity, but I reverted to using hyperopt when I did my dissertation research, meaning this PR is moot. In the time since I created this PR, raytune stopped playing nicely with the rest of my code, meaning I never managed to get parallel hyperparameter optimization trials working. Instead, I used hyperopt and initiated several jobs on UW-Madison's CHTC.