Quickstart

This file will enable you to set up your model for hyperparameter search using the cluster-hyperopt repository.

It is separated into 3 Sections:

• Section 1: Preconditions

• Section 2: Creating a wrapper function, which executes the training and evaluation of your model

• Section 3: Installation of cluster-hyperopt and creation of necessary files

A more detailed Tutorial can be found in the Tutorial.md file.

Section 1 Preconditions

You need:

• access to SigOpt
• access to BWUniCluster or HoreKA
• a GitHub repo for your project/model
• access to Repositories: make sure you have access to cluster-hyperopt and your models repository from the server. Create an ssh key pair if needed.
• a conda environment (we will call it "E1") for your model on the BWUniCluster/HoreKA
• a training and an evaluation function for your model

Section 2

In this section we make your model compatible with Cluster Hyperopt. Therefore, we create a wrapper function (lets call it "run_hyperopt(config)", but you can name it how you want), that trains and evaluates the model. This is the only function that Cluster Hyperopt will execute from your model repository.

• Input: python dict
• Output: The loss/score/metric of the evaluated model
• Structure:
  • extract hyperparams from the input dict
  • run the training algorithm of your model using the extracted hyperparameters
  • run the evaluation algorithm of your model
  • return the loss/score/metric of your model
• Location: In an arbitrary python file in your github repo. Make sure, that it is uploaded to github.

Example structure of "run_hyperopt(config)":

defrun_hyperopt(config):
# extract hyperparams. Example:max_depth=config["suggestion"]["max_depth"]
#run the training algorithm. Example:model=train(max_depth)
#evaluiate the model. Example:loss=evaluate(model)
#return loss/score/metricreturnloss

The python dict mentioned as input has the following structure: We added some example hyperparameters.

dict: {
'suggestion': {
"hyperparameter_x": value # Example Hyperparameter
"bootstrap": 1, # Example Hyperparameter
"criterion": "gini", # Example Hyperparameter
"max_depth": 5, # Example Hyperparameter
"max_features": "log2", # Example Hyperparameter
"n_estimators": 2 # Example Hyperparameter
},
'dataset': 'path/to/dataset/'
'output_path': path/to/output/dir
} 

Section 3

Installation of cluster-hyperopt

• download the repository cluster-hyperopt to a location of your choice on the BWUniCluster or HoreKA.
• you can use an existing conda environment or a new one to install cluster-hyperopt (we will call it "E2")
• open a shell in the topmost directory of the cluster-hyperopt repository
• activate the conda environment of your choice (E2) and execute "pip install -e ." This will install cluster-hyperopt into the active environment.

Creation of the config file

The config.yaml file is needed so that cluster_hyperopt can find all the necessary information for a hyperparameter search.

• You can create it in an arbitrary place on the server (BWUniCluster/HoreKA)

• See config_file_variables.md for information about every variable

• Parameters and metrics need to match those used in Section 2

• Here is a minimalistic configuration file that you can fill out and use:

  model:
  entry_point: TODO # The python file name that includes the function for evaluating the suggestionsfunction_name: TODO # the function that executes the training and evaluationdata_options:
  dataset_path: TODO # can be empty eg: ""git_options:
  git_uri: TODObranch: TODO # Either branch or version can be used. Using the option version allows to load specific tagsexperiment:
  cluster: TODO # Either "bwunicluster" or "horeka"number_chain_jobs: TODO # How many times should a job - the suggestion evaluation - be chained together. It is used to# cirumvent the problem of time outs in the clusterobservation_budget: TODO # Max number of trialsparallel_bandwidth: TODO # Number of parallel evaluationsconda_env: TODO # name of the conda env used for the model. This is E1, see "Section 1: Preconditions"parameters:
  TODOmetrics:
  TODOsbatch_options:
  partition: TODOsigopt_options:
  dev_run: true # Change this to false if you debugged your programproject_id: TODOexperiment_name: TODOclient_id: TODO

• Example file with some example values:

  model:
  entry_point: "main.py"# The python file name that includes the function for evaluating the suggestionsfunction_name: "run_hyperopt"# the function that executes the training and evaluationdata_options:
  dataset_path: "/path/to/dataset"# can be empty git_options:
  git_uri: "git@github.com:u-adrian/Tutorial.git"branch: "main"# Either branch or version can be used. Using the option version allows to load specific tagsexperiment:
  cluster: "horeka"# Either "bwunicluster" or "horeka"number_chain_jobs: 4# How many times should a job - the suggestion evaluation - be chained together. It is used to# cirumvent the problem of time outs in the clusterobservation_budget: 60# Max number of trialsparallel_bandwidth: 4# Number of parallel evaluationsmultimetric_experiment: falseconda_env: name_of_env # name of the conda env used for the model parameters:
  - name: max_depthtype: intbounds:
  min: 1max: 10
  - name: n_estimatorstype: intbounds:
  min: 1max: 10
  - name: bootstraptype: intgrid:
  - 0
  - 1
  - name: max_featurestype: categoricalcategorical_values:
  - 'sqrt'
  - 'log2'
  - name: criteriontype: categoricalcategorical_values:
  - 'gini'
  - 'entropy'metrics:
  - name: accuracyobjective: maximizestrategy: optimizesbatch_options:
  partition: "dev_gpu_4"sigopt_options:
  dev_run: false # If the dev api of sigopt should be used to get the suggestionsproject_id: "test_project"experiment_name: "tutorial_project"client_id: 11949

Creation of SigOpt Token file:

To get value suggestions for the hyperparameters from SIGOPT, cluster_hyperopt needs the API Tokens. We create a file sigopt_token(but you can name it how you want) where we store those tokens.

You can create this file in a location of your choice on the BWUniCluster/HoreKA:

SIGOPT_TOKEN=**********************************************
SIGOPT_DEV_TOKEN=**********************************************

The difference between these two tokens is that the SIGOPT_DEV_TOKEN only provides a random suggestion for developing purposes whereas SIGOPT_TOKEN provides serious suggestions. This option can then be activated or deactivated in the config file via sigopt_options.dev_run

You can find your SIGOPT_TOKEN and SIGOPT_DEV_TOKEN on the SIGOPT Website: https://app.sigopt.com/tokens/info under the menu points:

• <Your Username>
  • "API Tokens"

The next step is to assign an environment variable called SIGOPT_ENV_FILE to the path of this newly created file:

export SIGOPT_ENV_FILE=path/sigopt_token

Now cluster hyperopt can find and access the file, read out the tokens and request value suggestions from SIGOPT.

Section 4 Run HyperOpt

• Open shell in arbitrary folder (but on the cluster).
• activate the conda env where cluster-hyperopt is installed (E2)

  conda activate name_of_E2

• execute cluster hyperopt:

  python /.../cluster-hyperopt/sigopt_hyperopt/hyperopt.py start --config_path=/path_to_config/config.yaml

  The hyperopt.py file which will be executed is located in the cluster_hyperopt repository that you downloaded and installed at the beginning of Section 3: Installation of cluster-hyperopt.