wip - Add Sweepable Pipeline

[LittleLittleCloud]

Allow users to sweep over pre-defined paramaters with similar API of creating/training a pipeline using sweepers defined in Microsoft.ML.Sweeper.

In the current state, only RandomSweeper and GridSearchSweeper is provided, and other sweepers like SMAC will be added soon.

Right now, only one sweepable trainer is allowed in each pipeline, however, we can support multi-trainer easily with a few lines change.

One of a sample pipeline here, it uses RandomSweeper over a user-defined paramater set and sweeps over 20 times

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoEstimatorChainTest.cs

Line 24 in 2ba9ec6

publicvoidRecommendationE2ETest_RandomSweeper()

And part of it's Output

Alpha=0.7001 Lambda=0.11 LearningRate=0.001
RMSE: 3.08841817087814
Alpha=0.7001 Lambda=0.01 LearningRate=0.0709999949
RMSE: 0.88847094121806
Alpha=0.3001 Lambda=0.91 LearningRate=0.021
RMSE: 1.33374709132294
...

Why I create this PR

Auto sweeping parameter is quite a useful feature in Automate ML, AutoML.Net supports it internally but those API is not exposed and sweeping value is fixed, moreover, it only supports a few trainers. It will be great if ML.Net has some external API to allow users to sweep over trainer's parameters on the pipeline level.

What will be next

• allow sweeping Enum and Int type
• allow user sets up discrete numeric options using OptionBase class.
• adding more sweepers (SMAC....)
• ( if I have time ) create a similar API experience with what AutoML.Net has (saying, creating customer experiment based on pipeline, providing CV training and evaluation, and generate CodeGen afterwards)

[mentorfloat]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

[LittleLittleCloud]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

@mentorfloat
It's just time wasn't mature at that time but we are actively discussing it now, will follow up sooner

[justinormont]

@mentorfloat: You should be able to use just the hyperparameter optimization of the AutoML code if you either (1) prefeaturizing your dataset, or (2) by specifying an estimator chain as a prefeaturizer in the API call (sample).

This will still do the model selection, but you can reduce the search space to only one trainer. E.g. only LightGBM.

The main control missing from this method is specifying the exact range of the hyperparameters to search over.

---

Another method for sweeping hyperparamers is using the command-line sweepers in MAML. The code is still in the repo, but I haven't tried if it works.

The attached sweeping command (Sweep_SmartSweep.rsp) would be run as: dotnet ./bin/AnyCPU.Release/Microsoft.ML.Console/netcoreapp2.1/MML.dll @Sweep_SmartSweep.rsp. To make it work, you'd have to put in a proxy program called maml.exe on Win or maml on macOS/Linux which calls dotnet MML.dll with its given parameters, to proxy along the pipeline for the individual models in the sweep.

The MAML sweep command is powerful, but exposing with a clean C# interface would be best.

[LittleLittleCloud]

Allow users to sweep over pre-defined paramaters with similar API of creating/training a pipeline using sweepers defined in Microsoft.ML.Sweeper.

In the current state, only RandomSweeper and GridSearchSweeper is provided, and other sweepers like SMAC will be added soon.

Right now, only one sweepable trainer is allowed in each pipeline, however, we can support multi-trainer easily with a few lines change.

One of a sample pipeline here, it uses RandomSweeper over a user-defined paramater set and sweeps over 20 times

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoEstimatorChainTest.cs

Line 24 in 2ba9ec6

publicvoidRecommendationE2ETest_RandomSweeper()

And part of it's Output

Alpha=0.7001 Lambda=0.11 LearningRate=0.001
RMSE: 3.08841817087814
Alpha=0.7001 Lambda=0.01 LearningRate=0.0709999949
RMSE: 0.88847094121806
Alpha=0.3001 Lambda=0.91 LearningRate=0.021
RMSE: 1.33374709132294
...

Why I create this PR

Auto sweeping parameter is quite a useful feature in Automate ML, AutoML.Net supports it internally but those API is not exposed and sweeping value is fixed, moreover, it only supports a few trainers. It will be great if ML.Net has some external API to allow users to sweep over trainer's parameters on the pipeline level.

What will be next

• allow sweeping Enum and Int type
• allow user sets up discrete numeric options using OptionBase class.
• adding more sweepers (SMAC....)
• ( if I have time ) create a similar API experience with what AutoML.Net has (saying, creating customer experiment based on pipeline, providing CV training and evaluation, and generate CodeGen afterwards)

[mentorfloat]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

[LittleLittleCloud]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

@mentorfloat
It's just time wasn't mature at that time but we are actively discussing it now, will follow up sooner

[justinormont]

@mentorfloat: You should be able to use just the hyperparameter optimization of the AutoML code if you either (1) prefeaturizing your dataset, or (2) by specifying an estimator chain as a prefeaturizer in the API call (sample).

This will still do the model selection, but you can reduce the search space to only one trainer. E.g. only LightGBM.

The main control missing from this method is specifying the exact range of the hyperparameters to search over.

---

Another method for sweeping hyperparamers is using the command-line sweepers in MAML. The code is still in the repo, but I haven't tried if it works.

The attached sweeping command (Sweep_SmartSweep.rsp) would be run as: dotnet ./bin/AnyCPU.Release/Microsoft.ML.Console/netcoreapp2.1/MML.dll @Sweep_SmartSweep.rsp. To make it work, you'd have to put in a proxy program called maml.exe on Win or maml on macOS/Linux which calls dotnet MML.dll with its given parameters, to proxy along the pipeline for the individual models in the sweep.

The MAML sweep command is powerful, but exposing with a clean C# interface would be best.

[LittleLittleCloud]

Allow users to sweep over pre-defined paramaters with similar API of creating/training a pipeline using sweepers defined in Microsoft.ML.Sweeper.

In the current state, only RandomSweeper and GridSearchSweeper is provided, and other sweepers like SMAC will be added soon.

Right now, only one sweepable trainer is allowed in each pipeline, however, we can support multi-trainer easily with a few lines change.

One of a sample pipeline here, it uses RandomSweeper over a user-defined paramater set and sweeps over 20 times

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoEstimatorChainTest.cs

Line 24 in 2ba9ec6

publicvoidRecommendationE2ETest_RandomSweeper()

And part of it's Output

Alpha=0.7001 Lambda=0.11 LearningRate=0.001
RMSE: 3.08841817087814
Alpha=0.7001 Lambda=0.01 LearningRate=0.0709999949
RMSE: 0.88847094121806
Alpha=0.3001 Lambda=0.91 LearningRate=0.021
RMSE: 1.33374709132294
...

Why I create this PR

Auto sweeping parameter is quite a useful feature in Automate ML, AutoML.Net supports it internally but those API is not exposed and sweeping value is fixed, moreover, it only supports a few trainers. It will be great if ML.Net has some external API to allow users to sweep over trainer's parameters on the pipeline level.

What will be next

• allow sweeping Enum and Int type
• allow user sets up discrete numeric options using OptionBase class.
• adding more sweepers (SMAC....)
• ( if I have time ) create a similar API experience with what AutoML.Net has (saying, creating customer experiment based on pipeline, providing CV training and evaluation, and generate CodeGen afterwards)

[mentorfloat]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

[LittleLittleCloud]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

@mentorfloat
It's just time wasn't mature at that time but we are actively discussing it now, will follow up sooner

[justinormont]

@mentorfloat: You should be able to use just the hyperparameter optimization of the AutoML code if you either (1) prefeaturizing your dataset, or (2) by specifying an estimator chain as a prefeaturizer in the API call (sample).

This will still do the model selection, but you can reduce the search space to only one trainer. E.g. only LightGBM.

The main control missing from this method is specifying the exact range of the hyperparameters to search over.

---

Another method for sweeping hyperparamers is using the command-line sweepers in MAML. The code is still in the repo, but I haven't tried if it works.

The attached sweeping command (Sweep_SmartSweep.rsp) would be run as: dotnet ./bin/AnyCPU.Release/Microsoft.ML.Console/netcoreapp2.1/MML.dll @Sweep_SmartSweep.rsp. To make it work, you'd have to put in a proxy program called maml.exe on Win or maml on macOS/Linux which calls dotnet MML.dll with its given parameters, to proxy along the pipeline for the individual models in the sweep.

The MAML sweep command is powerful, but exposing with a clean C# interface would be best.

[LittleLittleCloud]

Allow users to sweep over pre-defined paramaters with similar API of creating/training a pipeline using sweepers defined in Microsoft.ML.Sweeper.

In the current state, only RandomSweeper and GridSearchSweeper is provided, and other sweepers like SMAC will be added soon.

Right now, only one sweepable trainer is allowed in each pipeline, however, we can support multi-trainer easily with a few lines change.

One of a sample pipeline here, it uses RandomSweeper over a user-defined paramater set and sweeps over 20 times

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoEstimatorChainTest.cs

Line 24 in 2ba9ec6

publicvoidRecommendationE2ETest_RandomSweeper()

And part of it's Output

Alpha=0.7001 Lambda=0.11 LearningRate=0.001
RMSE: 3.08841817087814
Alpha=0.7001 Lambda=0.01 LearningRate=0.0709999949
RMSE: 0.88847094121806
Alpha=0.3001 Lambda=0.91 LearningRate=0.021
RMSE: 1.33374709132294
...

Why I create this PR

Auto sweeping parameter is quite a useful feature in Automate ML, AutoML.Net supports it internally but those API is not exposed and sweeping value is fixed, moreover, it only supports a few trainers. It will be great if ML.Net has some external API to allow users to sweep over trainer's parameters on the pipeline level.

What will be next

• allow sweeping Enum and Int type
• allow user sets up discrete numeric options using OptionBase class.
• adding more sweepers (SMAC....)
• ( if I have time ) create a similar API experience with what AutoML.Net has (saying, creating customer experiment based on pipeline, providing CV training and evaluation, and generate CodeGen afterwards)

[mentorfloat]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

[LittleLittleCloud]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

@mentorfloat
It's just time wasn't mature at that time but we are actively discussing it now, will follow up sooner

[justinormont]

@mentorfloat: You should be able to use just the hyperparameter optimization of the AutoML code if you either (1) prefeaturizing your dataset, or (2) by specifying an estimator chain as a prefeaturizer in the API call (sample).

This will still do the model selection, but you can reduce the search space to only one trainer. E.g. only LightGBM.

The main control missing from this method is specifying the exact range of the hyperparameters to search over.

---

Another method for sweeping hyperparamers is using the command-line sweepers in MAML. The code is still in the repo, but I haven't tried if it works.

The attached sweeping command (Sweep_SmartSweep.rsp) would be run as: dotnet ./bin/AnyCPU.Release/Microsoft.ML.Console/netcoreapp2.1/MML.dll @Sweep_SmartSweep.rsp. To make it work, you'd have to put in a proxy program called maml.exe on Win or maml on macOS/Linux which calls dotnet MML.dll with its given parameters, to proxy along the pipeline for the individual models in the sweep.

The MAML sweep command is powerful, but exposing with a clean C# interface would be best.

[LittleLittleCloud]

Allow users to sweep over pre-defined paramaters with similar API of creating/training a pipeline using sweepers defined in Microsoft.ML.Sweeper.

In the current state, only RandomSweeper and GridSearchSweeper is provided, and other sweepers like SMAC will be added soon.

Right now, only one sweepable trainer is allowed in each pipeline, however, we can support multi-trainer easily with a few lines change.

One of a sample pipeline here, it uses RandomSweeper over a user-defined paramater set and sweeps over 20 times

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoEstimatorChainTest.cs

Line 24 in 2ba9ec6

publicvoidRecommendationE2ETest_RandomSweeper()

And part of it's Output

Alpha=0.7001 Lambda=0.11 LearningRate=0.001
RMSE: 3.08841817087814
Alpha=0.7001 Lambda=0.01 LearningRate=0.0709999949
RMSE: 0.88847094121806
Alpha=0.3001 Lambda=0.91 LearningRate=0.021
RMSE: 1.33374709132294
...

Why I create this PR

Auto sweeping parameter is quite a useful feature in Automate ML, AutoML.Net supports it internally but those API is not exposed and sweeping value is fixed, moreover, it only supports a few trainers. It will be great if ML.Net has some external API to allow users to sweep over trainer's parameters on the pipeline level.

What will be next

• allow sweeping Enum and Int type
• allow user sets up discrete numeric options using OptionBase class.
• adding more sweepers (SMAC....)
• ( if I have time ) create a similar API experience with what AutoML.Net has (saying, creating customer experiment based on pipeline, providing CV training and evaluation, and generate CodeGen afterwards)

[mentorfloat]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

[LittleLittleCloud]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

@mentorfloat
It's just time wasn't mature at that time but we are actively discussing it now, will follow up sooner

[justinormont]

@mentorfloat: You should be able to use just the hyperparameter optimization of the AutoML code if you either (1) prefeaturizing your dataset, or (2) by specifying an estimator chain as a prefeaturizer in the API call (sample).

This will still do the model selection, but you can reduce the search space to only one trainer. E.g. only LightGBM.

The main control missing from this method is specifying the exact range of the hyperparameters to search over.

---

Another method for sweeping hyperparamers is using the command-line sweepers in MAML. The code is still in the repo, but I haven't tried if it works.

The attached sweeping command (Sweep_SmartSweep.rsp) would be run as: dotnet ./bin/AnyCPU.Release/Microsoft.ML.Console/netcoreapp2.1/MML.dll @Sweep_SmartSweep.rsp. To make it work, you'd have to put in a proxy program called maml.exe on Win or maml on macOS/Linux which calls dotnet MML.dll with its given parameters, to proxy along the pipeline for the individual models in the sweep.

The MAML sweep command is powerful, but exposing with a clean C# interface would be best.

[LittleLittleCloud]

Allow users to sweep over pre-defined paramaters with similar API of creating/training a pipeline using sweepers defined in Microsoft.ML.Sweeper.

In the current state, only RandomSweeper and GridSearchSweeper is provided, and other sweepers like SMAC will be added soon.

Right now, only one sweepable trainer is allowed in each pipeline, however, we can support multi-trainer easily with a few lines change.

One of a sample pipeline here, it uses RandomSweeper over a user-defined paramater set and sweeps over 20 times

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoEstimatorChainTest.cs

Line 24 in 2ba9ec6

publicvoidRecommendationE2ETest_RandomSweeper()

And part of it's Output

Alpha=0.7001 Lambda=0.11 LearningRate=0.001
RMSE: 3.08841817087814
Alpha=0.7001 Lambda=0.01 LearningRate=0.0709999949
RMSE: 0.88847094121806
Alpha=0.3001 Lambda=0.91 LearningRate=0.021
RMSE: 1.33374709132294
...

Why I create this PR

Auto sweeping parameter is quite a useful feature in Automate ML, AutoML.Net supports it internally but those API is not exposed and sweeping value is fixed, moreover, it only supports a few trainers. It will be great if ML.Net has some external API to allow users to sweep over trainer's parameters on the pipeline level.

What will be next

• allow sweeping Enum and Int type
• allow user sets up discrete numeric options using OptionBase class.
• adding more sweepers (SMAC....)
• ( if I have time ) create a similar API experience with what AutoML.Net has (saying, creating customer experiment based on pipeline, providing CV training and evaluation, and generate CodeGen afterwards)

[mentorfloat]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

[LittleLittleCloud]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

@mentorfloat
It's just time wasn't mature at that time but we are actively discussing it now, will follow up sooner

[justinormont]

@mentorfloat: You should be able to use just the hyperparameter optimization of the AutoML code if you either (1) prefeaturizing your dataset, or (2) by specifying an estimator chain as a prefeaturizer in the API call (sample).

This will still do the model selection, but you can reduce the search space to only one trainer. E.g. only LightGBM.

The main control missing from this method is specifying the exact range of the hyperparameters to search over.

---

Another method for sweeping hyperparamers is using the command-line sweepers in MAML. The code is still in the repo, but I haven't tried if it works.

The attached sweeping command (Sweep_SmartSweep.rsp) would be run as: dotnet ./bin/AnyCPU.Release/Microsoft.ML.Console/netcoreapp2.1/MML.dll @Sweep_SmartSweep.rsp. To make it work, you'd have to put in a proxy program called maml.exe on Win or maml on macOS/Linux which calls dotnet MML.dll with its given parameters, to proxy along the pipeline for the individual models in the sweep.

The MAML sweep command is powerful, but exposing with a clean C# interface would be best.

[LittleLittleCloud]

Allow users to sweep over pre-defined paramaters with similar API of creating/training a pipeline using sweepers defined in Microsoft.ML.Sweeper.

In the current state, only RandomSweeper and GridSearchSweeper is provided, and other sweepers like SMAC will be added soon.

Right now, only one sweepable trainer is allowed in each pipeline, however, we can support multi-trainer easily with a few lines change.

One of a sample pipeline here, it uses RandomSweeper over a user-defined paramater set and sweeps over 20 times

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoEstimatorChainTest.cs

Line 24 in 2ba9ec6

publicvoidRecommendationE2ETest_RandomSweeper()

And part of it's Output

Alpha=0.7001 Lambda=0.11 LearningRate=0.001
RMSE: 3.08841817087814
Alpha=0.7001 Lambda=0.01 LearningRate=0.0709999949
RMSE: 0.88847094121806
Alpha=0.3001 Lambda=0.91 LearningRate=0.021
RMSE: 1.33374709132294
...

Why I create this PR

Auto sweeping parameter is quite a useful feature in Automate ML, AutoML.Net supports it internally but those API is not exposed and sweeping value is fixed, moreover, it only supports a few trainers. It will be great if ML.Net has some external API to allow users to sweep over trainer's parameters on the pipeline level.

What will be next

• allow sweeping Enum and Int type
• allow user sets up discrete numeric options using OptionBase class.
• adding more sweepers (SMAC....)
• ( if I have time ) create a similar API experience with what AutoML.Net has (saying, creating customer experiment based on pipeline, providing CV training and evaluation, and generate CodeGen afterwards)

[mentorfloat]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

[LittleLittleCloud]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

@mentorfloat
It's just time wasn't mature at that time but we are actively discussing it now, will follow up sooner

[justinormont]

@mentorfloat: You should be able to use just the hyperparameter optimization of the AutoML code if you either (1) prefeaturizing your dataset, or (2) by specifying an estimator chain as a prefeaturizer in the API call (sample).

This will still do the model selection, but you can reduce the search space to only one trainer. E.g. only LightGBM.

The main control missing from this method is specifying the exact range of the hyperparameters to search over.

---

Another method for sweeping hyperparamers is using the command-line sweepers in MAML. The code is still in the repo, but I haven't tried if it works.

The attached sweeping command (Sweep_SmartSweep.rsp) would be run as: dotnet ./bin/AnyCPU.Release/Microsoft.ML.Console/netcoreapp2.1/MML.dll @Sweep_SmartSweep.rsp. To make it work, you'd have to put in a proxy program called maml.exe on Win or maml on macOS/Linux which calls dotnet MML.dll with its given parameters, to proxy along the pipeline for the individual models in the sweep.

The MAML sweep command is powerful, but exposing with a clean C# interface would be best.

[LittleLittleCloud]

Allow users to sweep over pre-defined paramaters with similar API of creating/training a pipeline using sweepers defined in Microsoft.ML.Sweeper.

In the current state, only RandomSweeper and GridSearchSweeper is provided, and other sweepers like SMAC will be added soon.

Right now, only one sweepable trainer is allowed in each pipeline, however, we can support multi-trainer easily with a few lines change.

One of a sample pipeline here, it uses RandomSweeper over a user-defined paramater set and sweeps over 20 times

machinelearning/test/Microsoft.ML.AutoML.Tests/AutoEstimatorChainTest.cs

Line 24 in 2ba9ec6

publicvoidRecommendationE2ETest_RandomSweeper()

And part of it's Output

Alpha=0.7001 Lambda=0.11 LearningRate=0.001
RMSE: 3.08841817087814
Alpha=0.7001 Lambda=0.01 LearningRate=0.0709999949
RMSE: 0.88847094121806
Alpha=0.3001 Lambda=0.91 LearningRate=0.021
RMSE: 1.33374709132294
...

Why I create this PR

Auto sweeping parameter is quite a useful feature in Automate ML, AutoML.Net supports it internally but those API is not exposed and sweeping value is fixed, moreover, it only supports a few trainers. It will be great if ML.Net has some external API to allow users to sweep over trainer's parameters on the pipeline level.

What will be next

• allow sweeping Enum and Int type
• allow user sets up discrete numeric options using OptionBase class.
• adding more sweepers (SMAC....)
• ( if I have time ) create a similar API experience with what AutoML.Net has (saying, creating customer experiment based on pipeline, providing CV training and evaluation, and generate CodeGen afterwards)

[mentorfloat]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

[LittleLittleCloud]

@LittleLittleCloud A method to perform automated hyperparameter tuning outside of using AutoML will be helpful. Was there a reason why you closed the PR?

@mentorfloat
It's just time wasn't mature at that time but we are actively discussing it now, will follow up sooner

[justinormont]

@mentorfloat: You should be able to use just the hyperparameter optimization of the AutoML code if you either (1) prefeaturizing your dataset, or (2) by specifying an estimator chain as a prefeaturizer in the API call (sample).

This will still do the model selection, but you can reduce the search space to only one trainer. E.g. only LightGBM.

The main control missing from this method is specifying the exact range of the hyperparameters to search over.

---

Another method for sweeping hyperparamers is using the command-line sweepers in MAML. The code is still in the repo, but I haven't tried if it works.

The attached sweeping command (Sweep_SmartSweep.rsp) would be run as: dotnet ./bin/AnyCPU.Release/Microsoft.ML.Console/netcoreapp2.1/MML.dll @Sweep_SmartSweep.rsp. To make it work, you'd have to put in a proxy program called maml.exe on Win or maml on macOS/Linux which calls dotnet MML.dll with its given parameters, to proxy along the pipeline for the individual models in the sweep.

The MAML sweep command is powerful, but exposing with a clean C# interface would be best.