The ML.Net library suffers from a lack of decoupling between data preparation and model training, required to do an efficient grid search over training parameters.
That is, ideally the API should be structured in such a way that it is possible to do the following:
- Prepare the data set once, so that it can be re-used multiple times. As much as possible, any pre-training calculations should be done up front (or perhaps cached to be re-used). For large data sets, the overhead of repeating this step each time is significant, taking as long or longer than the training itself.
- For algorithms with multiple training iterations, it should be straightforward to retain the intermediate trained models at each iteration (or at a specified set of iterations). This way, it is then easy to compute metrics for the intermediate models on training and validation data sets, and ultimately select one of the intermediate models for use in production without having to re-run the training.
For example, consider training a LightGBM model. This is the training method in LightGbmTrainerBase.cs:
publicvoidTrain(RoleMappedDatadata){Datasetdtrain;CategoricalMetaDatacatMetaData;using(varch=Host.Start("Loading data for LightGBM")){using(varpch=Host.StartProgressChannel("Loading data for LightGBM"))dtrain=LoadTrainingData(ch,data,outcatMetaData);ch.Done();}using(varch=Host.Start("Training with LightGBM")){using(varpch=Host.StartProgressChannel("Training with LightGBM"))TrainCore(ch,pch,dtrain,catMetaData);ch.Done();}dtrain.Dispose();DisposeParallelTraining();}In order to address point 1) above, the dtrain object returned by LoadTrainingData should be available to be re-used. This would require that the configuration parameters for data preparation are specified separately to those for training, instead of all thrown in together into the LightGbmArguments type.
Now, in regards to point 2) above, note that the TrainCore method calls WrappedLightGBMTraining.Train, which has the following structure:
publicstaticBoosterTrain(IChannelch,IProgressChannelpch,Dictionary<string,object>parameters,Datasetdtrain,Datasetdvalid=null,intnumIteration=100,boolverboseEval=true,intearlyStoppingRound=0){// create Booster.Boosterbst=newBooster(parameters,dtrain,dvalid);for(intiter=0;iter<numIteration;++iter){// training logic}returnbst;}In order to get the intermediate models, this method should return Booster [] instead of just the final Booster (or perhaps instead in this case, the Booster object should support extraction of a prediction model which only contains the first N trees of the ensemble).
Perhaps there is already the facility to do this in ML.Net, but I'm unable to find anything from my reading of the source or any of the examples.
I think 99.9% of all machine learning research requires doing a parameter grid search at some stage, and hence this is essential functionality that should be as efficient as possible.
The ML.Net library suffers from a lack of decoupling between data preparation and model training, required to do an efficient grid search over training parameters.
That is, ideally the API should be structured in such a way that it is possible to do the following:
For example, consider training a LightGBM model. This is the training method in
LightGbmTrainerBase.cs:In order to address point 1) above, the
dtrainobject returned byLoadTrainingDatashould be available to be re-used. This would require that the configuration parameters for data preparation are specified separately to those for training, instead of all thrown in together into theLightGbmArgumentstype.Now, in regards to point 2) above, note that the
TrainCoremethod callsWrappedLightGBMTraining.Train, which has the following structure:In order to get the intermediate models, this method should return
Booster []instead of just the finalBooster(or perhaps instead in this case, theBoosterobject should support extraction of a prediction model which only contains the firstNtrees of the ensemble).Perhaps there is already the facility to do this in ML.Net, but I'm unable to find anything from my reading of the source or any of the examples.
I think 99.9% of all machine learning research requires doing a parameter grid search at some stage, and hence this is essential functionality that should be as efficient as possible.