Predict hangs indefinetly

[cosmincatalin]

System information

• OS version/distro: Mac OS X 10.13
• .NET Version (eg., dotnet --info): 2.0.5

Issue

• I have a .csv file with data that I use for individual predictions. I use a FastTreeRegressor model loaded from disk. I make individual predictions and not batch predictions because I need the absolute error histogram. At first, predicting works fine, but then, after what appears to be a random number of executions, the Predict method hangs without anything else happening.
• I expect either Predict failing with some Exception, or better, fail for the same example or, even better, Predict to work for all my examples, which it should.

By random I mean that I have a few million examples I have to go through, but predicting stops anywhere between after a few hundreds predictions or at best after a few thousands.

Source code / logs

The stripped down source code for Program.cs. Note that I have also tried with version 0.2

using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.ML;
using PredictionModelling;

namespace PredictionMetrics
{
 class Program
 {

 static async Task Main(string[] args)
 {
 foreach (string culture in cultures)
 {
 var model = await PredictionModel.ReadAsync<Data, DataPrediction>("model.zip");
 StreamReader file = new StreamReader("test.csv");

 file.ReadLine(); // This is for the header

 string line;

 using (StreamWriter outputFile = new StreamWriter("test-predictions.csv"))
 {
 var i = 0;
 outputFile.WriteLine($"actual-value,predicted-value");
 while ((line = file.ReadLine()) != null)
 {
 var data = line.Split(new[] { ',' });
 var data = new Data()
 {
 Feature0 = float.Parse(data[0]),
 Feature1 = float.Parse(data[1])
 };
 var csvLine = $"{data[2]},{model.Predict(data).Value}";
 outputFile.WriteLine(csvLine);
 i++;
 if (i % 100 == 0)
 {
 outputFile.Flush();
 }
 }
 }
 file.Close();
 }
 }
 }
}

[glebuk]

Cosmin,
Do you happen to have any filters in your trained pipeline? Sometimes, if you filter the input row during training, the prediction engine would also filter rows to score and that can stall the pipeline. Ideally, the trained model does not have any filter transforms and all filtering is done before training. For example, if you have missing values it is good idea to replace the NAFilter with NAHandle transform.

[cosmincatalin]

I do not have any filters. The model pipeline looks like this:

var pipeline = new LearningPipeline();
pipeline.Add(new TextLoader<SearchesData>(DataPath, useHeader: true, separator: ","));
pipeline.Add(new ColumnCopier(("Value", "Label")));
pipeline.Add(new ColumnConcatenator("Features", "Feature1", "Feature2"));
pipeline.Add(new FastTreeRegressor() { DiskTranspose = true });

[shauheen]

The API has changed since this bug was filed, @shmoradims can you verify if this is still valid.