Microsoft.ML sentiment-analysis results issue with different versions of Microsoft.ML

[mathewknott]

Seems to be an issue with this sample

The sample works with Nuget Package Microsoft.ML 0.3.0
When running 0.4.0 you get different results.

That is, both results in the console are positive for 0.4.0. The first result should be negative if it is supposed to be the same as 0.3.0, which I'd assume it would.

---

Document details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: c8e22825-c4ee-5a36-f911-8ad456970ecc
• Version Independent ID: ad0bf640-7dfc-1184-077c-358bb14277a7
• Content: Use ML.NET in a sentiment analysis binary classification scenario
• Content Source: docs/machine-learning/tutorials/sentiment-analysis.md
• Product: dotnet-ml
• GitHub Login: @JRAlexander
• Microsoft Alias: johalex

[LogOg]

Same issue, get both as positive even if I use exact wording from sample, i.e. 'RUDE'
Also noted that if use 2.0 in csproj file get lowere percentages than if use 2.1.
Tried bringing in the csproj, project.cs and SentimentData.cs files from samples on GitHub but got the same, thats how I saw the test results differ based on .net core version

[pkulikov]

Indeed, output with 0.3.0 version:

PredictionModel quality metrics evaluation
------------------------------------------
Accuracy: 66.67%
Auc: 94.44%
F1Score: 75.00%

Sentiment Predictions
---------------------
Sentiment: Please refrain from adding nonsense to Wikipedia. | Prediction: Negative
Sentiment: He is the best, and the article should say that. | Prediction: Positive

Output with 0.4.0 version:

PredictionModel quality metrics evaluation
------------------------------------------
Accuracy: 61.11%
Auc: 85.19%
F1Score: 72.00%

Sentiment Predictions
---------------------
Sentiment: Please refrain from adding nonsense to Wikipedia. | Prediction: Positive
Sentiment: He is the best, and the article should say that. | Prediction: Positive

Note, evaluation metrics values have also changed.

(I've run the code that is in the dotnet/samples repo)

It's not clear from the release notes what might have caused the observed change.

The used trainer is FastTreeBinaryClassifier. Its history shows mostly doc changes with some API update changes.

One more guess is that this commit might have introduced different behavior.

@OliaG do you know who might explain the observed difference in the tutorial output?

[JRAlexander]

@GalOshri, or @sfilipi, any insight on this?

[GalOshri]

This is likely due to the datasets being very small so random fluctuations can cause large differences in metrics (the 5% difference in accuracy just means that 1 out of the 18 examples in the test data flipped).

The other factor is that the tree parameters ({ NumLeaves = 5, NumTrees = 5, MinDocumentsInLeafs = 2 }) are very small which limits the performance of the trees. When I switch to default parameters, the AUC matches between 0.3 and 0.4. The accuracy is still different but only corresponds to one example's prediction being flipped.

I tried the same pipeline on a much larger dataset and the AUC/accuracy differed by around 0.3% (which likely is still due to the relatively small tree parameters and current inability to set a random seed).

[pkulikov]

@GalOshri

This is likely due to the datasets being very small so random fluctuations can cause large differences in metrics

Is it possible to produce such random fluctuations running against the fixed version of the Microsoft.ML? What causes the results be different in versions 0.3 and 0.4, but stay stable within one version?

[pkulikov]

@GalOshri the code uses TextFeaturizer that seems to have been influenced by dotnet/machinelearning#548 (am I correct here?). So, two sets of numerical features produced by 0.3 and 0.4 versions might be different. Thus, trained models differ. Is my understanding correct?

[GalOshri]

Good point. It seems like the implementation changed which likely affects the predictions. @zeahmed, can you please verify this?

[zeahmed]

Due the change in dotnet/machinelearning#548, one of the prediction changed.

Previously, it was 12/18 = 0.666666
Now, it is 11/18 = 0.61111

The dataset is so small to make any conclusion. However, @justinormont may help us on this.
@justinormont, Do we have benchmarks for it? Can you confirm if the change in TextFeaturizer has adversely effected the benchmarks?