When calling ProduceWordBags with weighting parameter specified it gets lost and the results always use WeightingCriteria.Tf.
The simplest repro-steps I have (based on LdaTransform sample):
// Get a small dataset as an IEnumerable and then read it as a ML.NET data set.IEnumerable<SamplesUtils.DatasetUtils.SampleTopicsData>data=SamplesUtils.DatasetUtils.GetTopicsData();vartrainData=ml.Data.LoadFromEnumerable(data);stringreview=nameof(SamplesUtils.DatasetUtils.SampleTopicsData.Review);// A pipeline for featurizing the "Review" columnvarpipeline=ml.Transforms.Text.ProduceWordBags("bags",review,ngramLength:1,weighting:Transforms.Text.NgramExtractingEstimator.WeightingCriteria.TfIdf);// The transformed datavartransformer=pipeline.Fit(trainData);vartransformed_data=transformer.Transform(trainData);varpreview=transformed_data.Preview();varbagsColumn=transformed_data.GetColumn<VBuffer<float>>("bags");foreach(varfeatureRowinbagsColumn){foreach(varvalueinfeatureRow.GetValues())Console.Write($"{value} ");Console.WriteLine("");}Expected output:
1.386294 0.6931472 0.6931472 1.386294 0.6931472 0.2876821 0 0 0 0 0 0 0
0 0.6931472 0.6931472 0 0.6931472 0.2876821 1.386294 1.386294 0 0 0 0 0
0.6931472 0.6931472 0.6931472 0.6931472 0.6931472
0 0 0 0 0 0.2876821 0 0 0.6931472 0.6931472 0.6931472 0.6931472 0.6931472
Actual output:
1 1 1 1 1 1 0 0 0 0 0 0 0
0 1 1 0 1 1 1 1 0 0 0 0 0
1 1 1 1 1
0 0 0 0 0 1 0 0 1 1 1 1 1
I'll send a PR with a fix shortly.
When calling
ProduceWordBagswithweightingparameter specified it gets lost and the results always useWeightingCriteria.Tf.The simplest repro-steps I have (based on
LdaTransformsample):Expected output:
Actual output:
I'll send a PR with a fix shortly.