Uh oh!
There was an error while loading. Please reload this page.
Normalize documentation - #3244
Conversation
Codecov Report
@@ Coverage Diff @@## master #3244 +/- ##
==========================================
- Coverage 72.65% 72.64% -0.01%
==========================================
Files 807 807 Lines 145191 145191 Branches 16223 16223 ==========================================
- Hits 105485 105480 -5 - Misses 35290 35293 +3 - Partials 4416 4418 +2
|
| foreach (var row in columnFixZero) | ||
| Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4")))); | ||
| // 1.0000, 0.0000, 1.0000, 0.0000 | ||
| // 0.5000, 0.5000, 0.0000, 0.5000 |
There was a problem hiding this comment.
nit: could you add (here and in other files)
// Expected output: #Resolved
There was a problem hiding this comment.
I suspect expected values are incorrect: input: 2, 2, 2, 0 minmaxnormalized returns 1,1,1,0 or am I missing something?
| var transformParams = (normalizeTransform.GetNormalizerModelParameters(0) as Microsoft.ML.Transforms.NormalizingTransformer.BinNormalizerModelParameters<ImmutableArray<float>>); | ||
| Console.WriteLine($"Values for slot 0 would be transfromed by applying y = (Index(x) / {transformParams.Density[0]}) - {(transformParams.Offset.Length == 0 ? 0 : transformParams.Offset[0])}"); | ||
| Console.WriteLine("Where Index(x) is index of bin to which x belongs"); | ||
| Console.WriteLine($"Bins upper borders are: {string.Join(" ", transformParams.UpperBounds[0])}"); |
There was a problem hiding this comment.
borders [](start = 43, length = 7)
Maybe would suggest to use "bounds" instead of "borders". #Resolved
| // 0.0000, -0.5000, 0.0000, 1.0000 | ||
| // 0.0000, -0.5000, 0.0000, 1.0000 | ||
| // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for function. |
There was a problem hiding this comment.
for function. [](start = 118, length = 13)
.... as parameter to GetNormalizerModelParameters. #Resolved
| // 0.0000, -0.5000, 0.0000, 1.0000 | ||
| // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for function. | ||
| // If case if we have multiple column transformations we need to pass index of InputOutputColumnPair. |
There was a problem hiding this comment.
If case [](start = 15, length = 8)
Remove "If case", ere and in other files. #Resolved
| // If case if we have multiple column transformations we need to pass index of InputOutputColumnPair. | ||
| var transformParams = (normalizeTransform.GetNormalizerModelParameters(0) as Microsoft.ML.Transforms.NormalizingTransformer.AffineNormalizerModelParameters<ImmutableArray<float>>); | ||
| Console.WriteLine($"Values for slot 1 would be transfromed by applying y= (x - ({(transformParams.Offset.Length == 0 ? 0 : transformParams.Offset[1])})) * {transformParams.Scale[1]}"); | ||
| // Values for slot 1 would be transfromed by applying y= (x - (-1)) * 0.3333333 |
There was a problem hiding this comment.
y= [](start = 66, length = 3)
could you add a space after "y"? #Resolved
| var normalize = mlContext.Transforms.NormalizeMeanVariance("Features", useCdf: true); | ||
| // NormalizeMeanVariance normalizes the data based on the computed mean and variance of the data. | ||
| var normalizeNoCdf = mlContext.Transforms.NormalizeMeanVariance("Features", useCdf: false); |
There was a problem hiding this comment.
I would not repeat what NormalizeMeanVariance does.
| // NormalizeBinning normalizes the data by constructing equidensity bins and produce output based on | ||
| // to which bin original value belong but make sure zero values would remain zero after normalization. | ||
| // Helps preserve sparsity. |
There was a problem hiding this comment.
Maybe I would not repeat this explanation. Here and also in other samples.
| Console.WriteLine("Where Index(x) is index of bin to which x belongs"); | ||
| Console.WriteLine($"Bins upper borders are: {string.Join(" ", fixZeroParams.UpperBounds[1])}"); | ||
| // Values for slot 1 would be transfromed by applying y = (Index(x) / 2) - 0.5 | ||
| // Where Index(x) is index of bin to which x belongs |
There was a problem hiding this comment.
Where Index(x) is index of bin to which x belongs [](start = 15, length = 49)
// Where Index(x) is the index of the bin to which x belongs" #Resolved
| var normalize = mlContext.Transforms.NormalizeBinning("Features", maximumBinCount: 4, fixZero: false); | ||
| // NormalizeBinning normalizes the data by constructing equidensity bins and produce output based on | ||
| // to which bin original value belong but make sure zero values would remain zero after normalization. |
There was a problem hiding this comment.
duplicate comment...see line 26-27 #Closed
There was a problem hiding this comment.
| // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters. | ||
| // If we have multiple column transformations we need to pass index of InputOutputColumnPair. | ||
| var transformParams = (normalizeTransform.GetNormalizerModelParameters(0) as Microsoft.ML.Transforms.NormalizingTransformer.BinNormalizerModelParameters<ImmutableArray<float>>); |
There was a problem hiding this comment.
Microsoft.ML.Transforms [](start = 89, length = 23)
Can we make it a using statement? This line is getting very long. #Resolved
| // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters. | ||
| // If we have multiple column transformations we need to pass index of InputOutputColumnPair. | ||
| var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>; | ||
| Console.WriteLine($"Values for slot 1 would be transfromed by applying y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))" ); |
There was a problem hiding this comment.
slot 1 [](start = 43, length = 6)
What are slot 0 and slot 1? In this particular sample this one and below are both slot 1? #Resolved
There was a problem hiding this comment.
| // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters. | ||
| // If we have multiple column transformations we need to pass index of InputOutputColumnPair. | ||
| var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>; | ||
| Console.WriteLine($"Values for slot 1 would be transfromed by applying y = (x - ({(transformParams.Offset.Length == 0 ? 0 : transformParams.Offset[1])})) * {transformParams.Scale[1]}"); |
There was a problem hiding this comment.
slot 1 [](start = 43, length = 6)
see my comments in other sample. #Resolved
| namespace Samples.Dynamic | ||
| { | ||
| public class NormalizeBinning |
There was a problem hiding this comment.
NormalizeBinning [](start = 17, length = 16)
one line comment about what this does.
There was a problem hiding this comment.
| // NormalizeBinning normalizes the data by constructing equidensity bins and produce output based on | ||
| // to which bin original value belong but make sure zero values would remain zero after normalization. | ||
| // Helps preserve sparsity. | ||
| var normalizeFixZero = mlContext.Transforms.NormalizeBinning("Features", maximumBinCount: 4, fixZero: true); |
There was a problem hiding this comment.
fixZero: true [](start = 105, length = 13)
love the variation on the parameter! #WontFix
| // Expected output: | ||
| // 1.0000, 0.6667, 1.0000, 0.0000 | ||
| // 0.6667, 1.0000, 0.6667, 0.0000 | ||
| // 0.3333, 0.3333, 0.3333, 0.0000 |
There was a problem hiding this comment.
0.3333 [](start = 24, length = 6)
I'd emphasize this value.
There was a problem hiding this comment.
| // 0.3333, 0.0000, 0.3333, 0.0000 | ||
| // 0.0000, -0.3333, 0.0000, 1.0000 | ||
| // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters. |
There was a problem hiding this comment.
we need to pass 0, the index of this column in the dataview, as parameter for..
There was a problem hiding this comment.
| // 0.0000, -0.3333, 0.0000, 1.0000 | ||
| // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters. | ||
| // If we have multiple column transformations we need to pass index of InputOutputColumnPair. |
There was a problem hiding this comment.
multiple column transformations [](start = 26, length = 31)
multiple columns #Resolved
| var mlContext = new MLContext(); | ||
| var samples = new List<DataPoint>() | ||
| { | ||
| new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} }, |
There was a problem hiding this comment.
maybe no need to have two columns with the same exact values.
| /// ]]> | ||
| /// </format> | ||
| /// </example> | ||
| public static NormalizingEstimator NormalizeMinMax(this TransformsCatalog catalog, InputOutputColumnPair[] columns, |
There was a problem hiding this comment.
NormalizeMinMax [](start = 43, length = 15)
you're not invoking this API. I would not put the example here. #Resolved
| /// <format type="text/markdown"> | ||
| /// <] | ||
| /// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs)] |
There was a problem hiding this comment.
[!code-csharp[NormalizeLogMeanVariance] [](start = 12, length = 39)
you're not invoking this API. I would not put the example here. #Resolved
| /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param> | ||
| /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param> | ||
| /// <param name="maximumBinCount">Maximum number of bins (power of 2 recommended).</param> | ||
| /// <example> |
There was a problem hiding this comment.
you're not invoking this API. I would not put the example here. #Resolved
| /// [!code-csharp[NormalizeBinning](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeSupervisedBinning.cs)] | ||
| /// ]]> | ||
| /// </format> | ||
| /// </example> |
There was a problem hiding this comment.
[](start = 21, length = 1)
you're not invoking this API. I would not put the example here. #Resolved
| using System.Linq; | ||
| using Microsoft.ML; | ||
| using Microsoft.ML.Data; | ||
| using static Microsoft.ML.Transforms.NormalizingTransformer; |
There was a problem hiding this comment.
Yes, otherwise I need to write Microsoft.ML.Transforms.NormalizingTransformer.BinNormalizerModelParameters which takes too much space
In reply to: 274057369 [](ancestors = 274057369)
| // If we have multiple column transformations we need to pass index of InputOutputColumnPair. | ||
| var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as BinNormalizerModelParameters<ImmutableArray<float>>; | ||
| Console.WriteLine($"Values for slot 0 would be transfromed by applying y = (Index(x) / {transformParams.Density[0]}) - {(transformParams.Offset.Length == 0 ? 0 : transformParams.Offset[0])}"); | ||
| Console.WriteLine("Where Index(x) is the index of the bin to which x belongs"); |
There was a problem hiding this comment.
please break super-long lines into two lines #Resolved
| using System.Linq; | ||
| using Microsoft.ML; | ||
| using Microsoft.ML.Data; | ||
| using static Microsoft.ML.Transforms.NormalizingTransformer; |
There was a problem hiding this comment.
sing static Microsoft.ML.Transforms.NormalizingTransformer; [](start = 1, length = 59)
ditto #Resolved
| var normalize = mlContext.Transforms.NormalizeBinning("Features", maximumBinCount: 4, fixZero: false); | ||
| // NormalizeBinning normalizes the data by constructing equidensity bins and produce output based on | ||
| // to which bin original value belong but make sure zero values would remain zero after normalization. |
There was a problem hiding this comment.
please move the details to similar to your other PR
towards #1209