Add FixZero for LogMeanVariance normalizer#3916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
artidoro merged 4 commits into from
Jul 1, 2019
Merged

Add FixZero for LogMeanVariance normalizer #3916

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9623019
add fixZero for LogMeanVariance normalizer
artidoro Jun 26, 2019
c829ad4
add test and sample reference in catalog
artidoro Jun 26, 2019
0527161
addressing review comments
artidoro Jun 28, 2019
52c19ed
adding the normalizer equation
artidoro Jul 1, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 16 additions & 16 deletions docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -12,15 +12,15 @@ public class NormalizeLogMeanVariance
{
public static void Example()
{
// Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
// Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
// as well as the source of randomness.
var mlContext = new MLContext();
var samples = new List<DataPoint>()
{
new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} },
new DataPoint(){ Features = new float[4] { 2, 2, 2, 0} },
new DataPoint(){ Features = new float[4] { 0, 0, 1, 0} },
new DataPoint(){ Features = new float[4] {-1,-1,-1, 1} }
new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },

@wschinwschinJun 27, 2019•
edited by artidoro
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to FixZero? #Resolved

@artidoroartidoroJun 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's related to the original issue posted by Lisa. They were encountering an issue with the MeanVariance normalizer which does not handle data that looks like float.MaxValue, float.MinValue. I think it would be useful to show that LogMeanVariance normalizer accepts this data in the related sample.

In reply to: 298254220 [](ancestors = 298254220)

new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
};
// Convert training data to IDataView, the general data type used in ML.NET.
var data = mlContext.Data.LoadFromEnumerable(samples);
Expand All@@ -41,19 +41,19 @@ public static void Example()
foreach (var row in column)
Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
// Expected output:
// 0.1587, 0.1587, 0.8654, 0.0000
// 0.8413, 0.8413, 0.5837, 0.0000
// 0.0000, 0.0000, 0.0940, 0.0000
// 0.0000, 0.0000, 0.0000, 0.0000
// 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
// 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
// 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
// 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
foreach (var row in columnFixZero)
Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
// Expected output:
// 1.8854, 1.8854, 5.2970, 0.0000
// 4.7708, 4.7708, 3.0925, 0.0000
// -1.0000,-1.0000, 0.8879, 0.0000
// 3.8854,-3.8854,-3.5213, 0.0000
// 1.8854, 1.8854, 5.2970, 0.0000, 7670682000000000000000000000000000000.0000
// 4.7708, 4.7708, 3.0925, 0.0000, -7670682000000000000000000000000000000.0000
// -1.0000,-1.0000, 0.8879, 0.0000, -1.0000
// -3.8854,-3.8854,-3.5213, 0.0000, -0.9775

// Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
// If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
Expand All@@ -64,18 +64,18 @@ public static void Example()
// ERF is https://en.wikipedia.org/wiki/Error_function.
// Expected output:
// The 1-index value in resulting array would be produce by:
// y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
// y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
var scale = noCdfParams.Scale[1];
Console.WriteLine($"The 1-index value in resulting array would be produce by: y = (x - ({offset})) * {scale}");
// Expected output:
// The 1-index value in resulting array would be produce by: y = (x - (2.88539)) * 0.3465736
// The 1-index value in resulting array would be produce by: y = (x - (0.3465736)) * 2.88539
}

private class DataPoint
{
[VectorType(4)]
[VectorType(5)]
public float[] Features { get; set; }
}
}
Expand Down
82 changes: 82 additions & 0 deletions docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.NormalizingTransformer;

namespace Samples.Dynamic
{
public class NormalizeLogMeanVarianceFixZero
{
public static void Example()

@wschinwschinJun 27, 2019•
edited by artidoro
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have two independent cases in Example(...). Could you split them into two files? #Pending

@artidoroartidoroJun 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have followed the same pattern as the MeanVariance normalizer and LogMeanVariance normalizer sample. Would you prefer that I split them all into two files? Or should I just split this one? Or keep it like the others for consistency?

In reply to: 298257288 [](ancestors = 298257288)

@wschinwschinJun 29, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating them is more reader-friendly.

{
// Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
// as well as the source of randomness.
var mlContext = new MLContext();
var samples = new List<DataPoint>()
{
new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
};
// Convert training data to IDataView, the general data type used in ML.NET.
var data = mlContext.Data.LoadFromEnumerable(samples);
// NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
// Uses Cumulative distribution function as output.
var normalize = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: true);

// NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.

@wschinwschinJun 27, 2019•
edited by artidoro
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data. doesn't tell much more than the line below. We need some equations here.

( log(x) - Mean( Log(x) ) ) / Var( log(x) )

maybe? #Resolved

@artidoroartidoroJun 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that would be nice I can add that here.

In reply to: 298254677 [](ancestors = 298254677)

@artidoroartidoroJun 28, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I told you the other day I am still concerned about that equation. I am going to do a separate PR where I add the equations. (Same for the documentation comment in the catalog that you had).
There is no documentation on how to derive them so I am going through the math again and trying to understand the rational.

In reply to: 298311132 [](ancestors = 298311132,298254677)

var normalizeNoCdf = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: false);

// Now we can transform the data and look at the output to confirm the behavior of the estimator.
// This operation doesn't actually evaluate data until we read the data below.
var normalizeTransform = normalize.Fit(data);
var transformedData = normalizeTransform.Transform(data);
var normalizeNoCdfTransform = normalizeNoCdf.Fit(data);
var noCdfData = normalizeNoCdfTransform.Transform(data);
var column = transformedData.GetColumn<float[]>("Features").ToArray();
foreach (var row in column)
Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
// Expected output:
// 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
// 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
// 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
// 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
foreach (var row in columnFixZero)
Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
// Expected output:
// 2.0403, 2.0403, 4.0001, 0.0000, 5423991000000000000000000000000000000.0000
// 4.0806, 4.0806, 2.6667, 0.0000,-5423991000000000000000000000000000000.0000
// 0.0000, 0.0000, 1.3334, 0.0000, 0.0000
// -2.0403,-2.0403,-1.3334, 0.0000, 0.0159

// Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
// If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>;
Console.WriteLine("The values in the column with index 1 in the resulting array would be produced by:");
Console.WriteLine($"y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))");

@wschinwschinJun 27, 2019•
edited by artidoro
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have equation for columnFixZero now! May we have the same thing for column? #Resolved

@artidoroartidoroJun 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the equation from the LogMeanVariance normalizer sample :)
There is an equation for both settings of FixZero and for both Cdf and no Cdf. So I think it's complete.

In reply to: 298258190 [](ancestors = 298258190)


// ERF is https://en.wikipedia.org/wiki/Error_function.
// Expected output:
// The values in the column with index 1 in the resulting array would be produced by:
// y = 0.5 * (1 + ERF((Math.Log(x) - 0.3465736) / (0.3465736 * sqrt(2)))
var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
var scale = noCdfParams.Scale[1];
Console.WriteLine($"The values in the column with index 1 in the resulting array would be produced by: y = (x - ({offset})) * {scale}");
// Expected output:
// The values in the column with index 1 in the resulting array would be produced by: y = (x - (0)) * 2.040279
}

private class DataPoint
{
[VectorType(5)]
public float[] Features { get; set; }
}
}
}
4 changes: 2 additions & 2 deletions src/Microsoft.ML.Data/Transforms/NormalizeColumnSng.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1557,7 +1557,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
{
var lim = column.MaximumExampleCount;
host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
return new MeanVarOneColumnFunctionBuilder(host, lim, false, getter, true, column.UseCdf);
return new MeanVarOneColumnFunctionBuilder(host, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
}

protected override bool ProcessValue(in TFloat origVal)
Expand DownExpand Up@@ -1633,7 +1633,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
var lim = column.MaximumExampleCount;
host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
var cv = srcType.Size;
return new MeanVarVecColumnFunctionBuilder(host, cv, lim, false, getter, true, column.UseCdf);
return new MeanVarVecColumnFunctionBuilder(host, cv, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
}

protected override bool ProcessValue(in VBuffer<TFloat> buffer)
Expand Down
15 changes: 11 additions & 4 deletions src/Microsoft.ML.Data/Transforms/Normalizer.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -42,7 +42,7 @@ namespace Microsoft.ML.Transforms
/// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
/// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
/// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
/// * Log Mean Variance - Rescale each row to unit variancebased on a log scale.
/// * Log Mean Variance - Rescale each row to unit variance, optionally, zero mean based on computations in log scale.
/// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.
/// * Supervised Binning - Bucketize the data in each row and performas a linear rescale based on the calculated bins. The bin calculation is based on correlation of the Label column.
///
Expand All@@ -54,6 +54,13 @@ namespace Microsoft.ML.Transforms
/// With Min Max, the distribution depends on how far away the number is from 0, resulting in the number with the largest distance being mapped to 1 if its a positive number
/// or -1 if its a negative number. The distance from 0 will affect the distribution with a majority of numbers that are closer together normalizing towards 0.
///
/// The equation for the output $y$ of applying both Mean Variance and Log Mean Variance on input $x$ without
/// using the CDF option is: $y = (x - \text{offset}) \text{scale}$. Where offset and scale are computed during training.
///
/// Using the CDF option it is: $y = 0.5 * (1 + \text{ERF}((x - \text{mean}) / (\text{standard deviation} * sqrt(2)))$.
/// Where ERF is the [Error Function](https://en.wikipedia.org/wiki/Error_function) used to approximate the CDF of a random variable assumed to
/// normally distributed. The mean and standard deviation are computing during training.
///
/// To create this estimator use one of the following:
/// * [NormalizeMinMax](xref:Microsoft.ML.NormalizationCatalog.NormalizeMinMax(Microsoft.ML.TransformsCatalog, System.String, System.String, System.Int64, System.Boolean))
/// * [NormalizeMeanVariance](xref:Microsoft.ML.NormalizationCatalog.NormalizeMeanVariance(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Boolean))
Expand DownExpand Up@@ -183,13 +190,13 @@ internal override IColumnFunctionBuilder MakeBuilder(IHost host, int srcIndex, D
}

[BestFriend]
internal sealed class LogMeanVarianceColumnOptions : ColumnOptionsBase
internal sealed class LogMeanVarianceColumnOptions : ControlZeroColumnOptionsBase
{
public readonly bool UseCdf;

public LogMeanVarianceColumnOptions(string outputColumnName, string inputColumnName = null,
long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf)
: base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount)
long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf, bool fixZero = Defaults.EnsureZeroUntouched)
: base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount, fixZero)
{
UseCdf = useCdf;
}
Expand Down
51 changes: 49 additions & 2 deletions src/Microsoft.ML.Transforms/NormalizerCatalog.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -141,7 +141,7 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
{
var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf);
var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, false);
return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
}

Expand All@@ -159,7 +159,54 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
columns.Select(column =>
new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf)).ToArray());
new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, false)).ToArray());

/// <summary>
/// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
/// </summary>
/// <param name="catalog">The transform catalog</param>
/// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
/// The data type on this column is the same as the input column.</param>
/// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
/// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
/// The data type on this column should be <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.</param>
/// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
/// <param name="useCdf">Whether to use CDF as the output.</param>
/// <example>
/// <format type="text/markdown">

@wschinwschinJun 27, 2019•
edited by artidoro
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses would love equations in documentations. #Resolved

@artidoroartidoroJun 27, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the equations here as well.

In reply to: 298260450 [](ancestors = 298260450)

@artidoroartidoroJun 28, 2019•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the equations. Yesterday I think I saw an issue. I would like to make sure I understand all the steps and will add those in a separate PR and check this in for now.

In reply to: 298310313 [](ancestors = 298310313,298260450)

/// <![CDATA[
/// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs)]
/// ]]>
/// </format>
/// </example>
public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog,
string outputColumnName,
bool fixZero,
string inputColumnName = null,
long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
{
var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, fixZero);
return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
}

/// <summary>
/// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
/// </summary>
/// <param name="catalog">The transform catalog</param>
/// <param name="columns">The pairs of input and output columns.
/// The input columns must be of data type <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.
/// The data type for the output column will be the same as the associated input column.</param>
/// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
/// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
/// <param name="useCdf">Whether to use CDF as the output.</param>
public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog, InputOutputColumnPair[] columns,
bool fixZero,
long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
columns.Select(column =>
new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, fixZero)).ToArray());

/// <summary>
/// Create a <see cref="NormalizingEstimator"/>, which normalizes by assigning the data into bins with equal density.
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Add FixZero for LogMeanVariance normalizer#3916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

artidoro
merged 4 commits into

from

Jul 1, 2019
 Merged


Add FixZero for LogMeanVariance normalizer
#3916
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
9623019

add fixZero for LogMeanVariance normalizer


artidoro Jun 26, 2019
c829ad4

add test and sample reference in catalog


artidoro Jun 26, 2019
0527161

addressing review comments


artidoro Jun 28, 2019
52c19ed

adding the normalizer equation


artidoro Jul 1, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


32 changes: 16 additions & 16 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,15 +12,15 @@ public class NormalizeLogMeanVariance
 {
 public static void Example()
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} },
 new DataPoint(){ Features = new float[4] { 2, 2, 2, 0} },
 new DataPoint(){ Features = new float[4] { 0, 0, 1, 0} },
 new DataPoint(){ Features = new float[4] {-1,-1,-1, 1} }
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to FixZero? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's related to the original issue posted by Lisa. They were encountering an issue with the MeanVariance normalizer which does not handle data that looks like float.MaxValue, float.MinValue. I think it would be useful to show that LogMeanVariance normalizer accepts this data in the related sample.
In reply to: 298254220 [](ancestors = 298254220)
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
Expand All@@ -41,19 +41,19 @@ public static void Example()
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000
 // 0.8413, 0.8413, 0.5837, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 1.8854, 1.8854, 5.2970, 0.0000
 // 4.7708, 4.7708, 3.0925, 0.0000
 // -1.0000,-1.0000, 0.8879, 0.0000
 // 3.8854,-3.8854,-3.5213, 0.0000
 // 1.8854, 1.8854, 5.2970, 0.0000, 7670682000000000000000000000000000000.0000
 // 4.7708, 4.7708, 3.0925, 0.0000, -7670682000000000000000000000000000000.0000
 // -1.0000,-1.0000, 0.8879, 0.0000, -1.0000
 // -3.8854,-3.8854,-3.5213, 0.0000, -0.9775

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
Expand All@@ -64,18 +64,18 @@ public static void Example()
 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The 1-index value in resulting array would be produce by:
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The 1-index value in resulting array would be produce by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The 1-index value in resulting array would be produce by: y = (x - (2.88539)) * 0.3465736
 // The 1-index value in resulting array would be produce by: y = (x - (0.3465736)) * 2.88539
 }

 private class DataPoint
 {
 [VectorType(4)]
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
Expand Down

82 changes: 82 additions & 0 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.NormalizingTransformer;

namespace Samples.Dynamic
{
 public class NormalizeLogMeanVarianceFixZero
 {
 public static void Example()
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have two independent cases in Example(...). Could you split them into two files? #Pending
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have followed the same pattern as the MeanVariance normalizer and LogMeanVariance normalizer sample. Would you prefer that I split them all into two files? Or should I just split this one? Or keep it like the others for consistency?
In reply to: 298257288 [](ancestors = 298257288)
@wschinwschinJun 29, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating them is more reader-friendly.
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
 // Uses Cumulative distribution function as output.
 var normalize = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: true);

 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data. doesn't tell much more than the line below. We need some equations here.
( log(x) - Mean( Log(x) ) ) / Var( log(x) )

maybe? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that would be nice I can add that here.
In reply to: 298254677 [](ancestors = 298254677)
@artidoroartidoroJun 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I told you the other day I am still concerned about that equation. I am going to do a separate PR where I add the equations. (Same for the documentation comment in the catalog that you had).

There is no documentation on how to derive them so I am going through the math again and trying to understand the rational.
In reply to: 298311132 [](ancestors = 298311132,298254677)
 var normalizeNoCdf = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: false);

 // Now we can transform the data and look at the output to confirm the behavior of the estimator.
 // This operation doesn't actually evaluate data until we read the data below.
 var normalizeTransform = normalize.Fit(data);
 var transformedData = normalizeTransform.Transform(data);
 var normalizeNoCdfTransform = normalizeNoCdf.Fit(data);
 var noCdfData = normalizeNoCdfTransform.Transform(data);
 var column = transformedData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 2.0403, 2.0403, 4.0001, 0.0000, 5423991000000000000000000000000000000.0000
 // 4.0806, 4.0806, 2.6667, 0.0000,-5423991000000000000000000000000000000.0000
 // 0.0000, 0.0000, 1.3334, 0.0000, 0.0000
 // -2.0403,-2.0403,-1.3334, 0.0000, 0.0159

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
 var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>;
 Console.WriteLine("The values in the column with index 1 in the resulting array would be produced by:");
 Console.WriteLine($"y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))");
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have equation for columnFixZero now! May we have the same thing for column? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the equation from the LogMeanVariance normalizer sample :)

There is an equation for both settings of FixZero and for both Cdf and no Cdf. So I think it's complete.
In reply to: 298258190 [](ancestors = 298258190)

 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by:
 // y = 0.5 * (1 + ERF((Math.Log(x) - 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The values in the column with index 1 in the resulting array would be produced by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by: y = (x - (0)) * 2.040279
 }

 private class DataPoint
 {
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
}

4 changes: 2 additions & 2 deletions
src/Microsoft.ML.Data/Transforms/NormalizeColumnSng.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1557,7 +1557,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 {
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 return new MeanVarOneColumnFunctionBuilder(host, lim, false, getter, true, column.UseCdf);
 return new MeanVarOneColumnFunctionBuilder(host, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in TFloat origVal)
Expand DownExpand Up@@ -1633,7 +1633,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 var cv = srcType.Size;
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, false, getter, true, column.UseCdf);
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in VBuffer<TFloat> buffer)
Expand Down

15 changes: 11 additions & 4 deletions
src/Microsoft.ML.Data/Transforms/Normalizer.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,7 +42,7 @@ namespace Microsoft.ML.Transforms
 /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
 /// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
 /// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
 /// * Log Mean Variance - Rescale each row to unit variancebased on a log scale.
 /// * Log Mean Variance - Rescale each row to unit variance, optionally, zero mean based on computations in log scale.
 /// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.
 /// * Supervised Binning - Bucketize the data in each row and performas a linear rescale based on the calculated bins. The bin calculation is based on correlation of the Label column.
 ///
Expand All@@ -54,6 +54,13 @@ namespace Microsoft.ML.Transforms
 /// With Min Max, the distribution depends on how far away the number is from 0, resulting in the number with the largest distance being mapped to 1 if its a positive number
 /// or -1 if its a negative number. The distance from 0 will affect the distribution with a majority of numbers that are closer together normalizing towards 0.
 ///
 /// The equation for the output $y$ of applying both Mean Variance and Log Mean Variance on input $x$ without
 /// using the CDF option is: $y = (x - \text{offset}) \text{scale}$. Where offset and scale are computed during training.
 ///
 /// Using the CDF option it is: $y = 0.5 * (1 + \text{ERF}((x - \text{mean}) / (\text{standard deviation} * sqrt(2)))$.
 /// Where ERF is the [Error Function](https://en.wikipedia.org/wiki/Error_function) used to approximate the CDF of a random variable assumed to
 /// normally distributed. The mean and standard deviation are computing during training.
 ///
 /// To create this estimator use one of the following:
 /// * [NormalizeMinMax](xref:Microsoft.ML.NormalizationCatalog.NormalizeMinMax(Microsoft.ML.TransformsCatalog, System.String, System.String, System.Int64, System.Boolean))
 /// * [NormalizeMeanVariance](xref:Microsoft.ML.NormalizationCatalog.NormalizeMeanVariance(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Boolean))
Expand DownExpand Up@@ -183,13 +190,13 @@ internal override IColumnFunctionBuilder MakeBuilder(IHost host, int srcIndex, D
 }

 [BestFriend]
 internal sealed class LogMeanVarianceColumnOptions : ColumnOptionsBase
 internal sealed class LogMeanVarianceColumnOptions : ControlZeroColumnOptionsBase
 {
 public readonly bool UseCdf;

 public LogMeanVarianceColumnOptions(string outputColumnName, string inputColumnName = null,
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount)
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf, bool fixZero = Defaults.EnsureZeroUntouched)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount, fixZero)
 {
 UseCdf = useCdf;
 }
Expand Down

51 changes: 49 additions & 2 deletions
src/Microsoft.ML.Transforms/NormalizerCatalog.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -141,7 +141,7 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf);
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, false);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

Expand All@@ -159,7 +159,54 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf)).ToArray());
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, false)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
 /// The data type on this column is the same as the input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
 /// The data type on this column should be <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 /// <example>
 /// <format type="text/markdown">
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses would love equations in documentations. #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the equations here as well.
In reply to: 298260450 [](ancestors = 298260450)
@artidoroartidoroJun 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the equations. Yesterday I think I saw an issue. I would like to make sure I understand all the steps and will add those in a separate PR and check this in for now.
In reply to: 298310313 [](ancestors = 298310313,298260450)
 /// <![CDATA[
 /// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs)]
 /// ]]>
 /// </format>
 /// </example>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog,
 string outputColumnName,
 bool fixZero,
 string inputColumnName = null,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, fixZero);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="columns">The pairs of input and output columns.
 /// The input columns must be of data type <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.
 /// The data type for the output column will be the same as the associated input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog, InputOutputColumnPair[] columns,
 bool fixZero,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, fixZero)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes by assigning the data into bins with equal density.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add FixZero for LogMeanVariance normalizer#3916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

artidoro
merged 4 commits into

from

Jul 1, 2019
 Merged


Add FixZero for LogMeanVariance normalizer
#3916
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
9623019

add fixZero for LogMeanVariance normalizer


artidoro Jun 26, 2019
c829ad4

add test and sample reference in catalog


artidoro Jun 26, 2019
0527161

addressing review comments


artidoro Jun 28, 2019
52c19ed

adding the normalizer equation


artidoro Jul 1, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


32 changes: 16 additions & 16 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,15 +12,15 @@ public class NormalizeLogMeanVariance
 {
 public static void Example()
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} },
 new DataPoint(){ Features = new float[4] { 2, 2, 2, 0} },
 new DataPoint(){ Features = new float[4] { 0, 0, 1, 0} },
 new DataPoint(){ Features = new float[4] {-1,-1,-1, 1} }
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to FixZero? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's related to the original issue posted by Lisa. They were encountering an issue with the MeanVariance normalizer which does not handle data that looks like float.MaxValue, float.MinValue. I think it would be useful to show that LogMeanVariance normalizer accepts this data in the related sample.
In reply to: 298254220 [](ancestors = 298254220)
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
Expand All@@ -41,19 +41,19 @@ public static void Example()
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000
 // 0.8413, 0.8413, 0.5837, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 1.8854, 1.8854, 5.2970, 0.0000
 // 4.7708, 4.7708, 3.0925, 0.0000
 // -1.0000,-1.0000, 0.8879, 0.0000
 // 3.8854,-3.8854,-3.5213, 0.0000
 // 1.8854, 1.8854, 5.2970, 0.0000, 7670682000000000000000000000000000000.0000
 // 4.7708, 4.7708, 3.0925, 0.0000, -7670682000000000000000000000000000000.0000
 // -1.0000,-1.0000, 0.8879, 0.0000, -1.0000
 // -3.8854,-3.8854,-3.5213, 0.0000, -0.9775

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
Expand All@@ -64,18 +64,18 @@ public static void Example()
 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The 1-index value in resulting array would be produce by:
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The 1-index value in resulting array would be produce by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The 1-index value in resulting array would be produce by: y = (x - (2.88539)) * 0.3465736
 // The 1-index value in resulting array would be produce by: y = (x - (0.3465736)) * 2.88539
 }

 private class DataPoint
 {
 [VectorType(4)]
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
Expand Down

82 changes: 82 additions & 0 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.NormalizingTransformer;

namespace Samples.Dynamic
{
 public class NormalizeLogMeanVarianceFixZero
 {
 public static void Example()
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have two independent cases in Example(...). Could you split them into two files? #Pending
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have followed the same pattern as the MeanVariance normalizer and LogMeanVariance normalizer sample. Would you prefer that I split them all into two files? Or should I just split this one? Or keep it like the others for consistency?
In reply to: 298257288 [](ancestors = 298257288)
@wschinwschinJun 29, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating them is more reader-friendly.
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
 // Uses Cumulative distribution function as output.
 var normalize = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: true);

 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data. doesn't tell much more than the line below. We need some equations here.
( log(x) - Mean( Log(x) ) ) / Var( log(x) )

maybe? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that would be nice I can add that here.
In reply to: 298254677 [](ancestors = 298254677)
@artidoroartidoroJun 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I told you the other day I am still concerned about that equation. I am going to do a separate PR where I add the equations. (Same for the documentation comment in the catalog that you had).

There is no documentation on how to derive them so I am going through the math again and trying to understand the rational.
In reply to: 298311132 [](ancestors = 298311132,298254677)
 var normalizeNoCdf = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: false);

 // Now we can transform the data and look at the output to confirm the behavior of the estimator.
 // This operation doesn't actually evaluate data until we read the data below.
 var normalizeTransform = normalize.Fit(data);
 var transformedData = normalizeTransform.Transform(data);
 var normalizeNoCdfTransform = normalizeNoCdf.Fit(data);
 var noCdfData = normalizeNoCdfTransform.Transform(data);
 var column = transformedData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 2.0403, 2.0403, 4.0001, 0.0000, 5423991000000000000000000000000000000.0000
 // 4.0806, 4.0806, 2.6667, 0.0000,-5423991000000000000000000000000000000.0000
 // 0.0000, 0.0000, 1.3334, 0.0000, 0.0000
 // -2.0403,-2.0403,-1.3334, 0.0000, 0.0159

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
 var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>;
 Console.WriteLine("The values in the column with index 1 in the resulting array would be produced by:");
 Console.WriteLine($"y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))");
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have equation for columnFixZero now! May we have the same thing for column? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the equation from the LogMeanVariance normalizer sample :)

There is an equation for both settings of FixZero and for both Cdf and no Cdf. So I think it's complete.
In reply to: 298258190 [](ancestors = 298258190)

 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by:
 // y = 0.5 * (1 + ERF((Math.Log(x) - 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The values in the column with index 1 in the resulting array would be produced by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by: y = (x - (0)) * 2.040279
 }

 private class DataPoint
 {
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
}

4 changes: 2 additions & 2 deletions
src/Microsoft.ML.Data/Transforms/NormalizeColumnSng.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1557,7 +1557,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 {
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 return new MeanVarOneColumnFunctionBuilder(host, lim, false, getter, true, column.UseCdf);
 return new MeanVarOneColumnFunctionBuilder(host, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in TFloat origVal)
Expand DownExpand Up@@ -1633,7 +1633,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 var cv = srcType.Size;
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, false, getter, true, column.UseCdf);
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in VBuffer<TFloat> buffer)
Expand Down

15 changes: 11 additions & 4 deletions
src/Microsoft.ML.Data/Transforms/Normalizer.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,7 +42,7 @@ namespace Microsoft.ML.Transforms
 /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
 /// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
 /// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
 /// * Log Mean Variance - Rescale each row to unit variancebased on a log scale.
 /// * Log Mean Variance - Rescale each row to unit variance, optionally, zero mean based on computations in log scale.
 /// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.
 /// * Supervised Binning - Bucketize the data in each row and performas a linear rescale based on the calculated bins. The bin calculation is based on correlation of the Label column.
 ///
Expand All@@ -54,6 +54,13 @@ namespace Microsoft.ML.Transforms
 /// With Min Max, the distribution depends on how far away the number is from 0, resulting in the number with the largest distance being mapped to 1 if its a positive number
 /// or -1 if its a negative number. The distance from 0 will affect the distribution with a majority of numbers that are closer together normalizing towards 0.
 ///
 /// The equation for the output $y$ of applying both Mean Variance and Log Mean Variance on input $x$ without
 /// using the CDF option is: $y = (x - \text{offset}) \text{scale}$. Where offset and scale are computed during training.
 ///
 /// Using the CDF option it is: $y = 0.5 * (1 + \text{ERF}((x - \text{mean}) / (\text{standard deviation} * sqrt(2)))$.
 /// Where ERF is the [Error Function](https://en.wikipedia.org/wiki/Error_function) used to approximate the CDF of a random variable assumed to
 /// normally distributed. The mean and standard deviation are computing during training.
 ///
 /// To create this estimator use one of the following:
 /// * [NormalizeMinMax](xref:Microsoft.ML.NormalizationCatalog.NormalizeMinMax(Microsoft.ML.TransformsCatalog, System.String, System.String, System.Int64, System.Boolean))
 /// * [NormalizeMeanVariance](xref:Microsoft.ML.NormalizationCatalog.NormalizeMeanVariance(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Boolean))
Expand DownExpand Up@@ -183,13 +190,13 @@ internal override IColumnFunctionBuilder MakeBuilder(IHost host, int srcIndex, D
 }

 [BestFriend]
 internal sealed class LogMeanVarianceColumnOptions : ColumnOptionsBase
 internal sealed class LogMeanVarianceColumnOptions : ControlZeroColumnOptionsBase
 {
 public readonly bool UseCdf;

 public LogMeanVarianceColumnOptions(string outputColumnName, string inputColumnName = null,
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount)
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf, bool fixZero = Defaults.EnsureZeroUntouched)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount, fixZero)
 {
 UseCdf = useCdf;
 }
Expand Down

51 changes: 49 additions & 2 deletions
src/Microsoft.ML.Transforms/NormalizerCatalog.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -141,7 +141,7 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf);
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, false);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

Expand All@@ -159,7 +159,54 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf)).ToArray());
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, false)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
 /// The data type on this column is the same as the input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
 /// The data type on this column should be <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 /// <example>
 /// <format type="text/markdown">
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses would love equations in documentations. #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the equations here as well.
In reply to: 298260450 [](ancestors = 298260450)
@artidoroartidoroJun 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the equations. Yesterday I think I saw an issue. I would like to make sure I understand all the steps and will add those in a separate PR and check this in for now.
In reply to: 298310313 [](ancestors = 298310313,298260450)
 /// <![CDATA[
 /// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs)]
 /// ]]>
 /// </format>
 /// </example>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog,
 string outputColumnName,
 bool fixZero,
 string inputColumnName = null,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, fixZero);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="columns">The pairs of input and output columns.
 /// The input columns must be of data type <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.
 /// The data type for the output column will be the same as the associated input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog, InputOutputColumnPair[] columns,
 bool fixZero,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, fixZero)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes by assigning the data into bins with equal density.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add FixZero for LogMeanVariance normalizer#3916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

artidoro
merged 4 commits into

from

Jul 1, 2019
 Merged


Add FixZero for LogMeanVariance normalizer
#3916
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
9623019

add fixZero for LogMeanVariance normalizer


artidoro Jun 26, 2019
c829ad4

add test and sample reference in catalog


artidoro Jun 26, 2019
0527161

addressing review comments


artidoro Jun 28, 2019
52c19ed

adding the normalizer equation


artidoro Jul 1, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


32 changes: 16 additions & 16 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,15 +12,15 @@ public class NormalizeLogMeanVariance
 {
 public static void Example()
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} },
 new DataPoint(){ Features = new float[4] { 2, 2, 2, 0} },
 new DataPoint(){ Features = new float[4] { 0, 0, 1, 0} },
 new DataPoint(){ Features = new float[4] {-1,-1,-1, 1} }
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to FixZero? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's related to the original issue posted by Lisa. They were encountering an issue with the MeanVariance normalizer which does not handle data that looks like float.MaxValue, float.MinValue. I think it would be useful to show that LogMeanVariance normalizer accepts this data in the related sample.
In reply to: 298254220 [](ancestors = 298254220)
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
Expand All@@ -41,19 +41,19 @@ public static void Example()
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000
 // 0.8413, 0.8413, 0.5837, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 1.8854, 1.8854, 5.2970, 0.0000
 // 4.7708, 4.7708, 3.0925, 0.0000
 // -1.0000,-1.0000, 0.8879, 0.0000
 // 3.8854,-3.8854,-3.5213, 0.0000
 // 1.8854, 1.8854, 5.2970, 0.0000, 7670682000000000000000000000000000000.0000
 // 4.7708, 4.7708, 3.0925, 0.0000, -7670682000000000000000000000000000000.0000
 // -1.0000,-1.0000, 0.8879, 0.0000, -1.0000
 // -3.8854,-3.8854,-3.5213, 0.0000, -0.9775

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
Expand All@@ -64,18 +64,18 @@ public static void Example()
 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The 1-index value in resulting array would be produce by:
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The 1-index value in resulting array would be produce by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The 1-index value in resulting array would be produce by: y = (x - (2.88539)) * 0.3465736
 // The 1-index value in resulting array would be produce by: y = (x - (0.3465736)) * 2.88539
 }

 private class DataPoint
 {
 [VectorType(4)]
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
Expand Down

82 changes: 82 additions & 0 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.NormalizingTransformer;

namespace Samples.Dynamic
{
 public class NormalizeLogMeanVarianceFixZero
 {
 public static void Example()
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have two independent cases in Example(...). Could you split them into two files? #Pending
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have followed the same pattern as the MeanVariance normalizer and LogMeanVariance normalizer sample. Would you prefer that I split them all into two files? Or should I just split this one? Or keep it like the others for consistency?
In reply to: 298257288 [](ancestors = 298257288)
@wschinwschinJun 29, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating them is more reader-friendly.
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
 // Uses Cumulative distribution function as output.
 var normalize = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: true);

 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data. doesn't tell much more than the line below. We need some equations here.
( log(x) - Mean( Log(x) ) ) / Var( log(x) )

maybe? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that would be nice I can add that here.
In reply to: 298254677 [](ancestors = 298254677)
@artidoroartidoroJun 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I told you the other day I am still concerned about that equation. I am going to do a separate PR where I add the equations. (Same for the documentation comment in the catalog that you had).

There is no documentation on how to derive them so I am going through the math again and trying to understand the rational.
In reply to: 298311132 [](ancestors = 298311132,298254677)
 var normalizeNoCdf = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: false);

 // Now we can transform the data and look at the output to confirm the behavior of the estimator.
 // This operation doesn't actually evaluate data until we read the data below.
 var normalizeTransform = normalize.Fit(data);
 var transformedData = normalizeTransform.Transform(data);
 var normalizeNoCdfTransform = normalizeNoCdf.Fit(data);
 var noCdfData = normalizeNoCdfTransform.Transform(data);
 var column = transformedData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 2.0403, 2.0403, 4.0001, 0.0000, 5423991000000000000000000000000000000.0000
 // 4.0806, 4.0806, 2.6667, 0.0000,-5423991000000000000000000000000000000.0000
 // 0.0000, 0.0000, 1.3334, 0.0000, 0.0000
 // -2.0403,-2.0403,-1.3334, 0.0000, 0.0159

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
 var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>;
 Console.WriteLine("The values in the column with index 1 in the resulting array would be produced by:");
 Console.WriteLine($"y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))");
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have equation for columnFixZero now! May we have the same thing for column? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the equation from the LogMeanVariance normalizer sample :)

There is an equation for both settings of FixZero and for both Cdf and no Cdf. So I think it's complete.
In reply to: 298258190 [](ancestors = 298258190)

 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by:
 // y = 0.5 * (1 + ERF((Math.Log(x) - 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The values in the column with index 1 in the resulting array would be produced by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by: y = (x - (0)) * 2.040279
 }

 private class DataPoint
 {
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
}

4 changes: 2 additions & 2 deletions
src/Microsoft.ML.Data/Transforms/NormalizeColumnSng.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1557,7 +1557,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 {
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 return new MeanVarOneColumnFunctionBuilder(host, lim, false, getter, true, column.UseCdf);
 return new MeanVarOneColumnFunctionBuilder(host, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in TFloat origVal)
Expand DownExpand Up@@ -1633,7 +1633,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 var cv = srcType.Size;
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, false, getter, true, column.UseCdf);
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in VBuffer<TFloat> buffer)
Expand Down

15 changes: 11 additions & 4 deletions
src/Microsoft.ML.Data/Transforms/Normalizer.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,7 +42,7 @@ namespace Microsoft.ML.Transforms
 /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
 /// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
 /// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
 /// * Log Mean Variance - Rescale each row to unit variancebased on a log scale.
 /// * Log Mean Variance - Rescale each row to unit variance, optionally, zero mean based on computations in log scale.
 /// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.
 /// * Supervised Binning - Bucketize the data in each row and performas a linear rescale based on the calculated bins. The bin calculation is based on correlation of the Label column.
 ///
Expand All@@ -54,6 +54,13 @@ namespace Microsoft.ML.Transforms
 /// With Min Max, the distribution depends on how far away the number is from 0, resulting in the number with the largest distance being mapped to 1 if its a positive number
 /// or -1 if its a negative number. The distance from 0 will affect the distribution with a majority of numbers that are closer together normalizing towards 0.
 ///
 /// The equation for the output $y$ of applying both Mean Variance and Log Mean Variance on input $x$ without
 /// using the CDF option is: $y = (x - \text{offset}) \text{scale}$. Where offset and scale are computed during training.
 ///
 /// Using the CDF option it is: $y = 0.5 * (1 + \text{ERF}((x - \text{mean}) / (\text{standard deviation} * sqrt(2)))$.
 /// Where ERF is the [Error Function](https://en.wikipedia.org/wiki/Error_function) used to approximate the CDF of a random variable assumed to
 /// normally distributed. The mean and standard deviation are computing during training.
 ///
 /// To create this estimator use one of the following:
 /// * [NormalizeMinMax](xref:Microsoft.ML.NormalizationCatalog.NormalizeMinMax(Microsoft.ML.TransformsCatalog, System.String, System.String, System.Int64, System.Boolean))
 /// * [NormalizeMeanVariance](xref:Microsoft.ML.NormalizationCatalog.NormalizeMeanVariance(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Boolean))
Expand DownExpand Up@@ -183,13 +190,13 @@ internal override IColumnFunctionBuilder MakeBuilder(IHost host, int srcIndex, D
 }

 [BestFriend]
 internal sealed class LogMeanVarianceColumnOptions : ColumnOptionsBase
 internal sealed class LogMeanVarianceColumnOptions : ControlZeroColumnOptionsBase
 {
 public readonly bool UseCdf;

 public LogMeanVarianceColumnOptions(string outputColumnName, string inputColumnName = null,
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount)
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf, bool fixZero = Defaults.EnsureZeroUntouched)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount, fixZero)
 {
 UseCdf = useCdf;
 }
Expand Down

51 changes: 49 additions & 2 deletions
src/Microsoft.ML.Transforms/NormalizerCatalog.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -141,7 +141,7 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf);
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, false);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

Expand All@@ -159,7 +159,54 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf)).ToArray());
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, false)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
 /// The data type on this column is the same as the input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
 /// The data type on this column should be <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 /// <example>
 /// <format type="text/markdown">
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses would love equations in documentations. #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the equations here as well.
In reply to: 298260450 [](ancestors = 298260450)
@artidoroartidoroJun 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the equations. Yesterday I think I saw an issue. I would like to make sure I understand all the steps and will add those in a separate PR and check this in for now.
In reply to: 298310313 [](ancestors = 298310313,298260450)
 /// <![CDATA[
 /// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs)]
 /// ]]>
 /// </format>
 /// </example>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog,
 string outputColumnName,
 bool fixZero,
 string inputColumnName = null,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, fixZero);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="columns">The pairs of input and output columns.
 /// The input columns must be of data type <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.
 /// The data type for the output column will be the same as the associated input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog, InputOutputColumnPair[] columns,
 bool fixZero,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, fixZero)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes by assigning the data into bins with equal density.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Add FixZero for LogMeanVariance normalizer#3916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

artidoro
merged 4 commits into

from

Jul 1, 2019
 Merged


Add FixZero for LogMeanVariance normalizer
#3916
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
9623019

add fixZero for LogMeanVariance normalizer


artidoro Jun 26, 2019
c829ad4

add test and sample reference in catalog


artidoro Jun 26, 2019
0527161

addressing review comments


artidoro Jun 28, 2019
52c19ed

adding the normalizer equation


artidoro Jul 1, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


32 changes: 16 additions & 16 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,15 +12,15 @@ public class NormalizeLogMeanVariance
 {
 public static void Example()
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} },
 new DataPoint(){ Features = new float[4] { 2, 2, 2, 0} },
 new DataPoint(){ Features = new float[4] { 0, 0, 1, 0} },
 new DataPoint(){ Features = new float[4] {-1,-1,-1, 1} }
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to FixZero? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's related to the original issue posted by Lisa. They were encountering an issue with the MeanVariance normalizer which does not handle data that looks like float.MaxValue, float.MinValue. I think it would be useful to show that LogMeanVariance normalizer accepts this data in the related sample.
In reply to: 298254220 [](ancestors = 298254220)
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
Expand All@@ -41,19 +41,19 @@ public static void Example()
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000
 // 0.8413, 0.8413, 0.5837, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 1.8854, 1.8854, 5.2970, 0.0000
 // 4.7708, 4.7708, 3.0925, 0.0000
 // -1.0000,-1.0000, 0.8879, 0.0000
 // 3.8854,-3.8854,-3.5213, 0.0000
 // 1.8854, 1.8854, 5.2970, 0.0000, 7670682000000000000000000000000000000.0000
 // 4.7708, 4.7708, 3.0925, 0.0000, -7670682000000000000000000000000000000.0000
 // -1.0000,-1.0000, 0.8879, 0.0000, -1.0000
 // -3.8854,-3.8854,-3.5213, 0.0000, -0.9775

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
Expand All@@ -64,18 +64,18 @@ public static void Example()
 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The 1-index value in resulting array would be produce by:
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The 1-index value in resulting array would be produce by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The 1-index value in resulting array would be produce by: y = (x - (2.88539)) * 0.3465736
 // The 1-index value in resulting array would be produce by: y = (x - (0.3465736)) * 2.88539
 }

 private class DataPoint
 {
 [VectorType(4)]
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
Expand Down

82 changes: 82 additions & 0 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.NormalizingTransformer;

namespace Samples.Dynamic
{
 public class NormalizeLogMeanVarianceFixZero
 {
 public static void Example()
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have two independent cases in Example(...). Could you split them into two files? #Pending
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have followed the same pattern as the MeanVariance normalizer and LogMeanVariance normalizer sample. Would you prefer that I split them all into two files? Or should I just split this one? Or keep it like the others for consistency?
In reply to: 298257288 [](ancestors = 298257288)
@wschinwschinJun 29, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating them is more reader-friendly.
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
 // Uses Cumulative distribution function as output.
 var normalize = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: true);

 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data. doesn't tell much more than the line below. We need some equations here.
( log(x) - Mean( Log(x) ) ) / Var( log(x) )

maybe? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that would be nice I can add that here.
In reply to: 298254677 [](ancestors = 298254677)
@artidoroartidoroJun 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I told you the other day I am still concerned about that equation. I am going to do a separate PR where I add the equations. (Same for the documentation comment in the catalog that you had).

There is no documentation on how to derive them so I am going through the math again and trying to understand the rational.
In reply to: 298311132 [](ancestors = 298311132,298254677)
 var normalizeNoCdf = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: false);

 // Now we can transform the data and look at the output to confirm the behavior of the estimator.
 // This operation doesn't actually evaluate data until we read the data below.
 var normalizeTransform = normalize.Fit(data);
 var transformedData = normalizeTransform.Transform(data);
 var normalizeNoCdfTransform = normalizeNoCdf.Fit(data);
 var noCdfData = normalizeNoCdfTransform.Transform(data);
 var column = transformedData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 2.0403, 2.0403, 4.0001, 0.0000, 5423991000000000000000000000000000000.0000
 // 4.0806, 4.0806, 2.6667, 0.0000,-5423991000000000000000000000000000000.0000
 // 0.0000, 0.0000, 1.3334, 0.0000, 0.0000
 // -2.0403,-2.0403,-1.3334, 0.0000, 0.0159

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
 var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>;
 Console.WriteLine("The values in the column with index 1 in the resulting array would be produced by:");
 Console.WriteLine($"y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))");
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have equation for columnFixZero now! May we have the same thing for column? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the equation from the LogMeanVariance normalizer sample :)

There is an equation for both settings of FixZero and for both Cdf and no Cdf. So I think it's complete.
In reply to: 298258190 [](ancestors = 298258190)

 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by:
 // y = 0.5 * (1 + ERF((Math.Log(x) - 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The values in the column with index 1 in the resulting array would be produced by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by: y = (x - (0)) * 2.040279
 }

 private class DataPoint
 {
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
}

4 changes: 2 additions & 2 deletions
src/Microsoft.ML.Data/Transforms/NormalizeColumnSng.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1557,7 +1557,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 {
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 return new MeanVarOneColumnFunctionBuilder(host, lim, false, getter, true, column.UseCdf);
 return new MeanVarOneColumnFunctionBuilder(host, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in TFloat origVal)
Expand DownExpand Up@@ -1633,7 +1633,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 var cv = srcType.Size;
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, false, getter, true, column.UseCdf);
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in VBuffer<TFloat> buffer)
Expand Down

15 changes: 11 additions & 4 deletions
src/Microsoft.ML.Data/Transforms/Normalizer.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,7 +42,7 @@ namespace Microsoft.ML.Transforms
 /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
 /// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
 /// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
 /// * Log Mean Variance - Rescale each row to unit variancebased on a log scale.
 /// * Log Mean Variance - Rescale each row to unit variance, optionally, zero mean based on computations in log scale.
 /// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.
 /// * Supervised Binning - Bucketize the data in each row and performas a linear rescale based on the calculated bins. The bin calculation is based on correlation of the Label column.
 ///
Expand All@@ -54,6 +54,13 @@ namespace Microsoft.ML.Transforms
 /// With Min Max, the distribution depends on how far away the number is from 0, resulting in the number with the largest distance being mapped to 1 if its a positive number
 /// or -1 if its a negative number. The distance from 0 will affect the distribution with a majority of numbers that are closer together normalizing towards 0.
 ///
 /// The equation for the output $y$ of applying both Mean Variance and Log Mean Variance on input $x$ without
 /// using the CDF option is: $y = (x - \text{offset}) \text{scale}$. Where offset and scale are computed during training.
 ///
 /// Using the CDF option it is: $y = 0.5 * (1 + \text{ERF}((x - \text{mean}) / (\text{standard deviation} * sqrt(2)))$.
 /// Where ERF is the [Error Function](https://en.wikipedia.org/wiki/Error_function) used to approximate the CDF of a random variable assumed to
 /// normally distributed. The mean and standard deviation are computing during training.
 ///
 /// To create this estimator use one of the following:
 /// * [NormalizeMinMax](xref:Microsoft.ML.NormalizationCatalog.NormalizeMinMax(Microsoft.ML.TransformsCatalog, System.String, System.String, System.Int64, System.Boolean))
 /// * [NormalizeMeanVariance](xref:Microsoft.ML.NormalizationCatalog.NormalizeMeanVariance(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Boolean))
Expand DownExpand Up@@ -183,13 +190,13 @@ internal override IColumnFunctionBuilder MakeBuilder(IHost host, int srcIndex, D
 }

 [BestFriend]
 internal sealed class LogMeanVarianceColumnOptions : ColumnOptionsBase
 internal sealed class LogMeanVarianceColumnOptions : ControlZeroColumnOptionsBase
 {
 public readonly bool UseCdf;

 public LogMeanVarianceColumnOptions(string outputColumnName, string inputColumnName = null,
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount)
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf, bool fixZero = Defaults.EnsureZeroUntouched)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount, fixZero)
 {
 UseCdf = useCdf;
 }
Expand Down

51 changes: 49 additions & 2 deletions
src/Microsoft.ML.Transforms/NormalizerCatalog.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -141,7 +141,7 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf);
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, false);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

Expand All@@ -159,7 +159,54 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf)).ToArray());
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, false)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
 /// The data type on this column is the same as the input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
 /// The data type on this column should be <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 /// <example>
 /// <format type="text/markdown">
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses would love equations in documentations. #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the equations here as well.
In reply to: 298260450 [](ancestors = 298260450)
@artidoroartidoroJun 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the equations. Yesterday I think I saw an issue. I would like to make sure I understand all the steps and will add those in a separate PR and check this in for now.
In reply to: 298310313 [](ancestors = 298310313,298260450)
 /// <![CDATA[
 /// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs)]
 /// ]]>
 /// </format>
 /// </example>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog,
 string outputColumnName,
 bool fixZero,
 string inputColumnName = null,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, fixZero);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="columns">The pairs of input and output columns.
 /// The input columns must be of data type <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.
 /// The data type for the output column will be the same as the associated input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog, InputOutputColumnPair[] columns,
 bool fixZero,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, fixZero)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes by assigning the data into bins with equal density.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add FixZero for LogMeanVariance normalizer#3916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

artidoro
merged 4 commits into

from

Jul 1, 2019
 Merged


Add FixZero for LogMeanVariance normalizer
#3916
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
9623019

add fixZero for LogMeanVariance normalizer


artidoro Jun 26, 2019
c829ad4

add test and sample reference in catalog


artidoro Jun 26, 2019
0527161

addressing review comments


artidoro Jun 28, 2019
52c19ed

adding the normalizer equation


artidoro Jul 1, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


32 changes: 16 additions & 16 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,15 +12,15 @@ public class NormalizeLogMeanVariance
 {
 public static void Example()
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} },
 new DataPoint(){ Features = new float[4] { 2, 2, 2, 0} },
 new DataPoint(){ Features = new float[4] { 0, 0, 1, 0} },
 new DataPoint(){ Features = new float[4] {-1,-1,-1, 1} }
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to FixZero? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's related to the original issue posted by Lisa. They were encountering an issue with the MeanVariance normalizer which does not handle data that looks like float.MaxValue, float.MinValue. I think it would be useful to show that LogMeanVariance normalizer accepts this data in the related sample.
In reply to: 298254220 [](ancestors = 298254220)
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
Expand All@@ -41,19 +41,19 @@ public static void Example()
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000
 // 0.8413, 0.8413, 0.5837, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 1.8854, 1.8854, 5.2970, 0.0000
 // 4.7708, 4.7708, 3.0925, 0.0000
 // -1.0000,-1.0000, 0.8879, 0.0000
 // 3.8854,-3.8854,-3.5213, 0.0000
 // 1.8854, 1.8854, 5.2970, 0.0000, 7670682000000000000000000000000000000.0000
 // 4.7708, 4.7708, 3.0925, 0.0000, -7670682000000000000000000000000000000.0000
 // -1.0000,-1.0000, 0.8879, 0.0000, -1.0000
 // -3.8854,-3.8854,-3.5213, 0.0000, -0.9775

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
Expand All@@ -64,18 +64,18 @@ public static void Example()
 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The 1-index value in resulting array would be produce by:
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The 1-index value in resulting array would be produce by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The 1-index value in resulting array would be produce by: y = (x - (2.88539)) * 0.3465736
 // The 1-index value in resulting array would be produce by: y = (x - (0.3465736)) * 2.88539
 }

 private class DataPoint
 {
 [VectorType(4)]
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
Expand Down

82 changes: 82 additions & 0 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.NormalizingTransformer;

namespace Samples.Dynamic
{
 public class NormalizeLogMeanVarianceFixZero
 {
 public static void Example()
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have two independent cases in Example(...). Could you split them into two files? #Pending
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have followed the same pattern as the MeanVariance normalizer and LogMeanVariance normalizer sample. Would you prefer that I split them all into two files? Or should I just split this one? Or keep it like the others for consistency?
In reply to: 298257288 [](ancestors = 298257288)
@wschinwschinJun 29, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating them is more reader-friendly.
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
 // Uses Cumulative distribution function as output.
 var normalize = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: true);

 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data. doesn't tell much more than the line below. We need some equations here.
( log(x) - Mean( Log(x) ) ) / Var( log(x) )

maybe? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that would be nice I can add that here.
In reply to: 298254677 [](ancestors = 298254677)
@artidoroartidoroJun 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I told you the other day I am still concerned about that equation. I am going to do a separate PR where I add the equations. (Same for the documentation comment in the catalog that you had).

There is no documentation on how to derive them so I am going through the math again and trying to understand the rational.
In reply to: 298311132 [](ancestors = 298311132,298254677)
 var normalizeNoCdf = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: false);

 // Now we can transform the data and look at the output to confirm the behavior of the estimator.
 // This operation doesn't actually evaluate data until we read the data below.
 var normalizeTransform = normalize.Fit(data);
 var transformedData = normalizeTransform.Transform(data);
 var normalizeNoCdfTransform = normalizeNoCdf.Fit(data);
 var noCdfData = normalizeNoCdfTransform.Transform(data);
 var column = transformedData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 2.0403, 2.0403, 4.0001, 0.0000, 5423991000000000000000000000000000000.0000
 // 4.0806, 4.0806, 2.6667, 0.0000,-5423991000000000000000000000000000000.0000
 // 0.0000, 0.0000, 1.3334, 0.0000, 0.0000
 // -2.0403,-2.0403,-1.3334, 0.0000, 0.0159

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
 var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>;
 Console.WriteLine("The values in the column with index 1 in the resulting array would be produced by:");
 Console.WriteLine($"y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))");
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have equation for columnFixZero now! May we have the same thing for column? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the equation from the LogMeanVariance normalizer sample :)

There is an equation for both settings of FixZero and for both Cdf and no Cdf. So I think it's complete.
In reply to: 298258190 [](ancestors = 298258190)

 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by:
 // y = 0.5 * (1 + ERF((Math.Log(x) - 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The values in the column with index 1 in the resulting array would be produced by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by: y = (x - (0)) * 2.040279
 }

 private class DataPoint
 {
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
}

4 changes: 2 additions & 2 deletions
src/Microsoft.ML.Data/Transforms/NormalizeColumnSng.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1557,7 +1557,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 {
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 return new MeanVarOneColumnFunctionBuilder(host, lim, false, getter, true, column.UseCdf);
 return new MeanVarOneColumnFunctionBuilder(host, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in TFloat origVal)
Expand DownExpand Up@@ -1633,7 +1633,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 var cv = srcType.Size;
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, false, getter, true, column.UseCdf);
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in VBuffer<TFloat> buffer)
Expand Down

15 changes: 11 additions & 4 deletions
src/Microsoft.ML.Data/Transforms/Normalizer.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,7 +42,7 @@ namespace Microsoft.ML.Transforms
 /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
 /// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
 /// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
 /// * Log Mean Variance - Rescale each row to unit variancebased on a log scale.
 /// * Log Mean Variance - Rescale each row to unit variance, optionally, zero mean based on computations in log scale.
 /// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.
 /// * Supervised Binning - Bucketize the data in each row and performas a linear rescale based on the calculated bins. The bin calculation is based on correlation of the Label column.
 ///
Expand All@@ -54,6 +54,13 @@ namespace Microsoft.ML.Transforms
 /// With Min Max, the distribution depends on how far away the number is from 0, resulting in the number with the largest distance being mapped to 1 if its a positive number
 /// or -1 if its a negative number. The distance from 0 will affect the distribution with a majority of numbers that are closer together normalizing towards 0.
 ///
 /// The equation for the output $y$ of applying both Mean Variance and Log Mean Variance on input $x$ without
 /// using the CDF option is: $y = (x - \text{offset}) \text{scale}$. Where offset and scale are computed during training.
 ///
 /// Using the CDF option it is: $y = 0.5 * (1 + \text{ERF}((x - \text{mean}) / (\text{standard deviation} * sqrt(2)))$.
 /// Where ERF is the [Error Function](https://en.wikipedia.org/wiki/Error_function) used to approximate the CDF of a random variable assumed to
 /// normally distributed. The mean and standard deviation are computing during training.
 ///
 /// To create this estimator use one of the following:
 /// * [NormalizeMinMax](xref:Microsoft.ML.NormalizationCatalog.NormalizeMinMax(Microsoft.ML.TransformsCatalog, System.String, System.String, System.Int64, System.Boolean))
 /// * [NormalizeMeanVariance](xref:Microsoft.ML.NormalizationCatalog.NormalizeMeanVariance(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Boolean))
Expand DownExpand Up@@ -183,13 +190,13 @@ internal override IColumnFunctionBuilder MakeBuilder(IHost host, int srcIndex, D
 }

 [BestFriend]
 internal sealed class LogMeanVarianceColumnOptions : ColumnOptionsBase
 internal sealed class LogMeanVarianceColumnOptions : ControlZeroColumnOptionsBase
 {
 public readonly bool UseCdf;

 public LogMeanVarianceColumnOptions(string outputColumnName, string inputColumnName = null,
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount)
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf, bool fixZero = Defaults.EnsureZeroUntouched)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount, fixZero)
 {
 UseCdf = useCdf;
 }
Expand Down

51 changes: 49 additions & 2 deletions
src/Microsoft.ML.Transforms/NormalizerCatalog.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -141,7 +141,7 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf);
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, false);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

Expand All@@ -159,7 +159,54 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf)).ToArray());
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, false)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
 /// The data type on this column is the same as the input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
 /// The data type on this column should be <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 /// <example>
 /// <format type="text/markdown">
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses would love equations in documentations. #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the equations here as well.
In reply to: 298260450 [](ancestors = 298260450)
@artidoroartidoroJun 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the equations. Yesterday I think I saw an issue. I would like to make sure I understand all the steps and will add those in a separate PR and check this in for now.
In reply to: 298310313 [](ancestors = 298310313,298260450)
 /// <![CDATA[
 /// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs)]
 /// ]]>
 /// </format>
 /// </example>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog,
 string outputColumnName,
 bool fixZero,
 string inputColumnName = null,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, fixZero);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="columns">The pairs of input and output columns.
 /// The input columns must be of data type <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.
 /// The data type for the output column will be the same as the associated input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog, InputOutputColumnPair[] columns,
 bool fixZero,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, fixZero)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes by assigning the data into bins with equal density.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add FixZero for LogMeanVariance normalizer#3916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

artidoro
merged 4 commits into

from

Jul 1, 2019
 Merged


Add FixZero for LogMeanVariance normalizer
#3916
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
9623019

add fixZero for LogMeanVariance normalizer


artidoro Jun 26, 2019
c829ad4

add test and sample reference in catalog


artidoro Jun 26, 2019
0527161

addressing review comments


artidoro Jun 28, 2019
52c19ed

adding the normalizer equation


artidoro Jul 1, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


32 changes: 16 additions & 16 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,15 +12,15 @@ public class NormalizeLogMeanVariance
 {
 public static void Example()
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} },
 new DataPoint(){ Features = new float[4] { 2, 2, 2, 0} },
 new DataPoint(){ Features = new float[4] { 0, 0, 1, 0} },
 new DataPoint(){ Features = new float[4] {-1,-1,-1, 1} }
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to FixZero? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's related to the original issue posted by Lisa. They were encountering an issue with the MeanVariance normalizer which does not handle data that looks like float.MaxValue, float.MinValue. I think it would be useful to show that LogMeanVariance normalizer accepts this data in the related sample.
In reply to: 298254220 [](ancestors = 298254220)
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
Expand All@@ -41,19 +41,19 @@ public static void Example()
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000
 // 0.8413, 0.8413, 0.5837, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 1.8854, 1.8854, 5.2970, 0.0000
 // 4.7708, 4.7708, 3.0925, 0.0000
 // -1.0000,-1.0000, 0.8879, 0.0000
 // 3.8854,-3.8854,-3.5213, 0.0000
 // 1.8854, 1.8854, 5.2970, 0.0000, 7670682000000000000000000000000000000.0000
 // 4.7708, 4.7708, 3.0925, 0.0000, -7670682000000000000000000000000000000.0000
 // -1.0000,-1.0000, 0.8879, 0.0000, -1.0000
 // -3.8854,-3.8854,-3.5213, 0.0000, -0.9775

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
Expand All@@ -64,18 +64,18 @@ public static void Example()
 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The 1-index value in resulting array would be produce by:
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The 1-index value in resulting array would be produce by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The 1-index value in resulting array would be produce by: y = (x - (2.88539)) * 0.3465736
 // The 1-index value in resulting array would be produce by: y = (x - (0.3465736)) * 2.88539
 }

 private class DataPoint
 {
 [VectorType(4)]
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
Expand Down

82 changes: 82 additions & 0 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.NormalizingTransformer;

namespace Samples.Dynamic
{
 public class NormalizeLogMeanVarianceFixZero
 {
 public static void Example()
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have two independent cases in Example(...). Could you split them into two files? #Pending
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have followed the same pattern as the MeanVariance normalizer and LogMeanVariance normalizer sample. Would you prefer that I split them all into two files? Or should I just split this one? Or keep it like the others for consistency?
In reply to: 298257288 [](ancestors = 298257288)
@wschinwschinJun 29, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating them is more reader-friendly.
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
 // Uses Cumulative distribution function as output.
 var normalize = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: true);

 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data. doesn't tell much more than the line below. We need some equations here.
( log(x) - Mean( Log(x) ) ) / Var( log(x) )

maybe? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that would be nice I can add that here.
In reply to: 298254677 [](ancestors = 298254677)
@artidoroartidoroJun 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I told you the other day I am still concerned about that equation. I am going to do a separate PR where I add the equations. (Same for the documentation comment in the catalog that you had).

There is no documentation on how to derive them so I am going through the math again and trying to understand the rational.
In reply to: 298311132 [](ancestors = 298311132,298254677)
 var normalizeNoCdf = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: false);

 // Now we can transform the data and look at the output to confirm the behavior of the estimator.
 // This operation doesn't actually evaluate data until we read the data below.
 var normalizeTransform = normalize.Fit(data);
 var transformedData = normalizeTransform.Transform(data);
 var normalizeNoCdfTransform = normalizeNoCdf.Fit(data);
 var noCdfData = normalizeNoCdfTransform.Transform(data);
 var column = transformedData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 2.0403, 2.0403, 4.0001, 0.0000, 5423991000000000000000000000000000000.0000
 // 4.0806, 4.0806, 2.6667, 0.0000,-5423991000000000000000000000000000000.0000
 // 0.0000, 0.0000, 1.3334, 0.0000, 0.0000
 // -2.0403,-2.0403,-1.3334, 0.0000, 0.0159

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
 var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>;
 Console.WriteLine("The values in the column with index 1 in the resulting array would be produced by:");
 Console.WriteLine($"y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))");
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have equation for columnFixZero now! May we have the same thing for column? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the equation from the LogMeanVariance normalizer sample :)

There is an equation for both settings of FixZero and for both Cdf and no Cdf. So I think it's complete.
In reply to: 298258190 [](ancestors = 298258190)

 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by:
 // y = 0.5 * (1 + ERF((Math.Log(x) - 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The values in the column with index 1 in the resulting array would be produced by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by: y = (x - (0)) * 2.040279
 }

 private class DataPoint
 {
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
}

4 changes: 2 additions & 2 deletions
src/Microsoft.ML.Data/Transforms/NormalizeColumnSng.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1557,7 +1557,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 {
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 return new MeanVarOneColumnFunctionBuilder(host, lim, false, getter, true, column.UseCdf);
 return new MeanVarOneColumnFunctionBuilder(host, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in TFloat origVal)
Expand DownExpand Up@@ -1633,7 +1633,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 var cv = srcType.Size;
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, false, getter, true, column.UseCdf);
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in VBuffer<TFloat> buffer)
Expand Down

15 changes: 11 additions & 4 deletions
src/Microsoft.ML.Data/Transforms/Normalizer.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,7 +42,7 @@ namespace Microsoft.ML.Transforms
 /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
 /// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
 /// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
 /// * Log Mean Variance - Rescale each row to unit variancebased on a log scale.
 /// * Log Mean Variance - Rescale each row to unit variance, optionally, zero mean based on computations in log scale.
 /// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.
 /// * Supervised Binning - Bucketize the data in each row and performas a linear rescale based on the calculated bins. The bin calculation is based on correlation of the Label column.
 ///
Expand All@@ -54,6 +54,13 @@ namespace Microsoft.ML.Transforms
 /// With Min Max, the distribution depends on how far away the number is from 0, resulting in the number with the largest distance being mapped to 1 if its a positive number
 /// or -1 if its a negative number. The distance from 0 will affect the distribution with a majority of numbers that are closer together normalizing towards 0.
 ///
 /// The equation for the output $y$ of applying both Mean Variance and Log Mean Variance on input $x$ without
 /// using the CDF option is: $y = (x - \text{offset}) \text{scale}$. Where offset and scale are computed during training.
 ///
 /// Using the CDF option it is: $y = 0.5 * (1 + \text{ERF}((x - \text{mean}) / (\text{standard deviation} * sqrt(2)))$.
 /// Where ERF is the [Error Function](https://en.wikipedia.org/wiki/Error_function) used to approximate the CDF of a random variable assumed to
 /// normally distributed. The mean and standard deviation are computing during training.
 ///
 /// To create this estimator use one of the following:
 /// * [NormalizeMinMax](xref:Microsoft.ML.NormalizationCatalog.NormalizeMinMax(Microsoft.ML.TransformsCatalog, System.String, System.String, System.Int64, System.Boolean))
 /// * [NormalizeMeanVariance](xref:Microsoft.ML.NormalizationCatalog.NormalizeMeanVariance(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Boolean))
Expand DownExpand Up@@ -183,13 +190,13 @@ internal override IColumnFunctionBuilder MakeBuilder(IHost host, int srcIndex, D
 }

 [BestFriend]
 internal sealed class LogMeanVarianceColumnOptions : ColumnOptionsBase
 internal sealed class LogMeanVarianceColumnOptions : ControlZeroColumnOptionsBase
 {
 public readonly bool UseCdf;

 public LogMeanVarianceColumnOptions(string outputColumnName, string inputColumnName = null,
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount)
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf, bool fixZero = Defaults.EnsureZeroUntouched)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount, fixZero)
 {
 UseCdf = useCdf;
 }
Expand Down

51 changes: 49 additions & 2 deletions
src/Microsoft.ML.Transforms/NormalizerCatalog.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -141,7 +141,7 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf);
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, false);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

Expand All@@ -159,7 +159,54 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf)).ToArray());
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, false)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
 /// The data type on this column is the same as the input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
 /// The data type on this column should be <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 /// <example>
 /// <format type="text/markdown">
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses would love equations in documentations. #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the equations here as well.
In reply to: 298260450 [](ancestors = 298260450)
@artidoroartidoroJun 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the equations. Yesterday I think I saw an issue. I would like to make sure I understand all the steps and will add those in a separate PR and check this in for now.
In reply to: 298310313 [](ancestors = 298310313,298260450)
 /// <![CDATA[
 /// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs)]
 /// ]]>
 /// </format>
 /// </example>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog,
 string outputColumnName,
 bool fixZero,
 string inputColumnName = null,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, fixZero);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="columns">The pairs of input and output columns.
 /// The input columns must be of data type <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.
 /// The data type for the output column will be the same as the associated input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog, InputOutputColumnPair[] columns,
 bool fixZero,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, fixZero)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes by assigning the data into bins with equal density.
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Add FixZero for LogMeanVariance normalizer#3916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

artidoro
merged 4 commits into

from

Jul 1, 2019
 Merged


Add FixZero for LogMeanVariance normalizer
#3916
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


4 commits


Select commit
Hold shift + click to select a range
9623019

add fixZero for LogMeanVariance normalizer


artidoro Jun 26, 2019
c829ad4

add test and sample reference in catalog


artidoro Jun 26, 2019
0527161

addressing review comments


artidoro Jun 28, 2019
52c19ed

adding the normalizer equation


artidoro Jul 1, 2019

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


32 changes: 16 additions & 16 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVariance.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,15 +12,15 @@ public class NormalizeLogMeanVariance
 {
 public static void Example()
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[4] { 1, 1, 3, 0} },
 new DataPoint(){ Features = new float[4] { 2, 2, 2, 0} },
 new DataPoint(){ Features = new float[4] { 0, 0, 1, 0} },
 new DataPoint(){ Features = new float[4] {-1,-1,-1, 1} }
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change related to FixZero? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's related to the original issue posted by Lisa. They were encountering an issue with the MeanVariance normalizer which does not handle data that looks like float.MaxValue, float.MinValue. I think it would be useful to show that LogMeanVariance normalizer accepts this data in the related sample.
In reply to: 298254220 [](ancestors = 298254220)
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
Expand All@@ -41,19 +41,19 @@ public static void Example()
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000
 // 0.8413, 0.8413, 0.5837, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 1.8854, 1.8854, 5.2970, 0.0000
 // 4.7708, 4.7708, 3.0925, 0.0000
 // -1.0000,-1.0000, 0.8879, 0.0000
 // 3.8854,-3.8854,-3.5213, 0.0000
 // 1.8854, 1.8854, 5.2970, 0.0000, 7670682000000000000000000000000000000.0000
 // 4.7708, 4.7708, 3.0925, 0.0000, -7670682000000000000000000000000000000.0000
 // -1.0000,-1.0000, 0.8879, 0.0000, -1.0000
 // -3.8854,-3.8854,-3.5213, 0.0000, -0.9775

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
Expand All@@ -64,18 +64,18 @@ public static void Example()
 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The 1-index value in resulting array would be produce by:
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 // y = 0.5* (1 + ERF((Math.Log(x)- 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The 1-index value in resulting array would be produce by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The 1-index value in resulting array would be produce by: y = (x - (2.88539)) * 0.3465736
 // The 1-index value in resulting array would be produce by: y = (x - (0.3465736)) * 2.88539
 }

 private class DataPoint
 {
 [VectorType(4)]
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
Expand Down

82 changes: 82 additions & 0 deletions
docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
using System;
using System.Collections.Generic;
using System.Collections.Immutable;
using System.Linq;
using Microsoft.ML;
using Microsoft.ML.Data;
using static Microsoft.ML.Transforms.NormalizingTransformer;

namespace Samples.Dynamic
{
 public class NormalizeLogMeanVarianceFixZero
 {
 public static void Example()
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have two independent cases in Example(...). Could you split them into two files? #Pending
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have followed the same pattern as the MeanVariance normalizer and LogMeanVariance normalizer sample. Would you prefer that I split them all into two files? Or should I just split this one? Or keep it like the others for consistency?
In reply to: 298257288 [](ancestors = 298257288)
@wschinwschinJun 29, 2019

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Separating them is more reader-friendly.
 {
 // Create a new ML context, for ML.NET operations. It can be used for exception tracking and logging,
 // as well as the source of randomness.
 var mlContext = new MLContext();
 var samples = new List<DataPoint>()
 {
 new DataPoint(){ Features = new float[5] { 1, 1, 3, 0, float.MaxValue } },
 new DataPoint(){ Features = new float[5] { 2, 2, 2, 0, float.MinValue } },
 new DataPoint(){ Features = new float[5] { 0, 0, 1, 0, 0} },
 new DataPoint(){ Features = new float[5] {-1,-1,-1, 1, 1} }
 };
 // Convert training data to IDataView, the general data type used in ML.NET.
 var data = mlContext.Data.LoadFromEnumerable(samples);
 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
 // Uses Cumulative distribution function as output.
 var normalize = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: true);

 // NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data.
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NormalizeLogMeanVariance normalizes the data based on the computed mean and variance of the logarithm of the data. doesn't tell much more than the line below. We need some equations here.
( log(x) - Mean( Log(x) ) ) / Var( log(x) )

maybe? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes that would be nice I can add that here.
In reply to: 298254677 [](ancestors = 298254677)
@artidoroartidoroJun 28, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I told you the other day I am still concerned about that equation. I am going to do a separate PR where I add the equations. (Same for the documentation comment in the catalog that you had).

There is no documentation on how to derive them so I am going through the math again and trying to understand the rational.
In reply to: 298311132 [](ancestors = 298311132,298254677)
 var normalizeNoCdf = mlContext.Transforms.NormalizeLogMeanVariance("Features", true, useCdf: false);

 // Now we can transform the data and look at the output to confirm the behavior of the estimator.
 // This operation doesn't actually evaluate data until we read the data below.
 var normalizeTransform = normalize.Fit(data);
 var transformedData = normalizeTransform.Transform(data);
 var normalizeNoCdfTransform = normalizeNoCdf.Fit(data);
 var noCdfData = normalizeNoCdfTransform.Transform(data);
 var column = transformedData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in column)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 0.1587, 0.1587, 0.8654, 0.0000, 0.8413
 // 0.8413, 0.8413, 0.5837, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0940, 0.0000, 0.0000
 // 0.0000, 0.0000, 0.0000, 0.0000, 0.1587

 var columnFixZero = noCdfData.GetColumn<float[]>("Features").ToArray();
 foreach (var row in columnFixZero)
 Console.WriteLine(string.Join(", ", row.Select(x => x.ToString("f4"))));
 // Expected output:
 // 2.0403, 2.0403, 4.0001, 0.0000, 5423991000000000000000000000000000000.0000
 // 4.0806, 4.0806, 2.6667, 0.0000,-5423991000000000000000000000000000000.0000
 // 0.0000, 0.0000, 1.3334, 0.0000, 0.0000
 // -2.0403,-2.0403,-1.3334, 0.0000, 0.0159

 // Let's get transformation parameters. Since we work with only one column we need to pass 0 as parameter for GetNormalizerModelParameters.
 // If we have multiple columns transformations we need to pass index of InputOutputColumnPair.
 var transformParams = normalizeTransform.GetNormalizerModelParameters(0) as CdfNormalizerModelParameters<ImmutableArray<float>>;
 Console.WriteLine("The values in the column with index 1 in the resulting array would be produced by:");
 Console.WriteLine($"y = 0.5* (1 + ERF((Math.Log(x)- {transformParams.Mean[1]}) / ({transformParams.StandardDeviation[1]} * sqrt(2)))");
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have equation for columnFixZero now! May we have the same thing for column? #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just copied the equation from the LogMeanVariance normalizer sample :)

There is an equation for both settings of FixZero and for both Cdf and no Cdf. So I think it's complete.
In reply to: 298258190 [](ancestors = 298258190)

 // ERF is https://en.wikipedia.org/wiki/Error_function.
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by:
 // y = 0.5 * (1 + ERF((Math.Log(x) - 0.3465736) / (0.3465736 * sqrt(2)))
 var noCdfParams = normalizeNoCdfTransform.GetNormalizerModelParameters(0) as AffineNormalizerModelParameters<ImmutableArray<float>>;
 var offset = noCdfParams.Offset.Length == 0 ? 0 : noCdfParams.Offset[1];
 var scale = noCdfParams.Scale[1];
 Console.WriteLine($"The values in the column with index 1 in the resulting array would be produced by: y = (x - ({offset})) * {scale}");
 // Expected output:
 // The values in the column with index 1 in the resulting array would be produced by: y = (x - (0)) * 2.040279
 }

 private class DataPoint
 {
 [VectorType(5)]
 public float[] Features { get; set; }
 }
 }
}

4 changes: 2 additions & 2 deletions
src/Microsoft.ML.Data/Transforms/NormalizeColumnSng.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1557,7 +1557,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 {
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 return new MeanVarOneColumnFunctionBuilder(host, lim, false, getter, true, column.UseCdf);
 return new MeanVarOneColumnFunctionBuilder(host, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in TFloat origVal)
Expand DownExpand Up@@ -1633,7 +1633,7 @@ public static IColumnFunctionBuilder Create(NormalizingEstimator.LogMeanVariance
 var lim = column.MaximumExampleCount;
 host.CheckUserArg(lim > 1, nameof(column.MaximumExampleCount), "Must be greater than 1");
 var cv = srcType.Size;
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, false, getter, true, column.UseCdf);
 return new MeanVarVecColumnFunctionBuilder(host, cv, lim, column.EnsureZeroUntouched, getter, true, column.UseCdf);
 }

 protected override bool ProcessValue(in VBuffer<TFloat> buffer)
Expand Down

15 changes: 11 additions & 4 deletions
src/Microsoft.ML.Data/Transforms/Normalizer.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -42,7 +42,7 @@ namespace Microsoft.ML.Transforms
 /// The resulting NormalizingEstimator will normalize the data in one of the following ways based upon how it was created:
 /// * Min Max - A linear rescale that is based upon the minimum and maximum values for each row.
 /// * Mean Variance - Rescale each row to unit variance and, optionally, zero mean.
 /// * Log Mean Variance - Rescale each row to unit variancebased on a log scale.
 /// * Log Mean Variance - Rescale each row to unit variance, optionally, zero mean based on computations in log scale.
 /// * Binning - Bucketizes the data in each row and performs a linear rescale based on the calculated bins.
 /// * Supervised Binning - Bucketize the data in each row and performas a linear rescale based on the calculated bins. The bin calculation is based on correlation of the Label column.
 ///
Expand All@@ -54,6 +54,13 @@ namespace Microsoft.ML.Transforms
 /// With Min Max, the distribution depends on how far away the number is from 0, resulting in the number with the largest distance being mapped to 1 if its a positive number
 /// or -1 if its a negative number. The distance from 0 will affect the distribution with a majority of numbers that are closer together normalizing towards 0.
 ///
 /// The equation for the output $y$ of applying both Mean Variance and Log Mean Variance on input $x$ without
 /// using the CDF option is: $y = (x - \text{offset}) \text{scale}$. Where offset and scale are computed during training.
 ///
 /// Using the CDF option it is: $y = 0.5 * (1 + \text{ERF}((x - \text{mean}) / (\text{standard deviation} * sqrt(2)))$.
 /// Where ERF is the [Error Function](https://en.wikipedia.org/wiki/Error_function) used to approximate the CDF of a random variable assumed to
 /// normally distributed. The mean and standard deviation are computing during training.
 ///
 /// To create this estimator use one of the following:
 /// * [NormalizeMinMax](xref:Microsoft.ML.NormalizationCatalog.NormalizeMinMax(Microsoft.ML.TransformsCatalog, System.String, System.String, System.Int64, System.Boolean))
 /// * [NormalizeMeanVariance](xref:Microsoft.ML.NormalizationCatalog.NormalizeMeanVariance(Microsoft.ML.TransformsCatalog,System.String,System.String,System.Int64,System.Boolean,System.Boolean))
Expand DownExpand Up@@ -183,13 +190,13 @@ internal override IColumnFunctionBuilder MakeBuilder(IHost host, int srcIndex, D
 }

 [BestFriend]
 internal sealed class LogMeanVarianceColumnOptions : ColumnOptionsBase
 internal sealed class LogMeanVarianceColumnOptions : ControlZeroColumnOptionsBase
 {
 public readonly bool UseCdf;

 public LogMeanVarianceColumnOptions(string outputColumnName, string inputColumnName = null,
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount)
 long maximumExampleCount = Defaults.MaximumExampleCount, bool useCdf = Defaults.LogMeanVarCdf, bool fixZero = Defaults.EnsureZeroUntouched)
 : base(outputColumnName, inputColumnName ?? outputColumnName, maximumExampleCount, fixZero)
 {
 UseCdf = useCdf;
 }
Expand Down

51 changes: 49 additions & 2 deletions
src/Microsoft.ML.Transforms/NormalizerCatalog.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -141,7 +141,7 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf);
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, false);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

Expand All@@ -159,7 +159,54 @@ public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatal
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf)).ToArray());
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, false)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="outputColumnName">Name of the column resulting from the transformation of <paramref name="inputColumnName"/>.
 /// The data type on this column is the same as the input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="inputColumnName">Name of the column to transform. If set to <see langword="null"/>, the value of the <paramref name="outputColumnName"/> will be used as source.
 /// The data type on this column should be <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 /// <example>
 /// <format type="text/markdown">
@wschinwschinJun 27, 2019•

edited by artidoro

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Uses would love equations in documentations. #Resolved
@artidoroartidoroJun 27, 2019

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the equations here as well.
In reply to: 298260450 [](ancestors = 298260450)
@artidoroartidoroJun 28, 2019•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am a little concerned about the equations. Yesterday I think I saw an issue. I would like to make sure I understand all the steps and will add those in a separate PR and check this in for now.
In reply to: 298310313 [](ancestors = 298310313,298260450)
 /// <![CDATA[
 /// [!code-csharp[NormalizeLogMeanVariance](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/NormalizeLogMeanVarianceFixZero.cs)]
 /// ]]>
 /// </format>
 /// </example>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog,
 string outputColumnName,
 bool fixZero,
 string inputColumnName = null,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf)
 {
 var columnOptions = new NormalizingEstimator.LogMeanVarianceColumnOptions(outputColumnName, inputColumnName, maximumExampleCount, useCdf, fixZero);
 return new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog), columnOptions);
 }

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes based on the computed mean and variance of the logarithm of the data.
 /// </summary>
 /// <param name="catalog">The transform catalog</param>
 /// <param name="columns">The pairs of input and output columns.
 /// The input columns must be of data type <see cref="System.Single"/>, <see cref="System.Double"/> or a known-sized vector of those types.
 /// The data type for the output column will be the same as the associated input column.</param>
 /// <param name="fixZero">Whether to map zero to zero, preserving sparsity.</param>
 /// <param name="maximumExampleCount">Maximum number of examples used to train the normalizer.</param>
 /// <param name="useCdf">Whether to use CDF as the output.</param>
 public static NormalizingEstimator NormalizeLogMeanVariance(this TransformsCatalog catalog, InputOutputColumnPair[] columns,
 bool fixZero,
 long maximumExampleCount = NormalizingEstimator.Defaults.MaximumExampleCount,
 bool useCdf = NormalizingEstimator.Defaults.LogMeanVarCdf) =>
 new NormalizingEstimator(CatalogUtils.GetEnvironment(catalog),
 columns.Select(column =>
 new NormalizingEstimator.LogMeanVarianceColumnOptions(column.OutputColumnName, column.InputColumnName, maximumExampleCount, useCdf, fixZero)).ToArray());

 /// <summary>
 /// Create a <see cref="NormalizingEstimator"/>, which normalizes by assigning the data into bins with equal density.
Expand Down
Loading