Multi column MapKeyToValue and MapValueToKey - #3187

Merged
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples
Apr 7, 2019
Merged

Multi column MapKeyToValue and MapValueToKey#3187
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples

Conversation

@sfilipi

Copy link
Copy Markdown
Member

Towards #1209 more samples for MapKeyToValue and MapValueToKey

@sfilipisfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019
@sfilipisfilipi self-assigned this Apr 3, 2019
@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
- Coverage 72.54% 72.53% -0.01% 
==========================================
Files 807 807 Lines 144774 144774 Branches 16208 16208 ==========================================
- Hits 105021 105017 -4 - Misses 35339 35343 +4 
Partials 4414 4414
FlagCoverage Δ
#Debug72.53% <ø> (-0.01%)⬇️
#production68.12% <ø> (-0.01%)⬇️
#test88.82% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs44.87% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs89.26% <0%> (-0.63%)⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs86.26% <0%> (+0.15%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs85.11% <0%> (+0.4%)⬆️

@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
+ Coverage 72.6% 72.62% +0.01% 
==========================================
Files 807 807 Lines 145077 145080 +3 Branches 16213 16213 ==========================================
+ Hits 105337 105366 +29 + Misses 35322 35296 -26 
Partials 4418 4418
FlagCoverage Δ
#Debug72.62% <ø> (+0.01%)⬆️
#production68.17% <ø> (+0.02%)⬆️
#test88.92% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs64.07% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs78.87% <0%> (ø)⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs70.39% <0%> (ø)⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs86.21% <0%> (ø)⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs97.31% <0%> (+0.05%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs84.9% <0%> (+0.2%)⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs91.12% <0%> (+1.02%)⬆️
...LogisticRegression/MulticlassLogisticRegression.cs67.61% <0%> (+1.74%)⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs92.34% <0%> (+3.27%)⬆️
... and 1 more


// at this point, the Label colum is tranformed from strings, to DataViewKeyType and
// the transformation has added the PredictedLabel column, with
var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.


In reply to: 271831815 [](ancestors = 271831815)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question


In reply to: 271866086 [](ancestors = 271866086,271831815)

new LookupMap { Key = "6-11yrs" },
new LookupMap { Key = "25+yrs" }

};

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}; [](start = 12, length = 2)

whitespace #Resolved

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public class MapKeyToValueInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public static class MapValueToKeyInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i had them on the same file, at some point :)


In reply to: 271927396 [](ancestors = 271927396)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs


In reply to: 271930498 [](ancestors = 271930498,271927396)

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating it.


In reply to: 271959976 [](ancestors = 271959976,271935868)

@abgoswamabgoswam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

using Microsoft.ML.SamplesUtils;
using Microsoft.ML.SamplesUtils;

namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought every class had its own distinctive name?


In reply to: 271977575 [](ancestors = 271977575)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, we call them the same between tasks. Reverting.


In reply to: 272288927 [](ancestors = 272288927,271977575)

var mlContext = new MLContext(seed: 0);

// Create a list of data examples.
var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have inline data like the other sample? #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.


In reply to: 271977941 [](ancestors = 271977941)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gani's PR is not checked in. I can switch the Generate after he checks in.


In reply to: 272332287 [](ancestors = 272332287,271977941)


// Get a small dataset as an IEnumerable.
var rawData = new[] {
new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.


In reply to: 271978738 [](ancestors = 271978738)

@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from 4171b5b to a5758c3CompareApril 4, 2019 18:12
@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from a5758c3 to 4e6c4afCompareApril 5, 2019 15:45

// TransformedData obtained post-transformation.
//
// StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 65, length = 15)

Course #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@shmoradimsshmoradimsApr 5, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

@shmoradimsshmoradims left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sfilipi
sfilipi merged commit 7a0d395 into dotnet:masterApr 7, 2019
@sfilipi
sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11
sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019
* Multi column MapKeyToValue and MapValueToKey
@ghostghost locked as resolved and limited conversation to collaborators Mar 23, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentationRelated to documentation of ML.NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sfilipi@abgoswam@shmoradims
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Multi column MapKeyToValue and MapValueToKey - #3187

Merged
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples
Apr 7, 2019
Merged

Multi column MapKeyToValue and MapValueToKey#3187
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples

Conversation

@sfilipi

Copy link
Copy Markdown
Member

Towards #1209 more samples for MapKeyToValue and MapValueToKey

@sfilipisfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019
@sfilipisfilipi self-assigned this Apr 3, 2019
@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
- Coverage 72.54% 72.53% -0.01% 
==========================================
Files 807 807 Lines 144774 144774 Branches 16208 16208 ==========================================
- Hits 105021 105017 -4 - Misses 35339 35343 +4 
Partials 4414 4414
FlagCoverage Δ
#Debug72.53% <ø> (-0.01%)⬇️
#production68.12% <ø> (-0.01%)⬇️
#test88.82% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs44.87% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs89.26% <0%> (-0.63%)⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs86.26% <0%> (+0.15%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs85.11% <0%> (+0.4%)⬆️

@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
+ Coverage 72.6% 72.62% +0.01% 
==========================================
Files 807 807 Lines 145077 145080 +3 Branches 16213 16213 ==========================================
+ Hits 105337 105366 +29 + Misses 35322 35296 -26 
Partials 4418 4418
FlagCoverage Δ
#Debug72.62% <ø> (+0.01%)⬆️
#production68.17% <ø> (+0.02%)⬆️
#test88.92% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs64.07% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs78.87% <0%> (ø)⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs70.39% <0%> (ø)⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs86.21% <0%> (ø)⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs97.31% <0%> (+0.05%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs84.9% <0%> (+0.2%)⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs91.12% <0%> (+1.02%)⬆️
...LogisticRegression/MulticlassLogisticRegression.cs67.61% <0%> (+1.74%)⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs92.34% <0%> (+3.27%)⬆️
... and 1 more


// at this point, the Label colum is tranformed from strings, to DataViewKeyType and
// the transformation has added the PredictedLabel column, with
var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.


In reply to: 271831815 [](ancestors = 271831815)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question


In reply to: 271866086 [](ancestors = 271866086,271831815)

new LookupMap { Key = "6-11yrs" },
new LookupMap { Key = "25+yrs" }

};

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}; [](start = 12, length = 2)

whitespace #Resolved

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public class MapKeyToValueInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public static class MapValueToKeyInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i had them on the same file, at some point :)


In reply to: 271927396 [](ancestors = 271927396)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs


In reply to: 271930498 [](ancestors = 271930498,271927396)

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating it.


In reply to: 271959976 [](ancestors = 271959976,271935868)

@abgoswamabgoswam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

using Microsoft.ML.SamplesUtils;
using Microsoft.ML.SamplesUtils;

namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought every class had its own distinctive name?


In reply to: 271977575 [](ancestors = 271977575)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, we call them the same between tasks. Reverting.


In reply to: 272288927 [](ancestors = 272288927,271977575)

var mlContext = new MLContext(seed: 0);

// Create a list of data examples.
var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have inline data like the other sample? #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.


In reply to: 271977941 [](ancestors = 271977941)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gani's PR is not checked in. I can switch the Generate after he checks in.


In reply to: 272332287 [](ancestors = 272332287,271977941)


// Get a small dataset as an IEnumerable.
var rawData = new[] {
new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.


In reply to: 271978738 [](ancestors = 271978738)

@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from 4171b5b to a5758c3CompareApril 4, 2019 18:12
@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from a5758c3 to 4e6c4afCompareApril 5, 2019 15:45

// TransformedData obtained post-transformation.
//
// StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 65, length = 15)

Course #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@shmoradimsshmoradimsApr 5, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

@shmoradimsshmoradims left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sfilipi
sfilipi merged commit 7a0d395 into dotnet:masterApr 7, 2019
@sfilipi
sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11
sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019
* Multi column MapKeyToValue and MapValueToKey
@ghostghost locked as resolved and limited conversation to collaborators Mar 23, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentationRelated to documentation of ML.NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sfilipi@abgoswam@shmoradims
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Multi column MapKeyToValue and MapValueToKey - #3187

Merged
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples
Apr 7, 2019
Merged

Multi column MapKeyToValue and MapValueToKey#3187
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples

Conversation

@sfilipi

Copy link
Copy Markdown
Member

Towards #1209 more samples for MapKeyToValue and MapValueToKey

@sfilipisfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019
@sfilipisfilipi self-assigned this Apr 3, 2019
@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
- Coverage 72.54% 72.53% -0.01% 
==========================================
Files 807 807 Lines 144774 144774 Branches 16208 16208 ==========================================
- Hits 105021 105017 -4 - Misses 35339 35343 +4 
Partials 4414 4414
FlagCoverage Δ
#Debug72.53% <ø> (-0.01%)⬇️
#production68.12% <ø> (-0.01%)⬇️
#test88.82% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs44.87% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs89.26% <0%> (-0.63%)⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs86.26% <0%> (+0.15%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs85.11% <0%> (+0.4%)⬆️

@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
+ Coverage 72.6% 72.62% +0.01% 
==========================================
Files 807 807 Lines 145077 145080 +3 Branches 16213 16213 ==========================================
+ Hits 105337 105366 +29 + Misses 35322 35296 -26 
Partials 4418 4418
FlagCoverage Δ
#Debug72.62% <ø> (+0.01%)⬆️
#production68.17% <ø> (+0.02%)⬆️
#test88.92% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs64.07% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs78.87% <0%> (ø)⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs70.39% <0%> (ø)⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs86.21% <0%> (ø)⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs97.31% <0%> (+0.05%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs84.9% <0%> (+0.2%)⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs91.12% <0%> (+1.02%)⬆️
...LogisticRegression/MulticlassLogisticRegression.cs67.61% <0%> (+1.74%)⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs92.34% <0%> (+3.27%)⬆️
... and 1 more


// at this point, the Label colum is tranformed from strings, to DataViewKeyType and
// the transformation has added the PredictedLabel column, with
var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.


In reply to: 271831815 [](ancestors = 271831815)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question


In reply to: 271866086 [](ancestors = 271866086,271831815)

new LookupMap { Key = "6-11yrs" },
new LookupMap { Key = "25+yrs" }

};

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}; [](start = 12, length = 2)

whitespace #Resolved

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public class MapKeyToValueInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public static class MapValueToKeyInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i had them on the same file, at some point :)


In reply to: 271927396 [](ancestors = 271927396)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs


In reply to: 271930498 [](ancestors = 271930498,271927396)

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating it.


In reply to: 271959976 [](ancestors = 271959976,271935868)

@abgoswamabgoswam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

using Microsoft.ML.SamplesUtils;
using Microsoft.ML.SamplesUtils;

namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought every class had its own distinctive name?


In reply to: 271977575 [](ancestors = 271977575)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, we call them the same between tasks. Reverting.


In reply to: 272288927 [](ancestors = 272288927,271977575)

var mlContext = new MLContext(seed: 0);

// Create a list of data examples.
var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have inline data like the other sample? #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.


In reply to: 271977941 [](ancestors = 271977941)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gani's PR is not checked in. I can switch the Generate after he checks in.


In reply to: 272332287 [](ancestors = 272332287,271977941)


// Get a small dataset as an IEnumerable.
var rawData = new[] {
new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.


In reply to: 271978738 [](ancestors = 271978738)

@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from 4171b5b to a5758c3CompareApril 4, 2019 18:12
@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from a5758c3 to 4e6c4afCompareApril 5, 2019 15:45

// TransformedData obtained post-transformation.
//
// StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 65, length = 15)

Course #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@shmoradimsshmoradimsApr 5, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

@shmoradimsshmoradims left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sfilipi
sfilipi merged commit 7a0d395 into dotnet:masterApr 7, 2019
@sfilipi
sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11
sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019
* Multi column MapKeyToValue and MapValueToKey
@ghostghost locked as resolved and limited conversation to collaborators Mar 23, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentationRelated to documentation of ML.NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sfilipi@abgoswam@shmoradims
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Multi column MapKeyToValue and MapValueToKey - #3187

Merged
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples
Apr 7, 2019
Merged

Multi column MapKeyToValue and MapValueToKey#3187
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples

Conversation

@sfilipi

Copy link
Copy Markdown
Member

Towards #1209 more samples for MapKeyToValue and MapValueToKey

@sfilipisfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019
@sfilipisfilipi self-assigned this Apr 3, 2019
@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
- Coverage 72.54% 72.53% -0.01% 
==========================================
Files 807 807 Lines 144774 144774 Branches 16208 16208 ==========================================
- Hits 105021 105017 -4 - Misses 35339 35343 +4 
Partials 4414 4414
FlagCoverage Δ
#Debug72.53% <ø> (-0.01%)⬇️
#production68.12% <ø> (-0.01%)⬇️
#test88.82% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs44.87% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs89.26% <0%> (-0.63%)⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs86.26% <0%> (+0.15%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs85.11% <0%> (+0.4%)⬆️

@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
+ Coverage 72.6% 72.62% +0.01% 
==========================================
Files 807 807 Lines 145077 145080 +3 Branches 16213 16213 ==========================================
+ Hits 105337 105366 +29 + Misses 35322 35296 -26 
Partials 4418 4418
FlagCoverage Δ
#Debug72.62% <ø> (+0.01%)⬆️
#production68.17% <ø> (+0.02%)⬆️
#test88.92% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs64.07% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs78.87% <0%> (ø)⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs70.39% <0%> (ø)⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs86.21% <0%> (ø)⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs97.31% <0%> (+0.05%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs84.9% <0%> (+0.2%)⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs91.12% <0%> (+1.02%)⬆️
...LogisticRegression/MulticlassLogisticRegression.cs67.61% <0%> (+1.74%)⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs92.34% <0%> (+3.27%)⬆️
... and 1 more


// at this point, the Label colum is tranformed from strings, to DataViewKeyType and
// the transformation has added the PredictedLabel column, with
var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.


In reply to: 271831815 [](ancestors = 271831815)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question


In reply to: 271866086 [](ancestors = 271866086,271831815)

new LookupMap { Key = "6-11yrs" },
new LookupMap { Key = "25+yrs" }

};

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}; [](start = 12, length = 2)

whitespace #Resolved

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public class MapKeyToValueInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public static class MapValueToKeyInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i had them on the same file, at some point :)


In reply to: 271927396 [](ancestors = 271927396)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs


In reply to: 271930498 [](ancestors = 271930498,271927396)

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating it.


In reply to: 271959976 [](ancestors = 271959976,271935868)

@abgoswamabgoswam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

using Microsoft.ML.SamplesUtils;
using Microsoft.ML.SamplesUtils;

namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought every class had its own distinctive name?


In reply to: 271977575 [](ancestors = 271977575)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, we call them the same between tasks. Reverting.


In reply to: 272288927 [](ancestors = 272288927,271977575)

var mlContext = new MLContext(seed: 0);

// Create a list of data examples.
var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have inline data like the other sample? #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.


In reply to: 271977941 [](ancestors = 271977941)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gani's PR is not checked in. I can switch the Generate after he checks in.


In reply to: 272332287 [](ancestors = 272332287,271977941)


// Get a small dataset as an IEnumerable.
var rawData = new[] {
new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.


In reply to: 271978738 [](ancestors = 271978738)

@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from 4171b5b to a5758c3CompareApril 4, 2019 18:12
@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from a5758c3 to 4e6c4afCompareApril 5, 2019 15:45

// TransformedData obtained post-transformation.
//
// StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 65, length = 15)

Course #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@shmoradimsshmoradimsApr 5, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

@shmoradimsshmoradims left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sfilipi
sfilipi merged commit 7a0d395 into dotnet:masterApr 7, 2019
@sfilipi
sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11
sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019
* Multi column MapKeyToValue and MapValueToKey
@ghostghost locked as resolved and limited conversation to collaborators Mar 23, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentationRelated to documentation of ML.NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sfilipi@abgoswam@shmoradims
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Multi column MapKeyToValue and MapValueToKey - #3187

Merged
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples
Apr 7, 2019
Merged

Multi column MapKeyToValue and MapValueToKey#3187
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples

Conversation

@sfilipi

Copy link
Copy Markdown
Member

Towards #1209 more samples for MapKeyToValue and MapValueToKey

@sfilipisfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019
@sfilipisfilipi self-assigned this Apr 3, 2019
@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
- Coverage 72.54% 72.53% -0.01% 
==========================================
Files 807 807 Lines 144774 144774 Branches 16208 16208 ==========================================
- Hits 105021 105017 -4 - Misses 35339 35343 +4 
Partials 4414 4414
FlagCoverage Δ
#Debug72.53% <ø> (-0.01%)⬇️
#production68.12% <ø> (-0.01%)⬇️
#test88.82% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs44.87% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs89.26% <0%> (-0.63%)⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs86.26% <0%> (+0.15%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs85.11% <0%> (+0.4%)⬆️

@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
+ Coverage 72.6% 72.62% +0.01% 
==========================================
Files 807 807 Lines 145077 145080 +3 Branches 16213 16213 ==========================================
+ Hits 105337 105366 +29 + Misses 35322 35296 -26 
Partials 4418 4418
FlagCoverage Δ
#Debug72.62% <ø> (+0.01%)⬆️
#production68.17% <ø> (+0.02%)⬆️
#test88.92% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs64.07% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs78.87% <0%> (ø)⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs70.39% <0%> (ø)⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs86.21% <0%> (ø)⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs97.31% <0%> (+0.05%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs84.9% <0%> (+0.2%)⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs91.12% <0%> (+1.02%)⬆️
...LogisticRegression/MulticlassLogisticRegression.cs67.61% <0%> (+1.74%)⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs92.34% <0%> (+3.27%)⬆️
... and 1 more


// at this point, the Label colum is tranformed from strings, to DataViewKeyType and
// the transformation has added the PredictedLabel column, with
var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.


In reply to: 271831815 [](ancestors = 271831815)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question


In reply to: 271866086 [](ancestors = 271866086,271831815)

new LookupMap { Key = "6-11yrs" },
new LookupMap { Key = "25+yrs" }

};

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}; [](start = 12, length = 2)

whitespace #Resolved

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public class MapKeyToValueInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public static class MapValueToKeyInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i had them on the same file, at some point :)


In reply to: 271927396 [](ancestors = 271927396)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs


In reply to: 271930498 [](ancestors = 271930498,271927396)

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating it.


In reply to: 271959976 [](ancestors = 271959976,271935868)

@abgoswamabgoswam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

using Microsoft.ML.SamplesUtils;
using Microsoft.ML.SamplesUtils;

namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought every class had its own distinctive name?


In reply to: 271977575 [](ancestors = 271977575)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, we call them the same between tasks. Reverting.


In reply to: 272288927 [](ancestors = 272288927,271977575)

var mlContext = new MLContext(seed: 0);

// Create a list of data examples.
var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have inline data like the other sample? #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.


In reply to: 271977941 [](ancestors = 271977941)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gani's PR is not checked in. I can switch the Generate after he checks in.


In reply to: 272332287 [](ancestors = 272332287,271977941)


// Get a small dataset as an IEnumerable.
var rawData = new[] {
new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.


In reply to: 271978738 [](ancestors = 271978738)

@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from 4171b5b to a5758c3CompareApril 4, 2019 18:12
@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from a5758c3 to 4e6c4afCompareApril 5, 2019 15:45

// TransformedData obtained post-transformation.
//
// StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 65, length = 15)

Course #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@shmoradimsshmoradimsApr 5, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

@shmoradimsshmoradims left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sfilipi
sfilipi merged commit 7a0d395 into dotnet:masterApr 7, 2019
@sfilipi
sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11
sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019
* Multi column MapKeyToValue and MapValueToKey
@ghostghost locked as resolved and limited conversation to collaborators Mar 23, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentationRelated to documentation of ML.NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sfilipi@abgoswam@shmoradims
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Multi column MapKeyToValue and MapValueToKey - #3187

Merged
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples
Apr 7, 2019
Merged

Multi column MapKeyToValue and MapValueToKey#3187
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples

Conversation

@sfilipi

Copy link
Copy Markdown
Member

Towards #1209 more samples for MapKeyToValue and MapValueToKey

@sfilipisfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019
@sfilipisfilipi self-assigned this Apr 3, 2019
@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
- Coverage 72.54% 72.53% -0.01% 
==========================================
Files 807 807 Lines 144774 144774 Branches 16208 16208 ==========================================
- Hits 105021 105017 -4 - Misses 35339 35343 +4 
Partials 4414 4414
FlagCoverage Δ
#Debug72.53% <ø> (-0.01%)⬇️
#production68.12% <ø> (-0.01%)⬇️
#test88.82% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs44.87% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs89.26% <0%> (-0.63%)⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs86.26% <0%> (+0.15%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs85.11% <0%> (+0.4%)⬆️

@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
+ Coverage 72.6% 72.62% +0.01% 
==========================================
Files 807 807 Lines 145077 145080 +3 Branches 16213 16213 ==========================================
+ Hits 105337 105366 +29 + Misses 35322 35296 -26 
Partials 4418 4418
FlagCoverage Δ
#Debug72.62% <ø> (+0.01%)⬆️
#production68.17% <ø> (+0.02%)⬆️
#test88.92% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs64.07% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs78.87% <0%> (ø)⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs70.39% <0%> (ø)⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs86.21% <0%> (ø)⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs97.31% <0%> (+0.05%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs84.9% <0%> (+0.2%)⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs91.12% <0%> (+1.02%)⬆️
...LogisticRegression/MulticlassLogisticRegression.cs67.61% <0%> (+1.74%)⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs92.34% <0%> (+3.27%)⬆️
... and 1 more


// at this point, the Label colum is tranformed from strings, to DataViewKeyType and
// the transformation has added the PredictedLabel column, with
var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.


In reply to: 271831815 [](ancestors = 271831815)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question


In reply to: 271866086 [](ancestors = 271866086,271831815)

new LookupMap { Key = "6-11yrs" },
new LookupMap { Key = "25+yrs" }

};

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}; [](start = 12, length = 2)

whitespace #Resolved

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public class MapKeyToValueInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public static class MapValueToKeyInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i had them on the same file, at some point :)


In reply to: 271927396 [](ancestors = 271927396)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs


In reply to: 271930498 [](ancestors = 271930498,271927396)

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating it.


In reply to: 271959976 [](ancestors = 271959976,271935868)

@abgoswamabgoswam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

using Microsoft.ML.SamplesUtils;
using Microsoft.ML.SamplesUtils;

namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought every class had its own distinctive name?


In reply to: 271977575 [](ancestors = 271977575)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, we call them the same between tasks. Reverting.


In reply to: 272288927 [](ancestors = 272288927,271977575)

var mlContext = new MLContext(seed: 0);

// Create a list of data examples.
var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have inline data like the other sample? #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.


In reply to: 271977941 [](ancestors = 271977941)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gani's PR is not checked in. I can switch the Generate after he checks in.


In reply to: 272332287 [](ancestors = 272332287,271977941)


// Get a small dataset as an IEnumerable.
var rawData = new[] {
new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.


In reply to: 271978738 [](ancestors = 271978738)

@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from 4171b5b to a5758c3CompareApril 4, 2019 18:12
@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from a5758c3 to 4e6c4afCompareApril 5, 2019 15:45

// TransformedData obtained post-transformation.
//
// StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 65, length = 15)

Course #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@shmoradimsshmoradimsApr 5, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

@shmoradimsshmoradims left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sfilipi
sfilipi merged commit 7a0d395 into dotnet:masterApr 7, 2019
@sfilipi
sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11
sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019
* Multi column MapKeyToValue and MapValueToKey
@ghostghost locked as resolved and limited conversation to collaborators Mar 23, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentationRelated to documentation of ML.NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sfilipi@abgoswam@shmoradims
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Multi column MapKeyToValue and MapValueToKey - #3187

Merged
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples
Apr 7, 2019
Merged

Multi column MapKeyToValue and MapValueToKey#3187
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples

Conversation

@sfilipi

Copy link
Copy Markdown
Member

Towards #1209 more samples for MapKeyToValue and MapValueToKey

@sfilipisfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019
@sfilipisfilipi self-assigned this Apr 3, 2019
@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
- Coverage 72.54% 72.53% -0.01% 
==========================================
Files 807 807 Lines 144774 144774 Branches 16208 16208 ==========================================
- Hits 105021 105017 -4 - Misses 35339 35343 +4 
Partials 4414 4414
FlagCoverage Δ
#Debug72.53% <ø> (-0.01%)⬇️
#production68.12% <ø> (-0.01%)⬇️
#test88.82% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs44.87% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs89.26% <0%> (-0.63%)⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs86.26% <0%> (+0.15%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs85.11% <0%> (+0.4%)⬆️

@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
+ Coverage 72.6% 72.62% +0.01% 
==========================================
Files 807 807 Lines 145077 145080 +3 Branches 16213 16213 ==========================================
+ Hits 105337 105366 +29 + Misses 35322 35296 -26 
Partials 4418 4418
FlagCoverage Δ
#Debug72.62% <ø> (+0.01%)⬆️
#production68.17% <ø> (+0.02%)⬆️
#test88.92% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs64.07% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs78.87% <0%> (ø)⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs70.39% <0%> (ø)⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs86.21% <0%> (ø)⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs97.31% <0%> (+0.05%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs84.9% <0%> (+0.2%)⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs91.12% <0%> (+1.02%)⬆️
...LogisticRegression/MulticlassLogisticRegression.cs67.61% <0%> (+1.74%)⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs92.34% <0%> (+3.27%)⬆️
... and 1 more


// at this point, the Label colum is tranformed from strings, to DataViewKeyType and
// the transformation has added the PredictedLabel column, with
var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.


In reply to: 271831815 [](ancestors = 271831815)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question


In reply to: 271866086 [](ancestors = 271866086,271831815)

new LookupMap { Key = "6-11yrs" },
new LookupMap { Key = "25+yrs" }

};

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}; [](start = 12, length = 2)

whitespace #Resolved

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public class MapKeyToValueInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public static class MapValueToKeyInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i had them on the same file, at some point :)


In reply to: 271927396 [](ancestors = 271927396)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs


In reply to: 271930498 [](ancestors = 271930498,271927396)

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating it.


In reply to: 271959976 [](ancestors = 271959976,271935868)

@abgoswamabgoswam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

using Microsoft.ML.SamplesUtils;
using Microsoft.ML.SamplesUtils;

namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought every class had its own distinctive name?


In reply to: 271977575 [](ancestors = 271977575)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, we call them the same between tasks. Reverting.


In reply to: 272288927 [](ancestors = 272288927,271977575)

var mlContext = new MLContext(seed: 0);

// Create a list of data examples.
var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have inline data like the other sample? #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.


In reply to: 271977941 [](ancestors = 271977941)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gani's PR is not checked in. I can switch the Generate after he checks in.


In reply to: 272332287 [](ancestors = 272332287,271977941)


// Get a small dataset as an IEnumerable.
var rawData = new[] {
new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.


In reply to: 271978738 [](ancestors = 271978738)

@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from 4171b5b to a5758c3CompareApril 4, 2019 18:12
@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from a5758c3 to 4e6c4afCompareApril 5, 2019 15:45

// TransformedData obtained post-transformation.
//
// StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 65, length = 15)

Course #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@shmoradimsshmoradimsApr 5, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

@shmoradimsshmoradims left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sfilipi
sfilipi merged commit 7a0d395 into dotnet:masterApr 7, 2019
@sfilipi
sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11
sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019
* Multi column MapKeyToValue and MapValueToKey
@ghostghost locked as resolved and limited conversation to collaborators Mar 23, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentationRelated to documentation of ML.NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sfilipi@abgoswam@shmoradims
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Multi column MapKeyToValue and MapValueToKey - #3187

Merged
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples
Apr 7, 2019
Merged

Multi column MapKeyToValue and MapValueToKey#3187
sfilipi merged 9 commits into
dotnet:masterfrom
sfilipi:moreConversionSamples

Conversation

@sfilipi

Copy link
Copy Markdown
Member

Towards #1209 more samples for MapKeyToValue and MapValueToKey

@sfilipisfilipi added the documentation Related to documentation of ML.NET label Apr 3, 2019
@sfilipisfilipi self-assigned this Apr 3, 2019
@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
- Coverage 72.54% 72.53% -0.01% 
==========================================
Files 807 807 Lines 144774 144774 Branches 16208 16208 ==========================================
- Hits 105021 105017 -4 - Misses 35339 35343 +4 
Partials 4414 4414
FlagCoverage Δ
#Debug72.53% <ø> (-0.01%)⬇️
#production68.12% <ø> (-0.01%)⬇️
#test88.82% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs44.87% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
src/Microsoft.ML.Transforms/Text/LdaTransform.cs89.26% <0%> (-0.63%)⬇️
...ML.Transforms/Text/StopWordsRemovingTransformer.cs86.26% <0%> (+0.15%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs85.11% <0%> (+0.4%)⬆️

@codecov

codecovBot commented Apr 3, 2019

Copy link
Copy Markdown

Codecov Report

Merging #3187 into master will increase coverage by 0.01%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #3187 +/- ##
==========================================
+ Coverage 72.6% 72.62% +0.01% 
==========================================
Files 807 807 Lines 145077 145080 +3 Branches 16213 16213 ==========================================
+ Hits 105337 105366 +29 + Misses 35322 35296 -26 
Partials 4418 4418
FlagCoverage Δ
#Debug72.62% <ø> (+0.01%)⬆️
#production68.17% <ø> (+0.02%)⬆️
#test88.92% <ø> (ø)⬆️
Impacted FilesCoverage Δ
...ML.Data/Transforms/ConversionsExtensionsCatalog.cs64.07% <ø> (ø)⬆️
src/Microsoft.ML.Maml/MAML.cs24.75% <0%> (-1.46%)⬇️
...osoft.ML.Recommender/SafeTrainingAndModelBuffer.cs78.87% <0%> (ø)⬆️
...osoft.ML.Recommender/MatrixFactorizationTrainer.cs70.39% <0%> (ø)⬆️
...oft.ML.Recommender/MatrixFactorizationPredictor.cs86.21% <0%> (ø)⬆️
.../Microsoft.ML.Tests/TrainerEstimators/SdcaTests.cs97.31% <0%> (+0.05%)⬆️
...soft.ML.Data/DataLoadSave/Text/TextLoaderCursor.cs84.9% <0%> (+0.2%)⬆️
...oft.ML.StandardTrainers/Standard/SdcaMulticlass.cs91.12% <0%> (+1.02%)⬆️
...LogisticRegression/MulticlassLogisticRegression.cs67.61% <0%> (+1.74%)⬆️
...oft.ML.StandardTrainers/StandardTrainersCatalog.cs92.34% <0%> (+3.27%)⬆️
... and 1 more


// at this point, the Label colum is tranformed from strings, to DataViewKeyType and
// the transformation has added the PredictedLabel column, with
var newPipeline = mlContext.Transforms.Conversion.MapKeyToValue(new[]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newPipeline [](start = 16, length = 11)

i am confused by this -- the newPipeline has no interaction with the previous pipeline.

So how does the newPipeline know about the mapping that pipeline generated in MapValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what MapKeyToValue does. The mapping is saved in the Annotations of the column.


In reply to: 271831815 [](ancestors = 271831815)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah i see.. makes sense..

we may want to add that as comment in the sample .. am sure users will have the same question


In reply to: 271866086 [](ancestors = 271866086,271831815)

new LookupMap { Key = "6-11yrs" },
new LookupMap { Key = "25+yrs" }

};

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

}; [](start = 12, length = 2)

whitespace #Resolved

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKeyInputOutputPair [](start = 98, length = 30)

this API isnt for IOPair..also path looks incorrect

perhaps a case of misplaced example ? #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect.. #Closed

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/MapValueToKeyInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 98, length = 28)

same comment as above #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapKeyToValueInputOutputPair.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueToKey [](start = 26, length = 10)

seems incorrect #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public class MapKeyToValueInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapKeyToValueInputOutputPair [](start = 17, length = 28)

class and file name should be MapKeyToValueMultiColumn #Resolved


namespace Microsoft.ML.Samples.Dynamic
{
public static class MapValueToKeyInputOutputPair

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MapValueToKeyInputOutputPair [](start = 24, length = 28)

we decided to use suffix MultiColumn for these APIs #Resolved

/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[ValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueValueToKey.cs)]
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversion/MapValueToKeyManyColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many [](start = 136, length = 4)

why fixing it in this PR .. this is not the multi column example ?

also.. file name is Multi not Many .. #Resolved

/// <example>
/// <format type="text/markdown">
/// <![CDATA[
/// [!code-csharp[MapValueToKey](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/Transforms/Conversions/MapKeyToValueMultiColumn.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValue [](start = 127, length = 10)

ValueToKey #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i had them on the same file, at some point :)


In reply to: 271927396 [](ancestors = 271927396)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i do not see the change .. i.e use of MapValueToKeyMultiColumn.cs for this API instead of MapKeyToValueMultiColumn.cs


In reply to: 271930498 [](ancestors = 271930498,271927396)

/// <![CDATA[
/// [!code-csharp[KeyToValueMappingEstimator](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/ValueMappingStringToKeyType.cs)]
/// ]]></format>
/// [!code-csharp[MapKeyToValue](~/../docs/samples/docs/samples/Microsoft.ML.Samples/Dynamic/KeyToValueToKey.cs)]

@abgoswamabgoswamApr 3, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KeyToValueToKey.cs [](start = 101, length = 18)

I do not see this file in the codebase #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updating it.


In reply to: 271959976 [](ancestors = 271959976,271935868)

@abgoswamabgoswam left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

using Microsoft.ML.SamplesUtils;
using Microsoft.ML.SamplesUtils;

namespace Microsoft.ML.Samples.Dynamic.Trainers.MulticlassClassification

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.Trainers.MulticlassClassification [](start = 38, length = 34)

why are we removing this? we're using the long namespace for trainers to prevent name conflicts #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought every class had its own distinctive name?


In reply to: 271977575 [](ancestors = 271977575)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ic, we call them the same between tasks. Reverting.


In reply to: 272288927 [](ancestors = 272288927,271977575)

var mlContext = new MLContext(seed: 0);

// Create a list of data examples.
var examples = DatasetUtils.GenerateRandomMulticlassClassificationExamples(1000);

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we have inline data like the other sample? #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like the most common use case for this transform would be this one: after multiclass/binary get back the original values, therefore used it in this context.


In reply to: 271977941 [](ancestors = 271977941)

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gani's PR is not checked in. I can switch the Generate after he checks in.


In reply to: 272332287 [](ancestors = 272332287,271977941)


// Get a small dataset as an IEnumerable.
var rawData = new[] {
new DataPoint() { StudyTime = "0-4yrs" , DevelopmentTime = "6-11yrs" },

@shmoradimsshmoradimsApr 4, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6-11yrs [](start = 76, length = 7)

can we use something other than time? Maybe CourseName? I don't want to give the impression that the values of the two columns should be similar or related in any way. #Resolved

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i kept it to the same things because of the lookup map. It can have keys for two distinct categories, but i think if they they do multicolumn, it will most likely be for two separate columns that have the same categories of values.


In reply to: 271978738 [](ancestors = 271978738)

@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from 4171b5b to a5758c3CompareApril 4, 2019 18:12
@sfilipi
sfilipiforce-pushed the moreConversionSamples branch from a5758c3 to 4e6c4afCompareApril 5, 2019 15:45

// TransformedData obtained post-transformation.
//
// StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 48, length = 15)

copy paste error #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@abgoswamabgoswamApr 5, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTime [](start = 65, length = 15)

Course #Resolved

// This will contain the newly created columns.
features = mlContext.Data.CreateEnumerable<TransformedData>(transformedData, reuseRowObject: false);

Console.WriteLine($" StudyTime StudyTimeCategory DevelopmentTime DevelopmentTimeCategory");

@shmoradimsshmoradimsApr 5, 2019

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DevelopmentTimeCategory [](start = 84, length = 23)

CourseCategory #Resolved

@shmoradimsshmoradims left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@sfilipi
sfilipi merged commit 7a0d395 into dotnet:masterApr 7, 2019
@sfilipi
sfilipi deleted the moreConversionSamples branch April 7, 2019 07:11
sfilipi added a commit to sfilipi/machinelearning-1 that referenced this pull request Apr 9, 2019
* Multi column MapKeyToValue and MapValueToKey
@ghostghost locked as resolved and limited conversation to collaborators Mar 23, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

documentationRelated to documentation of ML.NET

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@sfilipi@abgoswam@shmoradims