Combine the fold metrics into one data view in CV macro.#207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yaeldekel merged 12 commits into from
May 24, 2018
Merged

Combine the fold metrics into one data view in CV macro. #207

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
6f273a3
Add entry point to combine arrays of metrics data views into one data…
yaeldMS May 22, 2018
3543ddb
Add unit test for the multi-class case
yaeldMS May 22, 2018
d9aa28c
Merge
yaeldMS May 22, 2018
5676a27
Update CSharpApi.cs after merge
yaeldMS May 22, 2018
ae376a0
Fix unit test
yaeldMS May 22, 2018
e48378f
Handle the warnings data view, and address PR comment.
yaeldMS May 23, 2018
1c03107
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
yaeldMS May 23, 2018
a695e3e
Fix unit test merged from master.
yaeldMS May 23, 2018
c935ff6
Trigger build.
yaeldMS May 23, 2018
c91b5c1
Address PR comments.
yaeldMS May 24, 2018
733701b
Merge branch 'master' of https://github.com/dotnet/machinelearning in…
yaeldMS May 24, 2018
0ba0f1e
Fix merge problems.
yaeldMS May 24, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ZBaselines/Common/EntryPoints/core_ep-list.tsv
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,7 @@ Models.BinaryClassificationEvaluator Evaluates a binary classification scored da
Models.BinaryCrossValidator Cross validation for binary classification Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro CrossValidateBinary Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Models.ClassificationEvaluator Evaluates a multi class classification scored dataset. Microsoft.ML.Runtime.Data.Evaluate MultiClass Microsoft.ML.Runtime.Data.MultiClassMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.ClusterEvaluator Evaluates a clustering scored dataset. Microsoft.ML.Runtime.Data.Evaluate Clustering Microsoft.ML.Runtime.Data.ClusteringMamlEvaluator+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.CrossValidationResultsCombiner Combine the metric data views returned from cross validation. Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro CombineMetrics Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombineMetricsInput Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombinedOutput

@codemzscodemzsMay 23, 2018•
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel Seems like a positive change but out of curiosity what was the rationale behind it? I'll reference this change in my PR(not yet out) that augments CV and TrainTest macro to work with LearningPipeline framework. #Closed

@yaeldekelyaeldekelMay 23, 2018•
edited by codemzs
Loading

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @codemzs , the reason I was making this change is so that we won't have code duplication between different users of this macro, when processing its outputs (such as finding the weighted/unweighted metrics for each data view, computing the average and standard deviation of each metric, and making sure that multi-class metrics refer to the same classes in all folds). #Closed

Models.CrossValidator Cross validation for general learning Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro CrossValidate Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Arguments Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Output]
Models.CrossValidatorDatasetSplitter Split the dataset into the specified number of cross-validation folds (train and test sets) Microsoft.ML.Runtime.EntryPoints.CVSplit Split Microsoft.ML.Runtime.EntryPoints.CVSplit+Input Microsoft.ML.Runtime.EntryPoints.CVSplit+Output
Models.DatasetTransformer Applies a TransformModel to a dataset. Microsoft.ML.Runtime.EntryPoints.ModelOperations Apply Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelInput Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelOutput
Expand Down
130 changes: 114 additions & 16 deletions ZBaselines/Common/EntryPoints/core_manifest.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1238,6 +1238,116 @@
"IEvaluatorOutput"
]
},
{
"Name": "Models.CrossValidationResultsCombiner",
"Desc": "Combine the metric data views returned from cross validation.",
"FriendlyName": null,
"ShortName": null,
"Inputs": [
{
"Name": "OverallMetrics",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Desc": "Overall metrics datasets",
"Required": false,
"SortOrder": 1.0,
"IsNullable": false,
"Default": null
},
{
"Name": "PerInstanceMetrics",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Desc": "Per instance metrics datasets",
"Required": false,
"SortOrder": 2.0,
"IsNullable": false,
"Default": null
},
{
"Name": "ConfusionMatrix",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Desc": "Confusion matrix datasets",
"Required": false,
"SortOrder": 3.0,
"IsNullable": false,
"Default": null
},
{
"Name": "Warnings",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Desc": "Warning datasets",
"Required": false,
"SortOrder": 4.0,
"IsNullable": false,
"Default": null
},
{
"Name": "LabelColumn",
"Type": "String",
"Desc": "The label column name",
"Aliases": [
"Label"
],
"Required": false,
"SortOrder": 6.0,
"IsNullable": false,
"Default": "Label"
},
{
"Name": "Kind",
"Type": {
"Kind": "Enum",
"Values": [
"SignatureBinaryClassifierTrainer",
"SignatureMultiClassClassifierTrainer",
"SignatureRankerTrainer",
"SignatureRegressorTrainer",
"SignatureMultiOutputRegressorTrainer",
"SignatureAnomalyDetectorTrainer",
"SignatureClusteringTrainer"
]
},
"Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
"Required": true,
"SortOrder": 7.0,
"IsNullable": false,
"Default": "SignatureBinaryClassifierTrainer"
}
],
"Outputs": [
{
"Name": "Warnings",
"Type": "DataView",
"Desc": "Warning dataset"
},
{
"Name": "OverallMetrics",
"Type": "DataView",
"Desc": "Overall metrics dataset"
},
{
"Name": "PerInstanceMetrics",
"Type": "DataView",
"Desc": "Per instance metrics dataset"
},
{
"Name": "ConfusionMatrix",
"Type": "DataView",
"Desc": "Confusion matrix dataset"
}
]
},
{
"Name": "Models.CrossValidator",
"Desc": "Cross validation for general learning",
Expand DownExpand Up@@ -1368,34 +1478,22 @@
},
{
"Name": "Warnings",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Type": "DataView",
"Desc": "Warning dataset"
},
{
"Name": "OverallMetrics",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Type": "DataView",
"Desc": "Overall metrics dataset"
},
{
"Name": "PerInstanceMetrics",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Type": "DataView",
"Desc": "Per instance metrics dataset"
},
{
"Name": "ConfusionMatrix",
"Type": {
"Kind": "Array",
"ItemType": "DataView"
},
"Type": "DataView",
"Desc": "Confusion matrix dataset"
}
]
Expand Down
Loading
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Combine the fold metrics into one data view in CV macro.#207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

yaeldekel
merged 12 commits into

from

May 24, 2018
 Merged


Combine the fold metrics into one data view in CV macro.
#207
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
6f273a3

Add entry point to combine arrays of metrics data views into one data…


yaeldMS May 22, 2018
3543ddb

Add unit test for the multi-class case


yaeldMS May 22, 2018
d9aa28c

Merge


yaeldMS May 22, 2018
5676a27

Update CSharpApi.cs after merge


yaeldMS May 22, 2018
ae376a0

Fix unit test


yaeldMS May 22, 2018
e48378f

Handle the warnings data view, and address PR comment.


yaeldMS May 23, 2018
1c03107

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 23, 2018
a695e3e

Fix unit test merged from master.


yaeldMS May 23, 2018
c935ff6

Trigger build.


yaeldMS May 23, 2018
c91b5c1

Address PR comments.


yaeldMS May 24, 2018
733701b

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 24, 2018
0ba0f1e

Fix merge problems.


yaeldMS May 24, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
ZBaselines/Common/EntryPoints/core_ep-list.tsv

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,7 @@ Models.BinaryClassificationEvaluator	Evaluates a binary classification scored da
Models.BinaryCrossValidator	Cross validation for binary classification	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro	CrossValidateBinary	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Models.ClassificationEvaluator	Evaluates a multi class classification scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	MultiClass	Microsoft.ML.Runtime.Data.MultiClassMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.ClusterEvaluator	Evaluates a clustering scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	Clustering	Microsoft.ML.Runtime.Data.ClusteringMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.CrossValidationResultsCombiner	Combine the metric data views returned from cross validation.	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CombineMetrics	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombineMetricsInput	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombinedOutput
@codemzscodemzsMay 23, 2018•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel Seems like a positive change but out of curiosity what was the rationale behind it? I'll reference this change in my PR(not yet out) that augments CV and TrainTest macro to work with LearningPipeline framework. #Closed
@yaeldekelyaeldekelMay 23, 2018•

edited by codemzs

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @codemzs , the reason I was making this change is so that we won't have code duplication between different users of this macro, when processing its outputs (such as finding the weighted/unweighted metrics for each data view, computing the average and standard deviation of each metric, and making sure that multi-class metrics refer to the same classes in all folds). #Closed
Models.CrossValidator	Cross validation for general learning	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CrossValidate	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Output]
Models.CrossValidatorDatasetSplitter	Split the dataset into the specified number of cross-validation folds (train and test sets)	Microsoft.ML.Runtime.EntryPoints.CVSplit	Split	Microsoft.ML.Runtime.EntryPoints.CVSplit+Input	Microsoft.ML.Runtime.EntryPoints.CVSplit+Output
Models.DatasetTransformer	Applies a TransformModel to a dataset.	Microsoft.ML.Runtime.EntryPoints.ModelOperations	Apply	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelInput	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelOutput
Expand Down

130 changes: 114 additions & 16 deletions
ZBaselines/Common/EntryPoints/core_manifest.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1238,6 +1238,116 @@
 "IEvaluatorOutput"
 ]
 },
 {
 "Name": "Models.CrossValidationResultsCombiner",
 "Desc": "Combine the metric data views returned from cross validation.",
 "FriendlyName": null,
 "ShortName": null,
 "Inputs": [
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Overall metrics datasets",
 "Required": false,
 "SortOrder": 1.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Per instance metrics datasets",
 "Required": false,
 "SortOrder": 2.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Confusion matrix datasets",
 "Required": false,
 "SortOrder": 3.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Warning datasets",
 "Required": false,
 "SortOrder": 4.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "LabelColumn",
 "Type": "String",
 "Desc": "The label column name",
 "Aliases": [
 "Label"
 ],
 "Required": false,
 "SortOrder": 6.0,
 "IsNullable": false,
 "Default": "Label"
 },
 {
 "Name": "Kind",
 "Type": {
 "Kind": "Enum",
 "Values": [
 "SignatureBinaryClassifierTrainer",
 "SignatureMultiClassClassifierTrainer",
 "SignatureRankerTrainer",
 "SignatureRegressorTrainer",
 "SignatureMultiOutputRegressorTrainer",
 "SignatureAnomalyDetectorTrainer",
 "SignatureClusteringTrainer"
 ]
 },
 "Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
 "Required": true,
 "SortOrder": 7.0,
 "IsNullable": false,
 "Default": "SignatureBinaryClassifierTrainer"
 }
 ],
 "Outputs": [
 {
 "Name": "Warnings",
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
 },
 {
 "Name": "Models.CrossValidator",
 "Desc": "Cross validation for general learning",
Expand DownExpand Up@@ -1368,34 +1478,22 @@
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Combine the fold metrics into one data view in CV macro.#207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

yaeldekel
merged 12 commits into

from

May 24, 2018
 Merged


Combine the fold metrics into one data view in CV macro.
#207
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
6f273a3

Add entry point to combine arrays of metrics data views into one data…


yaeldMS May 22, 2018
3543ddb

Add unit test for the multi-class case


yaeldMS May 22, 2018
d9aa28c

Merge


yaeldMS May 22, 2018
5676a27

Update CSharpApi.cs after merge


yaeldMS May 22, 2018
ae376a0

Fix unit test


yaeldMS May 22, 2018
e48378f

Handle the warnings data view, and address PR comment.


yaeldMS May 23, 2018
1c03107

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 23, 2018
a695e3e

Fix unit test merged from master.


yaeldMS May 23, 2018
c935ff6

Trigger build.


yaeldMS May 23, 2018
c91b5c1

Address PR comments.


yaeldMS May 24, 2018
733701b

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 24, 2018
0ba0f1e

Fix merge problems.


yaeldMS May 24, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
ZBaselines/Common/EntryPoints/core_ep-list.tsv

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,7 @@ Models.BinaryClassificationEvaluator	Evaluates a binary classification scored da
Models.BinaryCrossValidator	Cross validation for binary classification	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro	CrossValidateBinary	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Models.ClassificationEvaluator	Evaluates a multi class classification scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	MultiClass	Microsoft.ML.Runtime.Data.MultiClassMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.ClusterEvaluator	Evaluates a clustering scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	Clustering	Microsoft.ML.Runtime.Data.ClusteringMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.CrossValidationResultsCombiner	Combine the metric data views returned from cross validation.	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CombineMetrics	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombineMetricsInput	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombinedOutput
@codemzscodemzsMay 23, 2018•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel Seems like a positive change but out of curiosity what was the rationale behind it? I'll reference this change in my PR(not yet out) that augments CV and TrainTest macro to work with LearningPipeline framework. #Closed
@yaeldekelyaeldekelMay 23, 2018•

edited by codemzs

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @codemzs , the reason I was making this change is so that we won't have code duplication between different users of this macro, when processing its outputs (such as finding the weighted/unweighted metrics for each data view, computing the average and standard deviation of each metric, and making sure that multi-class metrics refer to the same classes in all folds). #Closed
Models.CrossValidator	Cross validation for general learning	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CrossValidate	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Output]
Models.CrossValidatorDatasetSplitter	Split the dataset into the specified number of cross-validation folds (train and test sets)	Microsoft.ML.Runtime.EntryPoints.CVSplit	Split	Microsoft.ML.Runtime.EntryPoints.CVSplit+Input	Microsoft.ML.Runtime.EntryPoints.CVSplit+Output
Models.DatasetTransformer	Applies a TransformModel to a dataset.	Microsoft.ML.Runtime.EntryPoints.ModelOperations	Apply	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelInput	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelOutput
Expand Down

130 changes: 114 additions & 16 deletions
ZBaselines/Common/EntryPoints/core_manifest.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1238,6 +1238,116 @@
 "IEvaluatorOutput"
 ]
 },
 {
 "Name": "Models.CrossValidationResultsCombiner",
 "Desc": "Combine the metric data views returned from cross validation.",
 "FriendlyName": null,
 "ShortName": null,
 "Inputs": [
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Overall metrics datasets",
 "Required": false,
 "SortOrder": 1.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Per instance metrics datasets",
 "Required": false,
 "SortOrder": 2.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Confusion matrix datasets",
 "Required": false,
 "SortOrder": 3.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Warning datasets",
 "Required": false,
 "SortOrder": 4.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "LabelColumn",
 "Type": "String",
 "Desc": "The label column name",
 "Aliases": [
 "Label"
 ],
 "Required": false,
 "SortOrder": 6.0,
 "IsNullable": false,
 "Default": "Label"
 },
 {
 "Name": "Kind",
 "Type": {
 "Kind": "Enum",
 "Values": [
 "SignatureBinaryClassifierTrainer",
 "SignatureMultiClassClassifierTrainer",
 "SignatureRankerTrainer",
 "SignatureRegressorTrainer",
 "SignatureMultiOutputRegressorTrainer",
 "SignatureAnomalyDetectorTrainer",
 "SignatureClusteringTrainer"
 ]
 },
 "Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
 "Required": true,
 "SortOrder": 7.0,
 "IsNullable": false,
 "Default": "SignatureBinaryClassifierTrainer"
 }
 ],
 "Outputs": [
 {
 "Name": "Warnings",
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
 },
 {
 "Name": "Models.CrossValidator",
 "Desc": "Cross validation for general learning",
Expand DownExpand Up@@ -1368,34 +1478,22 @@
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Combine the fold metrics into one data view in CV macro.#207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

yaeldekel
merged 12 commits into

from

May 24, 2018
 Merged


Combine the fold metrics into one data view in CV macro.
#207
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
6f273a3

Add entry point to combine arrays of metrics data views into one data…


yaeldMS May 22, 2018
3543ddb

Add unit test for the multi-class case


yaeldMS May 22, 2018
d9aa28c

Merge


yaeldMS May 22, 2018
5676a27

Update CSharpApi.cs after merge


yaeldMS May 22, 2018
ae376a0

Fix unit test


yaeldMS May 22, 2018
e48378f

Handle the warnings data view, and address PR comment.


yaeldMS May 23, 2018
1c03107

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 23, 2018
a695e3e

Fix unit test merged from master.


yaeldMS May 23, 2018
c935ff6

Trigger build.


yaeldMS May 23, 2018
c91b5c1

Address PR comments.


yaeldMS May 24, 2018
733701b

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 24, 2018
0ba0f1e

Fix merge problems.


yaeldMS May 24, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
ZBaselines/Common/EntryPoints/core_ep-list.tsv

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,7 @@ Models.BinaryClassificationEvaluator	Evaluates a binary classification scored da
Models.BinaryCrossValidator	Cross validation for binary classification	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro	CrossValidateBinary	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Models.ClassificationEvaluator	Evaluates a multi class classification scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	MultiClass	Microsoft.ML.Runtime.Data.MultiClassMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.ClusterEvaluator	Evaluates a clustering scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	Clustering	Microsoft.ML.Runtime.Data.ClusteringMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.CrossValidationResultsCombiner	Combine the metric data views returned from cross validation.	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CombineMetrics	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombineMetricsInput	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombinedOutput
@codemzscodemzsMay 23, 2018•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel Seems like a positive change but out of curiosity what was the rationale behind it? I'll reference this change in my PR(not yet out) that augments CV and TrainTest macro to work with LearningPipeline framework. #Closed
@yaeldekelyaeldekelMay 23, 2018•

edited by codemzs

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @codemzs , the reason I was making this change is so that we won't have code duplication between different users of this macro, when processing its outputs (such as finding the weighted/unweighted metrics for each data view, computing the average and standard deviation of each metric, and making sure that multi-class metrics refer to the same classes in all folds). #Closed
Models.CrossValidator	Cross validation for general learning	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CrossValidate	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Output]
Models.CrossValidatorDatasetSplitter	Split the dataset into the specified number of cross-validation folds (train and test sets)	Microsoft.ML.Runtime.EntryPoints.CVSplit	Split	Microsoft.ML.Runtime.EntryPoints.CVSplit+Input	Microsoft.ML.Runtime.EntryPoints.CVSplit+Output
Models.DatasetTransformer	Applies a TransformModel to a dataset.	Microsoft.ML.Runtime.EntryPoints.ModelOperations	Apply	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelInput	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelOutput
Expand Down

130 changes: 114 additions & 16 deletions
ZBaselines/Common/EntryPoints/core_manifest.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1238,6 +1238,116 @@
 "IEvaluatorOutput"
 ]
 },
 {
 "Name": "Models.CrossValidationResultsCombiner",
 "Desc": "Combine the metric data views returned from cross validation.",
 "FriendlyName": null,
 "ShortName": null,
 "Inputs": [
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Overall metrics datasets",
 "Required": false,
 "SortOrder": 1.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Per instance metrics datasets",
 "Required": false,
 "SortOrder": 2.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Confusion matrix datasets",
 "Required": false,
 "SortOrder": 3.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Warning datasets",
 "Required": false,
 "SortOrder": 4.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "LabelColumn",
 "Type": "String",
 "Desc": "The label column name",
 "Aliases": [
 "Label"
 ],
 "Required": false,
 "SortOrder": 6.0,
 "IsNullable": false,
 "Default": "Label"
 },
 {
 "Name": "Kind",
 "Type": {
 "Kind": "Enum",
 "Values": [
 "SignatureBinaryClassifierTrainer",
 "SignatureMultiClassClassifierTrainer",
 "SignatureRankerTrainer",
 "SignatureRegressorTrainer",
 "SignatureMultiOutputRegressorTrainer",
 "SignatureAnomalyDetectorTrainer",
 "SignatureClusteringTrainer"
 ]
 },
 "Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
 "Required": true,
 "SortOrder": 7.0,
 "IsNullable": false,
 "Default": "SignatureBinaryClassifierTrainer"
 }
 ],
 "Outputs": [
 {
 "Name": "Warnings",
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
 },
 {
 "Name": "Models.CrossValidator",
 "Desc": "Cross validation for general learning",
Expand DownExpand Up@@ -1368,34 +1478,22 @@
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Combine the fold metrics into one data view in CV macro.#207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

yaeldekel
merged 12 commits into

from

May 24, 2018
 Merged


Combine the fold metrics into one data view in CV macro.
#207
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
6f273a3

Add entry point to combine arrays of metrics data views into one data…


yaeldMS May 22, 2018
3543ddb

Add unit test for the multi-class case


yaeldMS May 22, 2018
d9aa28c

Merge


yaeldMS May 22, 2018
5676a27

Update CSharpApi.cs after merge


yaeldMS May 22, 2018
ae376a0

Fix unit test


yaeldMS May 22, 2018
e48378f

Handle the warnings data view, and address PR comment.


yaeldMS May 23, 2018
1c03107

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 23, 2018
a695e3e

Fix unit test merged from master.


yaeldMS May 23, 2018
c935ff6

Trigger build.


yaeldMS May 23, 2018
c91b5c1

Address PR comments.


yaeldMS May 24, 2018
733701b

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 24, 2018
0ba0f1e

Fix merge problems.


yaeldMS May 24, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
ZBaselines/Common/EntryPoints/core_ep-list.tsv

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,7 @@ Models.BinaryClassificationEvaluator	Evaluates a binary classification scored da
Models.BinaryCrossValidator	Cross validation for binary classification	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro	CrossValidateBinary	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Models.ClassificationEvaluator	Evaluates a multi class classification scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	MultiClass	Microsoft.ML.Runtime.Data.MultiClassMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.ClusterEvaluator	Evaluates a clustering scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	Clustering	Microsoft.ML.Runtime.Data.ClusteringMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.CrossValidationResultsCombiner	Combine the metric data views returned from cross validation.	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CombineMetrics	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombineMetricsInput	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombinedOutput
@codemzscodemzsMay 23, 2018•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel Seems like a positive change but out of curiosity what was the rationale behind it? I'll reference this change in my PR(not yet out) that augments CV and TrainTest macro to work with LearningPipeline framework. #Closed
@yaeldekelyaeldekelMay 23, 2018•

edited by codemzs

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @codemzs , the reason I was making this change is so that we won't have code duplication between different users of this macro, when processing its outputs (such as finding the weighted/unweighted metrics for each data view, computing the average and standard deviation of each metric, and making sure that multi-class metrics refer to the same classes in all folds). #Closed
Models.CrossValidator	Cross validation for general learning	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CrossValidate	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Output]
Models.CrossValidatorDatasetSplitter	Split the dataset into the specified number of cross-validation folds (train and test sets)	Microsoft.ML.Runtime.EntryPoints.CVSplit	Split	Microsoft.ML.Runtime.EntryPoints.CVSplit+Input	Microsoft.ML.Runtime.EntryPoints.CVSplit+Output
Models.DatasetTransformer	Applies a TransformModel to a dataset.	Microsoft.ML.Runtime.EntryPoints.ModelOperations	Apply	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelInput	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelOutput
Expand Down

130 changes: 114 additions & 16 deletions
ZBaselines/Common/EntryPoints/core_manifest.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1238,6 +1238,116 @@
 "IEvaluatorOutput"
 ]
 },
 {
 "Name": "Models.CrossValidationResultsCombiner",
 "Desc": "Combine the metric data views returned from cross validation.",
 "FriendlyName": null,
 "ShortName": null,
 "Inputs": [
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Overall metrics datasets",
 "Required": false,
 "SortOrder": 1.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Per instance metrics datasets",
 "Required": false,
 "SortOrder": 2.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Confusion matrix datasets",
 "Required": false,
 "SortOrder": 3.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Warning datasets",
 "Required": false,
 "SortOrder": 4.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "LabelColumn",
 "Type": "String",
 "Desc": "The label column name",
 "Aliases": [
 "Label"
 ],
 "Required": false,
 "SortOrder": 6.0,
 "IsNullable": false,
 "Default": "Label"
 },
 {
 "Name": "Kind",
 "Type": {
 "Kind": "Enum",
 "Values": [
 "SignatureBinaryClassifierTrainer",
 "SignatureMultiClassClassifierTrainer",
 "SignatureRankerTrainer",
 "SignatureRegressorTrainer",
 "SignatureMultiOutputRegressorTrainer",
 "SignatureAnomalyDetectorTrainer",
 "SignatureClusteringTrainer"
 ]
 },
 "Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
 "Required": true,
 "SortOrder": 7.0,
 "IsNullable": false,
 "Default": "SignatureBinaryClassifierTrainer"
 }
 ],
 "Outputs": [
 {
 "Name": "Warnings",
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
 },
 {
 "Name": "Models.CrossValidator",
 "Desc": "Cross validation for general learning",
Expand DownExpand Up@@ -1368,34 +1478,22 @@
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Combine the fold metrics into one data view in CV macro.#207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

yaeldekel
merged 12 commits into

from

May 24, 2018
 Merged


Combine the fold metrics into one data view in CV macro.
#207
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
6f273a3

Add entry point to combine arrays of metrics data views into one data…


yaeldMS May 22, 2018
3543ddb

Add unit test for the multi-class case


yaeldMS May 22, 2018
d9aa28c

Merge


yaeldMS May 22, 2018
5676a27

Update CSharpApi.cs after merge


yaeldMS May 22, 2018
ae376a0

Fix unit test


yaeldMS May 22, 2018
e48378f

Handle the warnings data view, and address PR comment.


yaeldMS May 23, 2018
1c03107

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 23, 2018
a695e3e

Fix unit test merged from master.


yaeldMS May 23, 2018
c935ff6

Trigger build.


yaeldMS May 23, 2018
c91b5c1

Address PR comments.


yaeldMS May 24, 2018
733701b

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 24, 2018
0ba0f1e

Fix merge problems.


yaeldMS May 24, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
ZBaselines/Common/EntryPoints/core_ep-list.tsv

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,7 @@ Models.BinaryClassificationEvaluator	Evaluates a binary classification scored da
Models.BinaryCrossValidator	Cross validation for binary classification	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro	CrossValidateBinary	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Models.ClassificationEvaluator	Evaluates a multi class classification scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	MultiClass	Microsoft.ML.Runtime.Data.MultiClassMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.ClusterEvaluator	Evaluates a clustering scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	Clustering	Microsoft.ML.Runtime.Data.ClusteringMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.CrossValidationResultsCombiner	Combine the metric data views returned from cross validation.	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CombineMetrics	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombineMetricsInput	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombinedOutput
@codemzscodemzsMay 23, 2018•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel Seems like a positive change but out of curiosity what was the rationale behind it? I'll reference this change in my PR(not yet out) that augments CV and TrainTest macro to work with LearningPipeline framework. #Closed
@yaeldekelyaeldekelMay 23, 2018•

edited by codemzs

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @codemzs , the reason I was making this change is so that we won't have code duplication between different users of this macro, when processing its outputs (such as finding the weighted/unweighted metrics for each data view, computing the average and standard deviation of each metric, and making sure that multi-class metrics refer to the same classes in all folds). #Closed
Models.CrossValidator	Cross validation for general learning	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CrossValidate	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Output]
Models.CrossValidatorDatasetSplitter	Split the dataset into the specified number of cross-validation folds (train and test sets)	Microsoft.ML.Runtime.EntryPoints.CVSplit	Split	Microsoft.ML.Runtime.EntryPoints.CVSplit+Input	Microsoft.ML.Runtime.EntryPoints.CVSplit+Output
Models.DatasetTransformer	Applies a TransformModel to a dataset.	Microsoft.ML.Runtime.EntryPoints.ModelOperations	Apply	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelInput	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelOutput
Expand Down

130 changes: 114 additions & 16 deletions
ZBaselines/Common/EntryPoints/core_manifest.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1238,6 +1238,116 @@
 "IEvaluatorOutput"
 ]
 },
 {
 "Name": "Models.CrossValidationResultsCombiner",
 "Desc": "Combine the metric data views returned from cross validation.",
 "FriendlyName": null,
 "ShortName": null,
 "Inputs": [
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Overall metrics datasets",
 "Required": false,
 "SortOrder": 1.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Per instance metrics datasets",
 "Required": false,
 "SortOrder": 2.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Confusion matrix datasets",
 "Required": false,
 "SortOrder": 3.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Warning datasets",
 "Required": false,
 "SortOrder": 4.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "LabelColumn",
 "Type": "String",
 "Desc": "The label column name",
 "Aliases": [
 "Label"
 ],
 "Required": false,
 "SortOrder": 6.0,
 "IsNullable": false,
 "Default": "Label"
 },
 {
 "Name": "Kind",
 "Type": {
 "Kind": "Enum",
 "Values": [
 "SignatureBinaryClassifierTrainer",
 "SignatureMultiClassClassifierTrainer",
 "SignatureRankerTrainer",
 "SignatureRegressorTrainer",
 "SignatureMultiOutputRegressorTrainer",
 "SignatureAnomalyDetectorTrainer",
 "SignatureClusteringTrainer"
 ]
 },
 "Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
 "Required": true,
 "SortOrder": 7.0,
 "IsNullable": false,
 "Default": "SignatureBinaryClassifierTrainer"
 }
 ],
 "Outputs": [
 {
 "Name": "Warnings",
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
 },
 {
 "Name": "Models.CrossValidator",
 "Desc": "Cross validation for general learning",
Expand DownExpand Up@@ -1368,34 +1478,22 @@
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Combine the fold metrics into one data view in CV macro.#207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

yaeldekel
merged 12 commits into

from

May 24, 2018
 Merged


Combine the fold metrics into one data view in CV macro.
#207
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
6f273a3

Add entry point to combine arrays of metrics data views into one data…


yaeldMS May 22, 2018
3543ddb

Add unit test for the multi-class case


yaeldMS May 22, 2018
d9aa28c

Merge


yaeldMS May 22, 2018
5676a27

Update CSharpApi.cs after merge


yaeldMS May 22, 2018
ae376a0

Fix unit test


yaeldMS May 22, 2018
e48378f

Handle the warnings data view, and address PR comment.


yaeldMS May 23, 2018
1c03107

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 23, 2018
a695e3e

Fix unit test merged from master.


yaeldMS May 23, 2018
c935ff6

Trigger build.


yaeldMS May 23, 2018
c91b5c1

Address PR comments.


yaeldMS May 24, 2018
733701b

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 24, 2018
0ba0f1e

Fix merge problems.


yaeldMS May 24, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
ZBaselines/Common/EntryPoints/core_ep-list.tsv

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,7 @@ Models.BinaryClassificationEvaluator	Evaluates a binary classification scored da
Models.BinaryCrossValidator	Cross validation for binary classification	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro	CrossValidateBinary	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Models.ClassificationEvaluator	Evaluates a multi class classification scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	MultiClass	Microsoft.ML.Runtime.Data.MultiClassMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.ClusterEvaluator	Evaluates a clustering scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	Clustering	Microsoft.ML.Runtime.Data.ClusteringMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.CrossValidationResultsCombiner	Combine the metric data views returned from cross validation.	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CombineMetrics	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombineMetricsInput	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombinedOutput
@codemzscodemzsMay 23, 2018•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel Seems like a positive change but out of curiosity what was the rationale behind it? I'll reference this change in my PR(not yet out) that augments CV and TrainTest macro to work with LearningPipeline framework. #Closed
@yaeldekelyaeldekelMay 23, 2018•

edited by codemzs

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @codemzs , the reason I was making this change is so that we won't have code duplication between different users of this macro, when processing its outputs (such as finding the weighted/unweighted metrics for each data view, computing the average and standard deviation of each metric, and making sure that multi-class metrics refer to the same classes in all folds). #Closed
Models.CrossValidator	Cross validation for general learning	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CrossValidate	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Output]
Models.CrossValidatorDatasetSplitter	Split the dataset into the specified number of cross-validation folds (train and test sets)	Microsoft.ML.Runtime.EntryPoints.CVSplit	Split	Microsoft.ML.Runtime.EntryPoints.CVSplit+Input	Microsoft.ML.Runtime.EntryPoints.CVSplit+Output
Models.DatasetTransformer	Applies a TransformModel to a dataset.	Microsoft.ML.Runtime.EntryPoints.ModelOperations	Apply	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelInput	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelOutput
Expand Down

130 changes: 114 additions & 16 deletions
ZBaselines/Common/EntryPoints/core_manifest.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1238,6 +1238,116 @@
 "IEvaluatorOutput"
 ]
 },
 {
 "Name": "Models.CrossValidationResultsCombiner",
 "Desc": "Combine the metric data views returned from cross validation.",
 "FriendlyName": null,
 "ShortName": null,
 "Inputs": [
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Overall metrics datasets",
 "Required": false,
 "SortOrder": 1.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Per instance metrics datasets",
 "Required": false,
 "SortOrder": 2.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Confusion matrix datasets",
 "Required": false,
 "SortOrder": 3.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Warning datasets",
 "Required": false,
 "SortOrder": 4.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "LabelColumn",
 "Type": "String",
 "Desc": "The label column name",
 "Aliases": [
 "Label"
 ],
 "Required": false,
 "SortOrder": 6.0,
 "IsNullable": false,
 "Default": "Label"
 },
 {
 "Name": "Kind",
 "Type": {
 "Kind": "Enum",
 "Values": [
 "SignatureBinaryClassifierTrainer",
 "SignatureMultiClassClassifierTrainer",
 "SignatureRankerTrainer",
 "SignatureRegressorTrainer",
 "SignatureMultiOutputRegressorTrainer",
 "SignatureAnomalyDetectorTrainer",
 "SignatureClusteringTrainer"
 ]
 },
 "Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
 "Required": true,
 "SortOrder": 7.0,
 "IsNullable": false,
 "Default": "SignatureBinaryClassifierTrainer"
 }
 ],
 "Outputs": [
 {
 "Name": "Warnings",
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
 },
 {
 "Name": "Models.CrossValidator",
 "Desc": "Cross validation for general learning",
Expand DownExpand Up@@ -1368,34 +1478,22 @@
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
Expand Down
Loading

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Combine the fold metrics into one data view in CV macro.#207

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

yaeldekel
merged 12 commits into

from

May 24, 2018
 Merged


Combine the fold metrics into one data view in CV macro.
#207
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


12 commits


Select commit
Hold shift + click to select a range
6f273a3

Add entry point to combine arrays of metrics data views into one data…


yaeldMS May 22, 2018
3543ddb

Add unit test for the multi-class case


yaeldMS May 22, 2018
d9aa28c

Merge


yaeldMS May 22, 2018
5676a27

Update CSharpApi.cs after merge


yaeldMS May 22, 2018
ae376a0

Fix unit test


yaeldMS May 22, 2018
e48378f

Handle the warnings data view, and address PR comment.


yaeldMS May 23, 2018
1c03107

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 23, 2018
a695e3e

Fix unit test merged from master.


yaeldMS May 23, 2018
c935ff6

Trigger build.


yaeldMS May 23, 2018
c91b5c1

Address PR comments.


yaeldMS May 24, 2018
733701b

Merge branch 'master' of https://github.com/dotnet/machinelearning in…


yaeldMS May 24, 2018
0ba0f1e

Fix merge problems.


yaeldMS May 24, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


1 change: 1 addition & 0 deletions
ZBaselines/Common/EntryPoints/core_ep-list.tsv

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,7 @@ Models.BinaryClassificationEvaluator	Evaluates a binary classification scored da
Models.BinaryCrossValidator	Cross validation for binary classification	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro	CrossValidateBinary	Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationBinaryMacro+Output]
Models.ClassificationEvaluator	Evaluates a multi class classification scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	MultiClass	Microsoft.ML.Runtime.Data.MultiClassMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+ClassificationEvaluateOutput
Models.ClusterEvaluator	Evaluates a clustering scored dataset.	Microsoft.ML.Runtime.Data.Evaluate	Clustering	Microsoft.ML.Runtime.Data.ClusteringMamlEvaluator+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+CommonEvaluateOutput
Models.CrossValidationResultsCombiner	Combine the metric data views returned from cross validation.	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CombineMetrics	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombineMetricsInput	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+CombinedOutput
@codemzscodemzsMay 23, 2018•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yaeldekel Seems like a positive change but out of curiosity what was the rationale behind it? I'll reference this change in my PR(not yet out) that augments CV and TrainTest macro to work with LearningPipeline framework. #Closed
@yaeldekelyaeldekelMay 23, 2018•

edited by codemzs

Loading

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @codemzs , the reason I was making this change is so that we won't have code duplication between different users of this macro, when processing its outputs (such as finding the weighted/unweighted metrics for each data view, computing the average and standard deviation of each metric, and making sure that multi-class metrics refer to the same classes in all folds). #Closed
Models.CrossValidator	Cross validation for general learning	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro	CrossValidate	Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Arguments	Microsoft.ML.Runtime.EntryPoints.CommonOutputs+MacroOutput`1[Microsoft.ML.Runtime.EntryPoints.CrossValidationMacro+Output]
Models.CrossValidatorDatasetSplitter	Split the dataset into the specified number of cross-validation folds (train and test sets)	Microsoft.ML.Runtime.EntryPoints.CVSplit	Split	Microsoft.ML.Runtime.EntryPoints.CVSplit+Input	Microsoft.ML.Runtime.EntryPoints.CVSplit+Output
Models.DatasetTransformer	Applies a TransformModel to a dataset.	Microsoft.ML.Runtime.EntryPoints.ModelOperations	Apply	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelInput	Microsoft.ML.Runtime.EntryPoints.ModelOperations+ApplyTransformModelOutput
Expand Down

130 changes: 114 additions & 16 deletions
ZBaselines/Common/EntryPoints/core_manifest.json

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1238,6 +1238,116 @@
 "IEvaluatorOutput"
 ]
 },
 {
 "Name": "Models.CrossValidationResultsCombiner",
 "Desc": "Combine the metric data views returned from cross validation.",
 "FriendlyName": null,
 "ShortName": null,
 "Inputs": [
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Overall metrics datasets",
 "Required": false,
 "SortOrder": 1.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Per instance metrics datasets",
 "Required": false,
 "SortOrder": 2.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Confusion matrix datasets",
 "Required": false,
 "SortOrder": 3.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Desc": "Warning datasets",
 "Required": false,
 "SortOrder": 4.0,
 "IsNullable": false,
 "Default": null
 },
 {
 "Name": "LabelColumn",
 "Type": "String",
 "Desc": "The label column name",
 "Aliases": [
 "Label"
 ],
 "Required": false,
 "SortOrder": 6.0,
 "IsNullable": false,
 "Default": "Label"
 },
 {
 "Name": "Kind",
 "Type": {
 "Kind": "Enum",
 "Values": [
 "SignatureBinaryClassifierTrainer",
 "SignatureMultiClassClassifierTrainer",
 "SignatureRankerTrainer",
 "SignatureRegressorTrainer",
 "SignatureMultiOutputRegressorTrainer",
 "SignatureAnomalyDetectorTrainer",
 "SignatureClusteringTrainer"
 ]
 },
 "Desc": "Specifies the trainer kind, which determines the evaluator to be used.",
 "Required": true,
 "SortOrder": 7.0,
 "IsNullable": false,
 "Default": "SignatureBinaryClassifierTrainer"
 }
 ],
 "Outputs": [
 {
 "Name": "Warnings",
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
 },
 {
 "Name": "Models.CrossValidator",
 "Desc": "Cross validation for general learning",
Expand DownExpand Up@@ -1368,34 +1478,22 @@
 },
 {
 "Name": "Warnings",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Warning dataset"
 },
 {
 "Name": "OverallMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Overall metrics dataset"
 },
 {
 "Name": "PerInstanceMetrics",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Per instance metrics dataset"
 },
 {
 "Name": "ConfusionMatrix",
 "Type": {
 "Kind": "Array",
 "ItemType": "DataView"
 },
 "Type": "DataView",
 "Desc": "Confusion matrix dataset"
 }
 ]
Expand Down
Loading