Add new benchmarks to test\Microsoft.ML.Benchmarks - #722

Merged
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks
Aug 24, 2018
Merged

Add new benchmarks to test\Microsoft.ML.Benchmarks#722
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks

Conversation

@briancylui

@briancyluibriancylui commented Aug 23, 2018•
edited
Loading

Copy link
Copy Markdown

Submitting @yaeldekel's new benchmarks via this PR.

  • Added a new benchmark for KMeans and Logistic Regression (LR) under test\Microsoft.ML.Benchmarks
  • Added a new sentiment test inside the existing SDCA benchmark

cc: @yaeldekel@eerhardt

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/SentimentPredictionTests.cs Outdated
using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file can be reverted. No need to change this file.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are mostly refactoring that improves the style. There are actually multiple redundant usings here that I am going to remove in my next commit.

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this kind of style/clean up changes should not happen as part of another PR. It distracts from the real change in this PR, and it also clutters history (When some one looks at all the changes to this file, they see this change. Or when someone looks at this commit in the history, there is this needless change also included.)

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert the changes - thanks!

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
public void Setup()
{
s_dataPath = Program.GetDataPath("adult.train");
StochasticDualCoordinateAscentClassifierBench.s_metrics = Models.ClassificationMetrics.Empty;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like unfortunate coupling that I don't think we want. Can this be removed?

@Zruty0Zruty0Aug 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

In reply to: 212427929 [](ancestors = 212427929)

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
}
}

public class IrisData

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe IrisData and IrisPrediction are used. Can they be removed?

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs Outdated
{
// Pipeline
var loader = new TextLoader(env,
new TextLoader.Arguments()

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) this should be indented since it is a continuation of the line above.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also indented the subsequent lines since they are actually all inside one pair of parantheses, which may not be apparent from the current code style.

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadsrc/Microsoft.ML/Models/ClassificationMetrics.cs Outdated
/// </summary>
public sealed class ClassificationMetrics
{
public static ClassificationMetrics Empty = new ClassificationMetrics();

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be adding this public API in this PR.

Zruty0
Zruty0 approved these changes Aug 23, 2018
View reviewed changes

@Zruty0Zruty0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

eerhardt
eerhardt approved these changes Aug 23, 2018
View reviewed changes

@eerhardteerhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @briancylui

@eerhardt

eerhardt commented Aug 23, 2018

Copy link
Copy Markdown
Member

test OSX10.13 Debug
test OSX10.13 Release

@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

@eerhardt: dotnet build -c Release is fine, but dotnet run -c Release throws an exception. Deleting the two lines about s_metrics makes the GetValue(…) method in ClassificationMetricColumn not well-defined.

@eerhardteerhardt mentioned this pull request Aug 24, 2018
Closed
@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

Copying a relevant comment from #724:

Regarding the perf results posted on #724, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

@eerhardt

eerhardt commented Aug 24, 2018

Copy link
Copy Markdown
Member

@briancylui - do you think there are changes from #724 that should be brought over here?

@adamsitnik - I see the existing benchmark test has a bit of coupling between the Program.cs:ClassificationMetricsColumn and the StochasticDualCoordinateAscentClassifierBench test. Is it possible to have metrics columns that are specific to a set of tests? For example, the current test is a classification test, and those metrics only apply to that test. The new tests being added should have different metrics.

@Zruty0
Zruty0 merged commit 4fd8a9c into dotnet:masterAug 24, 2018
@adamsitnik

adamsitnik commented Aug 27, 2018

Copy link
Copy Markdown
Member

@briancylui@eerhardt I have solved this issue in #735

@ghostghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Ivanidzo4kaIvanidzo4kaAwaiting requested review from Ivanidzo4ka
@codemzscodemzsAwaiting requested review from codemzs
@TomFinleyTomFinleyAwaiting requested review from TomFinley
@justinormontjustinormontAwaiting requested review from justinormont
2 more reviewers
@eerhardteerhardteerhardt approved these changes
@Zruty0Zruty0Zruty0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@briancylui@eerhardt@adamsitnik@Zruty0@yaeldMS
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Add new benchmarks to test\Microsoft.ML.Benchmarks - #722

Merged
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks
Aug 24, 2018
Merged

Add new benchmarks to test\Microsoft.ML.Benchmarks#722
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks

Conversation

@briancylui

@briancyluibriancylui commented Aug 23, 2018•
edited
Loading

Copy link
Copy Markdown

Submitting @yaeldekel's new benchmarks via this PR.

  • Added a new benchmark for KMeans and Logistic Regression (LR) under test\Microsoft.ML.Benchmarks
  • Added a new sentiment test inside the existing SDCA benchmark

cc: @yaeldekel@eerhardt

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/SentimentPredictionTests.cs Outdated
using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file can be reverted. No need to change this file.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are mostly refactoring that improves the style. There are actually multiple redundant usings here that I am going to remove in my next commit.

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this kind of style/clean up changes should not happen as part of another PR. It distracts from the real change in this PR, and it also clutters history (When some one looks at all the changes to this file, they see this change. Or when someone looks at this commit in the history, there is this needless change also included.)

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert the changes - thanks!

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
public void Setup()
{
s_dataPath = Program.GetDataPath("adult.train");
StochasticDualCoordinateAscentClassifierBench.s_metrics = Models.ClassificationMetrics.Empty;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like unfortunate coupling that I don't think we want. Can this be removed?

@Zruty0Zruty0Aug 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

In reply to: 212427929 [](ancestors = 212427929)

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
}
}

public class IrisData

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe IrisData and IrisPrediction are used. Can they be removed?

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs Outdated
{
// Pipeline
var loader = new TextLoader(env,
new TextLoader.Arguments()

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) this should be indented since it is a continuation of the line above.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also indented the subsequent lines since they are actually all inside one pair of parantheses, which may not be apparent from the current code style.

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadsrc/Microsoft.ML/Models/ClassificationMetrics.cs Outdated
/// </summary>
public sealed class ClassificationMetrics
{
public static ClassificationMetrics Empty = new ClassificationMetrics();

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be adding this public API in this PR.

Zruty0
Zruty0 approved these changes Aug 23, 2018
View reviewed changes

@Zruty0Zruty0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

eerhardt
eerhardt approved these changes Aug 23, 2018
View reviewed changes

@eerhardteerhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @briancylui

@eerhardt

eerhardt commented Aug 23, 2018

Copy link
Copy Markdown
Member

test OSX10.13 Debug
test OSX10.13 Release

@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

@eerhardt: dotnet build -c Release is fine, but dotnet run -c Release throws an exception. Deleting the two lines about s_metrics makes the GetValue(…) method in ClassificationMetricColumn not well-defined.

@eerhardteerhardt mentioned this pull request Aug 24, 2018
Closed
@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

Copying a relevant comment from #724:

Regarding the perf results posted on #724, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

@eerhardt

eerhardt commented Aug 24, 2018

Copy link
Copy Markdown
Member

@briancylui - do you think there are changes from #724 that should be brought over here?

@adamsitnik - I see the existing benchmark test has a bit of coupling between the Program.cs:ClassificationMetricsColumn and the StochasticDualCoordinateAscentClassifierBench test. Is it possible to have metrics columns that are specific to a set of tests? For example, the current test is a classification test, and those metrics only apply to that test. The new tests being added should have different metrics.

@Zruty0
Zruty0 merged commit 4fd8a9c into dotnet:masterAug 24, 2018
@adamsitnik

adamsitnik commented Aug 27, 2018

Copy link
Copy Markdown
Member

@briancylui@eerhardt I have solved this issue in #735

@ghostghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Ivanidzo4kaIvanidzo4kaAwaiting requested review from Ivanidzo4ka
@codemzscodemzsAwaiting requested review from codemzs
@TomFinleyTomFinleyAwaiting requested review from TomFinley
@justinormontjustinormontAwaiting requested review from justinormont
2 more reviewers
@eerhardteerhardteerhardt approved these changes
@Zruty0Zruty0Zruty0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@briancylui@eerhardt@adamsitnik@Zruty0@yaeldMS
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add new benchmarks to test\Microsoft.ML.Benchmarks - #722

Merged
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks
Aug 24, 2018
Merged

Add new benchmarks to test\Microsoft.ML.Benchmarks#722
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks

Conversation

@briancylui

@briancyluibriancylui commented Aug 23, 2018•
edited
Loading

Copy link
Copy Markdown

Submitting @yaeldekel's new benchmarks via this PR.

  • Added a new benchmark for KMeans and Logistic Regression (LR) under test\Microsoft.ML.Benchmarks
  • Added a new sentiment test inside the existing SDCA benchmark

cc: @yaeldekel@eerhardt

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/SentimentPredictionTests.cs Outdated
using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file can be reverted. No need to change this file.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are mostly refactoring that improves the style. There are actually multiple redundant usings here that I am going to remove in my next commit.

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this kind of style/clean up changes should not happen as part of another PR. It distracts from the real change in this PR, and it also clutters history (When some one looks at all the changes to this file, they see this change. Or when someone looks at this commit in the history, there is this needless change also included.)

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert the changes - thanks!

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
public void Setup()
{
s_dataPath = Program.GetDataPath("adult.train");
StochasticDualCoordinateAscentClassifierBench.s_metrics = Models.ClassificationMetrics.Empty;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like unfortunate coupling that I don't think we want. Can this be removed?

@Zruty0Zruty0Aug 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

In reply to: 212427929 [](ancestors = 212427929)

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
}
}

public class IrisData

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe IrisData and IrisPrediction are used. Can they be removed?

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs Outdated
{
// Pipeline
var loader = new TextLoader(env,
new TextLoader.Arguments()

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) this should be indented since it is a continuation of the line above.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also indented the subsequent lines since they are actually all inside one pair of parantheses, which may not be apparent from the current code style.

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadsrc/Microsoft.ML/Models/ClassificationMetrics.cs Outdated
/// </summary>
public sealed class ClassificationMetrics
{
public static ClassificationMetrics Empty = new ClassificationMetrics();

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be adding this public API in this PR.

Zruty0
Zruty0 approved these changes Aug 23, 2018
View reviewed changes

@Zruty0Zruty0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

eerhardt
eerhardt approved these changes Aug 23, 2018
View reviewed changes

@eerhardteerhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @briancylui

@eerhardt

eerhardt commented Aug 23, 2018

Copy link
Copy Markdown
Member

test OSX10.13 Debug
test OSX10.13 Release

@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

@eerhardt: dotnet build -c Release is fine, but dotnet run -c Release throws an exception. Deleting the two lines about s_metrics makes the GetValue(…) method in ClassificationMetricColumn not well-defined.

@eerhardteerhardt mentioned this pull request Aug 24, 2018
Closed
@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

Copying a relevant comment from #724:

Regarding the perf results posted on #724, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

@eerhardt

eerhardt commented Aug 24, 2018

Copy link
Copy Markdown
Member

@briancylui - do you think there are changes from #724 that should be brought over here?

@adamsitnik - I see the existing benchmark test has a bit of coupling between the Program.cs:ClassificationMetricsColumn and the StochasticDualCoordinateAscentClassifierBench test. Is it possible to have metrics columns that are specific to a set of tests? For example, the current test is a classification test, and those metrics only apply to that test. The new tests being added should have different metrics.

@Zruty0
Zruty0 merged commit 4fd8a9c into dotnet:masterAug 24, 2018
@adamsitnik

adamsitnik commented Aug 27, 2018

Copy link
Copy Markdown
Member

@briancylui@eerhardt I have solved this issue in #735

@ghostghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Ivanidzo4kaIvanidzo4kaAwaiting requested review from Ivanidzo4ka
@codemzscodemzsAwaiting requested review from codemzs
@TomFinleyTomFinleyAwaiting requested review from TomFinley
@justinormontjustinormontAwaiting requested review from justinormont
2 more reviewers
@eerhardteerhardteerhardt approved these changes
@Zruty0Zruty0Zruty0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@briancylui@eerhardt@adamsitnik@Zruty0@yaeldMS
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add new benchmarks to test\Microsoft.ML.Benchmarks - #722

Merged
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks
Aug 24, 2018
Merged

Add new benchmarks to test\Microsoft.ML.Benchmarks#722
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks

Conversation

@briancylui

@briancyluibriancylui commented Aug 23, 2018•
edited
Loading

Copy link
Copy Markdown

Submitting @yaeldekel's new benchmarks via this PR.

  • Added a new benchmark for KMeans and Logistic Regression (LR) under test\Microsoft.ML.Benchmarks
  • Added a new sentiment test inside the existing SDCA benchmark

cc: @yaeldekel@eerhardt

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/SentimentPredictionTests.cs Outdated
using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file can be reverted. No need to change this file.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are mostly refactoring that improves the style. There are actually multiple redundant usings here that I am going to remove in my next commit.

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this kind of style/clean up changes should not happen as part of another PR. It distracts from the real change in this PR, and it also clutters history (When some one looks at all the changes to this file, they see this change. Or when someone looks at this commit in the history, there is this needless change also included.)

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert the changes - thanks!

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
public void Setup()
{
s_dataPath = Program.GetDataPath("adult.train");
StochasticDualCoordinateAscentClassifierBench.s_metrics = Models.ClassificationMetrics.Empty;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like unfortunate coupling that I don't think we want. Can this be removed?

@Zruty0Zruty0Aug 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

In reply to: 212427929 [](ancestors = 212427929)

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
}
}

public class IrisData

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe IrisData and IrisPrediction are used. Can they be removed?

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs Outdated
{
// Pipeline
var loader = new TextLoader(env,
new TextLoader.Arguments()

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) this should be indented since it is a continuation of the line above.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also indented the subsequent lines since they are actually all inside one pair of parantheses, which may not be apparent from the current code style.

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadsrc/Microsoft.ML/Models/ClassificationMetrics.cs Outdated
/// </summary>
public sealed class ClassificationMetrics
{
public static ClassificationMetrics Empty = new ClassificationMetrics();

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be adding this public API in this PR.

Zruty0
Zruty0 approved these changes Aug 23, 2018
View reviewed changes

@Zruty0Zruty0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

eerhardt
eerhardt approved these changes Aug 23, 2018
View reviewed changes

@eerhardteerhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @briancylui

@eerhardt

eerhardt commented Aug 23, 2018

Copy link
Copy Markdown
Member

test OSX10.13 Debug
test OSX10.13 Release

@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

@eerhardt: dotnet build -c Release is fine, but dotnet run -c Release throws an exception. Deleting the two lines about s_metrics makes the GetValue(…) method in ClassificationMetricColumn not well-defined.

@eerhardteerhardt mentioned this pull request Aug 24, 2018
Closed
@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

Copying a relevant comment from #724:

Regarding the perf results posted on #724, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

@eerhardt

eerhardt commented Aug 24, 2018

Copy link
Copy Markdown
Member

@briancylui - do you think there are changes from #724 that should be brought over here?

@adamsitnik - I see the existing benchmark test has a bit of coupling between the Program.cs:ClassificationMetricsColumn and the StochasticDualCoordinateAscentClassifierBench test. Is it possible to have metrics columns that are specific to a set of tests? For example, the current test is a classification test, and those metrics only apply to that test. The new tests being added should have different metrics.

@Zruty0
Zruty0 merged commit 4fd8a9c into dotnet:masterAug 24, 2018
@adamsitnik

adamsitnik commented Aug 27, 2018

Copy link
Copy Markdown
Member

@briancylui@eerhardt I have solved this issue in #735

@ghostghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Ivanidzo4kaIvanidzo4kaAwaiting requested review from Ivanidzo4ka
@codemzscodemzsAwaiting requested review from codemzs
@TomFinleyTomFinleyAwaiting requested review from TomFinley
@justinormontjustinormontAwaiting requested review from justinormont
2 more reviewers
@eerhardteerhardteerhardt approved these changes
@Zruty0Zruty0Zruty0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@briancylui@eerhardt@adamsitnik@Zruty0@yaeldMS
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Add new benchmarks to test\Microsoft.ML.Benchmarks - #722

Merged
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks
Aug 24, 2018
Merged

Add new benchmarks to test\Microsoft.ML.Benchmarks#722
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks

Conversation

@briancylui

@briancyluibriancylui commented Aug 23, 2018•
edited
Loading

Copy link
Copy Markdown

Submitting @yaeldekel's new benchmarks via this PR.

  • Added a new benchmark for KMeans and Logistic Regression (LR) under test\Microsoft.ML.Benchmarks
  • Added a new sentiment test inside the existing SDCA benchmark

cc: @yaeldekel@eerhardt

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/SentimentPredictionTests.cs Outdated
using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file can be reverted. No need to change this file.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are mostly refactoring that improves the style. There are actually multiple redundant usings here that I am going to remove in my next commit.

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this kind of style/clean up changes should not happen as part of another PR. It distracts from the real change in this PR, and it also clutters history (When some one looks at all the changes to this file, they see this change. Or when someone looks at this commit in the history, there is this needless change also included.)

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert the changes - thanks!

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
public void Setup()
{
s_dataPath = Program.GetDataPath("adult.train");
StochasticDualCoordinateAscentClassifierBench.s_metrics = Models.ClassificationMetrics.Empty;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like unfortunate coupling that I don't think we want. Can this be removed?

@Zruty0Zruty0Aug 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

In reply to: 212427929 [](ancestors = 212427929)

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
}
}

public class IrisData

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe IrisData and IrisPrediction are used. Can they be removed?

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs Outdated
{
// Pipeline
var loader = new TextLoader(env,
new TextLoader.Arguments()

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) this should be indented since it is a continuation of the line above.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also indented the subsequent lines since they are actually all inside one pair of parantheses, which may not be apparent from the current code style.

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadsrc/Microsoft.ML/Models/ClassificationMetrics.cs Outdated
/// </summary>
public sealed class ClassificationMetrics
{
public static ClassificationMetrics Empty = new ClassificationMetrics();

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be adding this public API in this PR.

Zruty0
Zruty0 approved these changes Aug 23, 2018
View reviewed changes

@Zruty0Zruty0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

eerhardt
eerhardt approved these changes Aug 23, 2018
View reviewed changes

@eerhardteerhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @briancylui

@eerhardt

eerhardt commented Aug 23, 2018

Copy link
Copy Markdown
Member

test OSX10.13 Debug
test OSX10.13 Release

@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

@eerhardt: dotnet build -c Release is fine, but dotnet run -c Release throws an exception. Deleting the two lines about s_metrics makes the GetValue(…) method in ClassificationMetricColumn not well-defined.

@eerhardteerhardt mentioned this pull request Aug 24, 2018
Closed
@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

Copying a relevant comment from #724:

Regarding the perf results posted on #724, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

@eerhardt

eerhardt commented Aug 24, 2018

Copy link
Copy Markdown
Member

@briancylui - do you think there are changes from #724 that should be brought over here?

@adamsitnik - I see the existing benchmark test has a bit of coupling between the Program.cs:ClassificationMetricsColumn and the StochasticDualCoordinateAscentClassifierBench test. Is it possible to have metrics columns that are specific to a set of tests? For example, the current test is a classification test, and those metrics only apply to that test. The new tests being added should have different metrics.

@Zruty0
Zruty0 merged commit 4fd8a9c into dotnet:masterAug 24, 2018
@adamsitnik

adamsitnik commented Aug 27, 2018

Copy link
Copy Markdown
Member

@briancylui@eerhardt I have solved this issue in #735

@ghostghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Ivanidzo4kaIvanidzo4kaAwaiting requested review from Ivanidzo4ka
@codemzscodemzsAwaiting requested review from codemzs
@TomFinleyTomFinleyAwaiting requested review from TomFinley
@justinormontjustinormontAwaiting requested review from justinormont
2 more reviewers
@eerhardteerhardteerhardt approved these changes
@Zruty0Zruty0Zruty0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@briancylui@eerhardt@adamsitnik@Zruty0@yaeldMS
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add new benchmarks to test\Microsoft.ML.Benchmarks - #722

Merged
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks
Aug 24, 2018
Merged

Add new benchmarks to test\Microsoft.ML.Benchmarks#722
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks

Conversation

@briancylui

@briancyluibriancylui commented Aug 23, 2018•
edited
Loading

Copy link
Copy Markdown

Submitting @yaeldekel's new benchmarks via this PR.

  • Added a new benchmark for KMeans and Logistic Regression (LR) under test\Microsoft.ML.Benchmarks
  • Added a new sentiment test inside the existing SDCA benchmark

cc: @yaeldekel@eerhardt

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/SentimentPredictionTests.cs Outdated
using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file can be reverted. No need to change this file.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are mostly refactoring that improves the style. There are actually multiple redundant usings here that I am going to remove in my next commit.

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this kind of style/clean up changes should not happen as part of another PR. It distracts from the real change in this PR, and it also clutters history (When some one looks at all the changes to this file, they see this change. Or when someone looks at this commit in the history, there is this needless change also included.)

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert the changes - thanks!

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
public void Setup()
{
s_dataPath = Program.GetDataPath("adult.train");
StochasticDualCoordinateAscentClassifierBench.s_metrics = Models.ClassificationMetrics.Empty;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like unfortunate coupling that I don't think we want. Can this be removed?

@Zruty0Zruty0Aug 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

In reply to: 212427929 [](ancestors = 212427929)

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
}
}

public class IrisData

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe IrisData and IrisPrediction are used. Can they be removed?

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs Outdated
{
// Pipeline
var loader = new TextLoader(env,
new TextLoader.Arguments()

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) this should be indented since it is a continuation of the line above.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also indented the subsequent lines since they are actually all inside one pair of parantheses, which may not be apparent from the current code style.

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadsrc/Microsoft.ML/Models/ClassificationMetrics.cs Outdated
/// </summary>
public sealed class ClassificationMetrics
{
public static ClassificationMetrics Empty = new ClassificationMetrics();

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be adding this public API in this PR.

Zruty0
Zruty0 approved these changes Aug 23, 2018
View reviewed changes

@Zruty0Zruty0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

eerhardt
eerhardt approved these changes Aug 23, 2018
View reviewed changes

@eerhardteerhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @briancylui

@eerhardt

eerhardt commented Aug 23, 2018

Copy link
Copy Markdown
Member

test OSX10.13 Debug
test OSX10.13 Release

@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

@eerhardt: dotnet build -c Release is fine, but dotnet run -c Release throws an exception. Deleting the two lines about s_metrics makes the GetValue(…) method in ClassificationMetricColumn not well-defined.

@eerhardteerhardt mentioned this pull request Aug 24, 2018
Closed
@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

Copying a relevant comment from #724:

Regarding the perf results posted on #724, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

@eerhardt

eerhardt commented Aug 24, 2018

Copy link
Copy Markdown
Member

@briancylui - do you think there are changes from #724 that should be brought over here?

@adamsitnik - I see the existing benchmark test has a bit of coupling between the Program.cs:ClassificationMetricsColumn and the StochasticDualCoordinateAscentClassifierBench test. Is it possible to have metrics columns that are specific to a set of tests? For example, the current test is a classification test, and those metrics only apply to that test. The new tests being added should have different metrics.

@Zruty0
Zruty0 merged commit 4fd8a9c into dotnet:masterAug 24, 2018
@adamsitnik

adamsitnik commented Aug 27, 2018

Copy link
Copy Markdown
Member

@briancylui@eerhardt I have solved this issue in #735

@ghostghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Ivanidzo4kaIvanidzo4kaAwaiting requested review from Ivanidzo4ka
@codemzscodemzsAwaiting requested review from codemzs
@TomFinleyTomFinleyAwaiting requested review from TomFinley
@justinormontjustinormontAwaiting requested review from justinormont
2 more reviewers
@eerhardteerhardteerhardt approved these changes
@Zruty0Zruty0Zruty0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@briancylui@eerhardt@adamsitnik@Zruty0@yaeldMS
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Add new benchmarks to test\Microsoft.ML.Benchmarks - #722

Merged
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks
Aug 24, 2018
Merged

Add new benchmarks to test\Microsoft.ML.Benchmarks#722
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks

Conversation

@briancylui

@briancyluibriancylui commented Aug 23, 2018•
edited
Loading

Copy link
Copy Markdown

Submitting @yaeldekel's new benchmarks via this PR.

  • Added a new benchmark for KMeans and Logistic Regression (LR) under test\Microsoft.ML.Benchmarks
  • Added a new sentiment test inside the existing SDCA benchmark

cc: @yaeldekel@eerhardt

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/SentimentPredictionTests.cs Outdated
using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file can be reverted. No need to change this file.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are mostly refactoring that improves the style. There are actually multiple redundant usings here that I am going to remove in my next commit.

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this kind of style/clean up changes should not happen as part of another PR. It distracts from the real change in this PR, and it also clutters history (When some one looks at all the changes to this file, they see this change. Or when someone looks at this commit in the history, there is this needless change also included.)

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert the changes - thanks!

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
public void Setup()
{
s_dataPath = Program.GetDataPath("adult.train");
StochasticDualCoordinateAscentClassifierBench.s_metrics = Models.ClassificationMetrics.Empty;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like unfortunate coupling that I don't think we want. Can this be removed?

@Zruty0Zruty0Aug 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

In reply to: 212427929 [](ancestors = 212427929)

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
}
}

public class IrisData

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe IrisData and IrisPrediction are used. Can they be removed?

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs Outdated
{
// Pipeline
var loader = new TextLoader(env,
new TextLoader.Arguments()

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) this should be indented since it is a continuation of the line above.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also indented the subsequent lines since they are actually all inside one pair of parantheses, which may not be apparent from the current code style.

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadsrc/Microsoft.ML/Models/ClassificationMetrics.cs Outdated
/// </summary>
public sealed class ClassificationMetrics
{
public static ClassificationMetrics Empty = new ClassificationMetrics();

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be adding this public API in this PR.

Zruty0
Zruty0 approved these changes Aug 23, 2018
View reviewed changes

@Zruty0Zruty0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

eerhardt
eerhardt approved these changes Aug 23, 2018
View reviewed changes

@eerhardteerhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @briancylui

@eerhardt

eerhardt commented Aug 23, 2018

Copy link
Copy Markdown
Member

test OSX10.13 Debug
test OSX10.13 Release

@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

@eerhardt: dotnet build -c Release is fine, but dotnet run -c Release throws an exception. Deleting the two lines about s_metrics makes the GetValue(…) method in ClassificationMetricColumn not well-defined.

@eerhardteerhardt mentioned this pull request Aug 24, 2018
Closed
@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

Copying a relevant comment from #724:

Regarding the perf results posted on #724, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

@eerhardt

eerhardt commented Aug 24, 2018

Copy link
Copy Markdown
Member

@briancylui - do you think there are changes from #724 that should be brought over here?

@adamsitnik - I see the existing benchmark test has a bit of coupling between the Program.cs:ClassificationMetricsColumn and the StochasticDualCoordinateAscentClassifierBench test. Is it possible to have metrics columns that are specific to a set of tests? For example, the current test is a classification test, and those metrics only apply to that test. The new tests being added should have different metrics.

@Zruty0
Zruty0 merged commit 4fd8a9c into dotnet:masterAug 24, 2018
@adamsitnik

adamsitnik commented Aug 27, 2018

Copy link
Copy Markdown
Member

@briancylui@eerhardt I have solved this issue in #735

@ghostghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Ivanidzo4kaIvanidzo4kaAwaiting requested review from Ivanidzo4ka
@codemzscodemzsAwaiting requested review from codemzs
@TomFinleyTomFinleyAwaiting requested review from TomFinley
@justinormontjustinormontAwaiting requested review from justinormont
2 more reviewers
@eerhardteerhardteerhardt approved these changes
@Zruty0Zruty0Zruty0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@briancylui@eerhardt@adamsitnik@Zruty0@yaeldMS
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Add new benchmarks to test\Microsoft.ML.Benchmarks - #722

Merged
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks
Aug 24, 2018
Merged

Add new benchmarks to test\Microsoft.ML.Benchmarks#722
Zruty0 merged 6 commits into
dotnet:masterfrom
briancylui:newBenchmarks

Conversation

@briancylui

@briancyluibriancylui commented Aug 23, 2018•
edited
Loading

Copy link
Copy Markdown

Submitting @yaeldekel's new benchmarks via this PR.

  • Added a new benchmark for KMeans and Logistic Regression (LR) under test\Microsoft.ML.Benchmarks
  • Added a new sentiment test inside the existing SDCA benchmark

cc: @yaeldekel@eerhardt

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Tests/ScenariosWithDirectInstantiation/SentimentPredictionTests.cs Outdated
using Microsoft.ML.Runtime.Data;
using Microsoft.ML.Runtime.FastTree;
using Microsoft.ML.Runtime.Internal.Calibration;
using Microsoft.ML.Runtime.Learners;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file can be reverted. No need to change this file.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes in this file are mostly refactoring that improves the style. There are actually multiple redundant usings here that I am going to remove in my next commit.

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, this kind of style/clean up changes should not happen as part of another PR. It distracts from the real change in this PR, and it also clutters history (When some one looks at all the changes to this file, they see this change. Or when someone looks at this commit in the history, there is this needless change also included.)

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will revert the changes - thanks!

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
public void Setup()
{
s_dataPath = Program.GetDataPath("adult.train");
StochasticDualCoordinateAscentClassifierBench.s_metrics = Models.ClassificationMetrics.Empty;

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like unfortunate coupling that I don't think we want. Can this be removed?

@Zruty0Zruty0Aug 23, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

In reply to: 212427929 [](ancestors = 212427929)

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/KMeansAndLogisticRegressionBench.cs Outdated
}
}

public class IrisData

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe IrisData and IrisPrediction are used. Can they be removed?

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadtest/Microsoft.ML.Benchmarks/StochasticDualCoordinateAscentClassifierBench.cs Outdated
{
// Pipeline
var loader = new TextLoader(env,
new TextLoader.Arguments()

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) this should be indented since it is a continuation of the line above.

@briancyluibriancyluiAug 23, 2018

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I also indented the subsequent lines since they are actually all inside one pair of parantheses, which may not be apparent from the current code style.

eerhardt
eerhardt reviewed Aug 23, 2018
View reviewed changes
Comment threadsrc/Microsoft.ML/Models/ClassificationMetrics.cs Outdated
/// </summary>
public sealed class ClassificationMetrics
{
public static ClassificationMetrics Empty = new ClassificationMetrics();

@eerhardteerhardtAug 23, 2018

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be adding this public API in this PR.

Zruty0
Zruty0 approved these changes Aug 23, 2018
View reviewed changes

@Zruty0Zruty0 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

eerhardt
eerhardt approved these changes Aug 23, 2018
View reviewed changes

@eerhardteerhardt left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @briancylui

@eerhardt

eerhardt commented Aug 23, 2018

Copy link
Copy Markdown
Member

test OSX10.13 Debug
test OSX10.13 Release

@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

@eerhardt: dotnet build -c Release is fine, but dotnet run -c Release throws an exception. Deleting the two lines about s_metrics makes the GetValue(…) method in ClassificationMetricColumn not well-defined.

@eerhardteerhardt mentioned this pull request Aug 24, 2018
Closed
@briancylui

briancylui commented Aug 24, 2018

Copy link
Copy Markdown
Author

Copying a relevant comment from #724:

Regarding the perf results posted on #724, KMeansAndLogisticRegression (KMeans+LR) shares the same AccuracyMacro with SDCA (0.98), but since the GetValue method of Program.cs:ClassificationMetricsColumn only references StochasticDualCoordinateAscentClassifierBench.s_metrics (link), which is irrelevant to KMeans+LR, it might be possible that KMeans+LR displayed SDCA's AccuracyMacro as its own metric in the perf results. When I ran dotnet run -c Release and chose KMeans+LR only, the AccuracyMacro was displayed as 0. It may have something to do with the added public variable ClassificationMetrics Empty.

@eerhardt

eerhardt commented Aug 24, 2018

Copy link
Copy Markdown
Member

@briancylui - do you think there are changes from #724 that should be brought over here?

@adamsitnik - I see the existing benchmark test has a bit of coupling between the Program.cs:ClassificationMetricsColumn and the StochasticDualCoordinateAscentClassifierBench test. Is it possible to have metrics columns that are specific to a set of tests? For example, the current test is a classification test, and those metrics only apply to that test. The new tests being added should have different metrics.

@Zruty0
Zruty0 merged commit 4fd8a9c into dotnet:masterAug 24, 2018
@adamsitnik

adamsitnik commented Aug 27, 2018

Copy link
Copy Markdown
Member

@briancylui@eerhardt I have solved this issue in #735

@ghostghost locked as resolved and limited conversation to collaborators Mar 29, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@Ivanidzo4kaIvanidzo4kaAwaiting requested review from Ivanidzo4ka
@codemzscodemzsAwaiting requested review from codemzs
@TomFinleyTomFinleyAwaiting requested review from TomFinley
@justinormontjustinormontAwaiting requested review from justinormont
2 more reviewers
@eerhardteerhardteerhardt approved these changes
@Zruty0Zruty0Zruty0 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@briancylui@eerhardt@adamsitnik@Zruty0@yaeldMS