Fix DataFrame to allow to store columns with size more than 2 Gb - #6710

Merged
JakeRadMSFT merged 5 commits into
dotnet:mainfrom
asmirnov82:6699_support_large_columns
Jul 6, 2023
Merged

Fix DataFrame to allow to store columns with size more than 2 Gb#6710
JakeRadMSFT merged 5 commits into
dotnet:mainfrom
asmirnov82:6699_support_large_columns

Conversation

@asmirnov82

@asmirnov82asmirnov82 commented May 25, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes#6699

  1. Decrease limit from Int.Max value on buffer size to the maximum amount of bytes supported by .Net Arrays
  2. Fix calculation of items amount for filling remaining capacity of fully allocated chunk
  3. Fix calculation of buffer size during default growing strategy (double the initial size)

@codecov

codecovBot commented May 25, 2023•
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #6710 (ee9cc4c) into main (184e661) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Additional details and impacted files
@@ Coverage Diff @@## main #6710 +/- ##
=======================================
Coverage 68.87% 68.88% =======================================
Files 1216 1216 Lines 250915 250916 +1 Branches 26259 26259 =======================================
+ Hits 172825 172841 +16 + Misses 71265 71258 -7 + Partials 6825 6817 -8
FlagCoverage Δ
Debug68.88% <100.00%> (+<0.01%)⬆️
production63.38% <100.00%> (+<0.01%)⬆️
test88.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted FilesCoverage Δ
test/Microsoft.Data.Analysis.Tests/BufferTests.cs100.00% <ø> (ø)
src/Microsoft.Data.Analysis/DataFrameBuffer.cs82.97% <100.00%> (+0.37%)⬆️
...icrosoft.Data.Analysis/PrimitiveColumnContainer.cs83.04% <100.00%> (ø)
...Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs47.36% <100.00%> (ø)

... and 7 files with indirect coverage changes

michaelgsharp
michaelgsharp approved these changes Jun 1, 2023
View reviewed changes

@michaelgsharpmichaelgsharp left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@torronentorronen mentioned this pull request Jun 3, 2023
Closed
@michaelgsharp

michaelgsharp commented Jun 13, 2023

Copy link
Copy Markdown
Contributor

/azp run

@azure-pipelines

azure-pipelinesBot commented Jun 13, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

@asmirnov82

asmirnov82 commented Jun 22, 2023

Copy link
Copy Markdown
ContributorAuthor

/azp run

@azure-pipelines

azure-pipelinesBot commented Jun 22, 2023

Copy link
Copy Markdown
Commenter does not have sufficient privileges for PR 6710 in repo dotnet/machinelearning

@asmirnov82

asmirnov82 commented Jun 22, 2023

Copy link
Copy Markdown
ContributorAuthor

@michaelgsharp, tests are failing due to "Unable to load DLL 'lib_lightgbm' or one of its dependencies: The specified module could not be found. (0x8007007E)" in ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2. It doesn't look to be somehow related to changes done in this PR. More over, I see that the same error happens from time to time during build in many other my PRs. Is it possible to disable this test temporaly (until the fix for the test is found)?

@JakeRadMSFT

JakeRadMSFT commented Jun 23, 2023

Copy link
Copy Markdown
Member

/azp run

@azure-pipelines

azure-pipelinesBot commented Jun 23, 2023

Copy link
Copy Markdown
Azure Pipelines successfully started running 2 pipeline(s).

JakeRadMSFT
JakeRadMSFT reviewed Jul 6, 2023
View reviewed changes
Comment threadtest/Microsoft.Data.Analysis.Tests/BufferTests.cs
Assert.Null(clone[i]);
}

/* Don't run tests during build as they fail, because build if build machine doesn't have enought memory

@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 are you sure this is the reason?

Can we add a flag so these still run locally?

@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to do a #if and then set a flag in build definition

@JakeRadMSFTJakeRadMSFTJul 6, 2023•
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 can you open an issue. I don't want to block on this.

@tannergooding said this might actually be a netfx thing? There might need to be something set to support > 2gb. Comment below.

"For the more than 2GB one, it will be allowed but will then fail on .NET Framework anyways due to GC limitations

You have to opt in with gcAllowVeryLargeObjects iirc, and that's only for 64-bit"

@asmirnov82asmirnov82Jul 7, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakeRadMSFT I fixed tests using X64FactAttribute. Please take a look at #6758 and #6759

@JakeRadMSFT
JakeRadMSFT merged commit 26c2446 into dotnet:mainJul 6, 2023
@asmirnov82
asmirnov82 deleted the 6699_support_large_columns branch July 7, 2023 05:48
@asmirnov82asmirnov82 mentioned this pull request Jul 7, 2023
Merged
@ghostghost locked as resolved and limited conversation to collaborators Aug 6, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@JakeRadMSFTJakeRadMSFTJakeRadMSFT left review comments
+1 more reviewer
@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

community-contribution

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

DataFrame crashes on attempt to create column with size higher than 2,147,483,591 bytes

3 participants

@asmirnov82@michaelgsharp@JakeRadMSFT
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Fix DataFrame to allow to store columns with size more than 2 Gb - #6710
Merged
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Jul 6, 2023
Merged
Fix DataFrame to allow to store columns with size more than 2 Gb#6710
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Conversation
@asmirnov82
@asmirnov82asmirnov82
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#6699
  1. Decrease limit from Int.Max value on buffer size to the maximum amount of bytes supported by .Net Arrays
  2. Fix calculation of items amount for filling remaining capacity of fully allocated chunk
  3. Fix calculation of buffer size during default growing strategy (double the initial size)
@codecov
codecovBot
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #6710 (ee9cc4c) into main (184e661) will increase coverage by 0.00%.

The diff coverage is 100.00%.
Additional details and impacted files
@@ Coverage Diff @@## main #6710 +/- ##
=======================================
Coverage 68.87% 68.88% =======================================
Files 1216 1216 Lines 250915 250916 +1 Branches 26259 26259 =======================================
+ Hits 172825 172841 +16 + Misses 71265 71258 -7 + Partials 6825 6817 -8 
FlagCoverage Δ
Debug68.88% <100.00%> (+<0.01%)⬆️
production63.38% <100.00%> (+<0.01%)⬆️
test88.90% <ø> (ø)
Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted FilesCoverage Δ
test/Microsoft.Data.Analysis.Tests/BufferTests.cs100.00% <ø> (ø)
src/Microsoft.Data.Analysis/DataFrameBuffer.cs82.97% <100.00%> (+0.37%)⬆️
...icrosoft.Data.Analysis/PrimitiveColumnContainer.cs83.04% <100.00%> (ø)
...Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs47.36% <100.00%> (ø)
... and 7 files with indirect coverage changes
michaelgsharp
michaelgsharp
approved these changes
Jun 1, 2023
View reviewed changes
@michaelgsharpmichaelgsharp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@torronentorronen
mentioned this pull request
Jun 3, 2023

Closed

@michaelgsharp
michaelgsharp
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 13, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 22, 2023

Copy link


Copy Markdown


Commenter does not have sufficient privileges for PR 6710 in repo dotnet/machinelearning
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
@michaelgsharp, tests are failing due to "Unable to load DLL 'lib_lightgbm' or one of its dependencies: The specified module could not be found. (0x8007007E)" in ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2. It doesn't look to be somehow related to changes done in this PR. More over, I see that the same error happens from time to time during build in many other my PRs. Is it possible to disable this test temporaly (until the fix for the test is found)?
@JakeRadMSFT
JakeRadMSFT
commented
Jun 23, 2023

Copy link


Copy Markdown

Member
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
JakeRadMSFT
JakeRadMSFT
reviewed
Jul 6, 2023
View reviewed changes
Comment threadtest/Microsoft.Data.Analysis.Tests/BufferTests.cs
 Assert.Null(clone[i]);
 }

 /* Don't run tests during build as they fail, because build if build machine doesn't have enought memory
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 are you sure this is the reason?
Can we add a flag so these still run locally?
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to do a #if and then set a flag in build definition
@JakeRadMSFTJakeRadMSFTJul 6, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 can you open an issue. I don't want to block on this.
@tannergooding said this might actually be a netfx thing? There might need to be something set to support > 2gb. Comment below.
"For the more than 2GB one, it will be allowed but will then fail on .NET Framework anyways due to GC limitations
You have to opt in with gcAllowVeryLargeObjects iirc, and that's only for 64-bit"
@asmirnov82asmirnov82Jul 7, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakeRadMSFT I fixed tests using X64FactAttribute. Please take a look at #6758 and #6759
@JakeRadMSFT
JakeRadMSFT
merged commit 26c2446
into
dotnet:mainJul 6, 2023
@asmirnov82
asmirnov82
deleted the

6699_support_large_columns

branch
July 7, 2023 05:48
@asmirnov82asmirnov82
mentioned this pull request
Jul 7, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Aug 6, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@JakeRadMSFTJakeRadMSFTJakeRadMSFT left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

community-contribution

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

DataFrame crashes on attempt to create column with size higher than 2,147,483,591 bytes


3 participants

@asmirnov82@michaelgsharp@JakeRadMSFT

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix DataFrame to allow to store columns with size more than 2 Gb - #6710
Merged
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Jul 6, 2023
Merged
Fix DataFrame to allow to store columns with size more than 2 Gb#6710
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Conversation
@asmirnov82
@asmirnov82asmirnov82
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#6699
  1. Decrease limit from Int.Max value on buffer size to the maximum amount of bytes supported by .Net Arrays
  2. Fix calculation of items amount for filling remaining capacity of fully allocated chunk
  3. Fix calculation of buffer size during default growing strategy (double the initial size)
@codecov
codecovBot
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #6710 (ee9cc4c) into main (184e661) will increase coverage by 0.00%.

The diff coverage is 100.00%.
Additional details and impacted files
@@ Coverage Diff @@## main #6710 +/- ##
=======================================
Coverage 68.87% 68.88% =======================================
Files 1216 1216 Lines 250915 250916 +1 Branches 26259 26259 =======================================
+ Hits 172825 172841 +16 + Misses 71265 71258 -7 + Partials 6825 6817 -8 
FlagCoverage Δ
Debug68.88% <100.00%> (+<0.01%)⬆️
production63.38% <100.00%> (+<0.01%)⬆️
test88.90% <ø> (ø)
Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted FilesCoverage Δ
test/Microsoft.Data.Analysis.Tests/BufferTests.cs100.00% <ø> (ø)
src/Microsoft.Data.Analysis/DataFrameBuffer.cs82.97% <100.00%> (+0.37%)⬆️
...icrosoft.Data.Analysis/PrimitiveColumnContainer.cs83.04% <100.00%> (ø)
...Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs47.36% <100.00%> (ø)
... and 7 files with indirect coverage changes
michaelgsharp
michaelgsharp
approved these changes
Jun 1, 2023
View reviewed changes
@michaelgsharpmichaelgsharp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@torronentorronen
mentioned this pull request
Jun 3, 2023

Closed

@michaelgsharp
michaelgsharp
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 13, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 22, 2023

Copy link


Copy Markdown


Commenter does not have sufficient privileges for PR 6710 in repo dotnet/machinelearning
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
@michaelgsharp, tests are failing due to "Unable to load DLL 'lib_lightgbm' or one of its dependencies: The specified module could not be found. (0x8007007E)" in ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2. It doesn't look to be somehow related to changes done in this PR. More over, I see that the same error happens from time to time during build in many other my PRs. Is it possible to disable this test temporaly (until the fix for the test is found)?
@JakeRadMSFT
JakeRadMSFT
commented
Jun 23, 2023

Copy link


Copy Markdown

Member
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
JakeRadMSFT
JakeRadMSFT
reviewed
Jul 6, 2023
View reviewed changes
Comment threadtest/Microsoft.Data.Analysis.Tests/BufferTests.cs
 Assert.Null(clone[i]);
 }

 /* Don't run tests during build as they fail, because build if build machine doesn't have enought memory
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 are you sure this is the reason?
Can we add a flag so these still run locally?
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to do a #if and then set a flag in build definition
@JakeRadMSFTJakeRadMSFTJul 6, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 can you open an issue. I don't want to block on this.
@tannergooding said this might actually be a netfx thing? There might need to be something set to support > 2gb. Comment below.
"For the more than 2GB one, it will be allowed but will then fail on .NET Framework anyways due to GC limitations
You have to opt in with gcAllowVeryLargeObjects iirc, and that's only for 64-bit"
@asmirnov82asmirnov82Jul 7, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakeRadMSFT I fixed tests using X64FactAttribute. Please take a look at #6758 and #6759
@JakeRadMSFT
JakeRadMSFT
merged commit 26c2446
into
dotnet:mainJul 6, 2023
@asmirnov82
asmirnov82
deleted the

6699_support_large_columns

branch
July 7, 2023 05:48
@asmirnov82asmirnov82
mentioned this pull request
Jul 7, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Aug 6, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@JakeRadMSFTJakeRadMSFTJakeRadMSFT left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

community-contribution

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

DataFrame crashes on attempt to create column with size higher than 2,147,483,591 bytes


3 participants

@asmirnov82@michaelgsharp@JakeRadMSFT

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix DataFrame to allow to store columns with size more than 2 Gb - #6710
Merged
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Jul 6, 2023
Merged
Fix DataFrame to allow to store columns with size more than 2 Gb#6710
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Conversation
@asmirnov82
@asmirnov82asmirnov82
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#6699
  1. Decrease limit from Int.Max value on buffer size to the maximum amount of bytes supported by .Net Arrays
  2. Fix calculation of items amount for filling remaining capacity of fully allocated chunk
  3. Fix calculation of buffer size during default growing strategy (double the initial size)
@codecov
codecovBot
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #6710 (ee9cc4c) into main (184e661) will increase coverage by 0.00%.

The diff coverage is 100.00%.
Additional details and impacted files
@@ Coverage Diff @@## main #6710 +/- ##
=======================================
Coverage 68.87% 68.88% =======================================
Files 1216 1216 Lines 250915 250916 +1 Branches 26259 26259 =======================================
+ Hits 172825 172841 +16 + Misses 71265 71258 -7 + Partials 6825 6817 -8 
FlagCoverage Δ
Debug68.88% <100.00%> (+<0.01%)⬆️
production63.38% <100.00%> (+<0.01%)⬆️
test88.90% <ø> (ø)
Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted FilesCoverage Δ
test/Microsoft.Data.Analysis.Tests/BufferTests.cs100.00% <ø> (ø)
src/Microsoft.Data.Analysis/DataFrameBuffer.cs82.97% <100.00%> (+0.37%)⬆️
...icrosoft.Data.Analysis/PrimitiveColumnContainer.cs83.04% <100.00%> (ø)
...Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs47.36% <100.00%> (ø)
... and 7 files with indirect coverage changes
michaelgsharp
michaelgsharp
approved these changes
Jun 1, 2023
View reviewed changes
@michaelgsharpmichaelgsharp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@torronentorronen
mentioned this pull request
Jun 3, 2023

Closed

@michaelgsharp
michaelgsharp
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 13, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 22, 2023

Copy link


Copy Markdown


Commenter does not have sufficient privileges for PR 6710 in repo dotnet/machinelearning
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
@michaelgsharp, tests are failing due to "Unable to load DLL 'lib_lightgbm' or one of its dependencies: The specified module could not be found. (0x8007007E)" in ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2. It doesn't look to be somehow related to changes done in this PR. More over, I see that the same error happens from time to time during build in many other my PRs. Is it possible to disable this test temporaly (until the fix for the test is found)?
@JakeRadMSFT
JakeRadMSFT
commented
Jun 23, 2023

Copy link


Copy Markdown

Member
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
JakeRadMSFT
JakeRadMSFT
reviewed
Jul 6, 2023
View reviewed changes
Comment threadtest/Microsoft.Data.Analysis.Tests/BufferTests.cs
 Assert.Null(clone[i]);
 }

 /* Don't run tests during build as they fail, because build if build machine doesn't have enought memory
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 are you sure this is the reason?
Can we add a flag so these still run locally?
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to do a #if and then set a flag in build definition
@JakeRadMSFTJakeRadMSFTJul 6, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 can you open an issue. I don't want to block on this.
@tannergooding said this might actually be a netfx thing? There might need to be something set to support > 2gb. Comment below.
"For the more than 2GB one, it will be allowed but will then fail on .NET Framework anyways due to GC limitations
You have to opt in with gcAllowVeryLargeObjects iirc, and that's only for 64-bit"
@asmirnov82asmirnov82Jul 7, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakeRadMSFT I fixed tests using X64FactAttribute. Please take a look at #6758 and #6759
@JakeRadMSFT
JakeRadMSFT
merged commit 26c2446
into
dotnet:mainJul 6, 2023
@asmirnov82
asmirnov82
deleted the

6699_support_large_columns

branch
July 7, 2023 05:48
@asmirnov82asmirnov82
mentioned this pull request
Jul 7, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Aug 6, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@JakeRadMSFTJakeRadMSFTJakeRadMSFT left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

community-contribution

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

DataFrame crashes on attempt to create column with size higher than 2,147,483,591 bytes


3 participants

@asmirnov82@michaelgsharp@JakeRadMSFT

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Fix DataFrame to allow to store columns with size more than 2 Gb - #6710
Merged
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Jul 6, 2023
Merged
Fix DataFrame to allow to store columns with size more than 2 Gb#6710
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Conversation
@asmirnov82
@asmirnov82asmirnov82
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#6699
  1. Decrease limit from Int.Max value on buffer size to the maximum amount of bytes supported by .Net Arrays
  2. Fix calculation of items amount for filling remaining capacity of fully allocated chunk
  3. Fix calculation of buffer size during default growing strategy (double the initial size)
@codecov
codecovBot
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #6710 (ee9cc4c) into main (184e661) will increase coverage by 0.00%.

The diff coverage is 100.00%.
Additional details and impacted files
@@ Coverage Diff @@## main #6710 +/- ##
=======================================
Coverage 68.87% 68.88% =======================================
Files 1216 1216 Lines 250915 250916 +1 Branches 26259 26259 =======================================
+ Hits 172825 172841 +16 + Misses 71265 71258 -7 + Partials 6825 6817 -8 
FlagCoverage Δ
Debug68.88% <100.00%> (+<0.01%)⬆️
production63.38% <100.00%> (+<0.01%)⬆️
test88.90% <ø> (ø)
Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted FilesCoverage Δ
test/Microsoft.Data.Analysis.Tests/BufferTests.cs100.00% <ø> (ø)
src/Microsoft.Data.Analysis/DataFrameBuffer.cs82.97% <100.00%> (+0.37%)⬆️
...icrosoft.Data.Analysis/PrimitiveColumnContainer.cs83.04% <100.00%> (ø)
...Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs47.36% <100.00%> (ø)
... and 7 files with indirect coverage changes
michaelgsharp
michaelgsharp
approved these changes
Jun 1, 2023
View reviewed changes
@michaelgsharpmichaelgsharp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@torronentorronen
mentioned this pull request
Jun 3, 2023

Closed

@michaelgsharp
michaelgsharp
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 13, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 22, 2023

Copy link


Copy Markdown


Commenter does not have sufficient privileges for PR 6710 in repo dotnet/machinelearning
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
@michaelgsharp, tests are failing due to "Unable to load DLL 'lib_lightgbm' or one of its dependencies: The specified module could not be found. (0x8007007E)" in ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2. It doesn't look to be somehow related to changes done in this PR. More over, I see that the same error happens from time to time during build in many other my PRs. Is it possible to disable this test temporaly (until the fix for the test is found)?
@JakeRadMSFT
JakeRadMSFT
commented
Jun 23, 2023

Copy link


Copy Markdown

Member
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
JakeRadMSFT
JakeRadMSFT
reviewed
Jul 6, 2023
View reviewed changes
Comment threadtest/Microsoft.Data.Analysis.Tests/BufferTests.cs
 Assert.Null(clone[i]);
 }

 /* Don't run tests during build as they fail, because build if build machine doesn't have enought memory
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 are you sure this is the reason?
Can we add a flag so these still run locally?
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to do a #if and then set a flag in build definition
@JakeRadMSFTJakeRadMSFTJul 6, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 can you open an issue. I don't want to block on this.
@tannergooding said this might actually be a netfx thing? There might need to be something set to support > 2gb. Comment below.
"For the more than 2GB one, it will be allowed but will then fail on .NET Framework anyways due to GC limitations
You have to opt in with gcAllowVeryLargeObjects iirc, and that's only for 64-bit"
@asmirnov82asmirnov82Jul 7, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakeRadMSFT I fixed tests using X64FactAttribute. Please take a look at #6758 and #6759
@JakeRadMSFT
JakeRadMSFT
merged commit 26c2446
into
dotnet:mainJul 6, 2023
@asmirnov82
asmirnov82
deleted the

6699_support_large_columns

branch
July 7, 2023 05:48
@asmirnov82asmirnov82
mentioned this pull request
Jul 7, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Aug 6, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@JakeRadMSFTJakeRadMSFTJakeRadMSFT left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

community-contribution

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

DataFrame crashes on attempt to create column with size higher than 2,147,483,591 bytes


3 participants

@asmirnov82@michaelgsharp@JakeRadMSFT

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix DataFrame to allow to store columns with size more than 2 Gb - #6710
Merged
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Jul 6, 2023
Merged
Fix DataFrame to allow to store columns with size more than 2 Gb#6710
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Conversation
@asmirnov82
@asmirnov82asmirnov82
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#6699
  1. Decrease limit from Int.Max value on buffer size to the maximum amount of bytes supported by .Net Arrays
  2. Fix calculation of items amount for filling remaining capacity of fully allocated chunk
  3. Fix calculation of buffer size during default growing strategy (double the initial size)
@codecov
codecovBot
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #6710 (ee9cc4c) into main (184e661) will increase coverage by 0.00%.

The diff coverage is 100.00%.
Additional details and impacted files
@@ Coverage Diff @@## main #6710 +/- ##
=======================================
Coverage 68.87% 68.88% =======================================
Files 1216 1216 Lines 250915 250916 +1 Branches 26259 26259 =======================================
+ Hits 172825 172841 +16 + Misses 71265 71258 -7 + Partials 6825 6817 -8 
FlagCoverage Δ
Debug68.88% <100.00%> (+<0.01%)⬆️
production63.38% <100.00%> (+<0.01%)⬆️
test88.90% <ø> (ø)
Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted FilesCoverage Δ
test/Microsoft.Data.Analysis.Tests/BufferTests.cs100.00% <ø> (ø)
src/Microsoft.Data.Analysis/DataFrameBuffer.cs82.97% <100.00%> (+0.37%)⬆️
...icrosoft.Data.Analysis/PrimitiveColumnContainer.cs83.04% <100.00%> (ø)
...Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs47.36% <100.00%> (ø)
... and 7 files with indirect coverage changes
michaelgsharp
michaelgsharp
approved these changes
Jun 1, 2023
View reviewed changes
@michaelgsharpmichaelgsharp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@torronentorronen
mentioned this pull request
Jun 3, 2023

Closed

@michaelgsharp
michaelgsharp
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 13, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 22, 2023

Copy link


Copy Markdown


Commenter does not have sufficient privileges for PR 6710 in repo dotnet/machinelearning
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
@michaelgsharp, tests are failing due to "Unable to load DLL 'lib_lightgbm' or one of its dependencies: The specified module could not be found. (0x8007007E)" in ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2. It doesn't look to be somehow related to changes done in this PR. More over, I see that the same error happens from time to time during build in many other my PRs. Is it possible to disable this test temporaly (until the fix for the test is found)?
@JakeRadMSFT
JakeRadMSFT
commented
Jun 23, 2023

Copy link


Copy Markdown

Member
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
JakeRadMSFT
JakeRadMSFT
reviewed
Jul 6, 2023
View reviewed changes
Comment threadtest/Microsoft.Data.Analysis.Tests/BufferTests.cs
 Assert.Null(clone[i]);
 }

 /* Don't run tests during build as they fail, because build if build machine doesn't have enought memory
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 are you sure this is the reason?
Can we add a flag so these still run locally?
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to do a #if and then set a flag in build definition
@JakeRadMSFTJakeRadMSFTJul 6, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 can you open an issue. I don't want to block on this.
@tannergooding said this might actually be a netfx thing? There might need to be something set to support > 2gb. Comment below.
"For the more than 2GB one, it will be allowed but will then fail on .NET Framework anyways due to GC limitations
You have to opt in with gcAllowVeryLargeObjects iirc, and that's only for 64-bit"
@asmirnov82asmirnov82Jul 7, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakeRadMSFT I fixed tests using X64FactAttribute. Please take a look at #6758 and #6759
@JakeRadMSFT
JakeRadMSFT
merged commit 26c2446
into
dotnet:mainJul 6, 2023
@asmirnov82
asmirnov82
deleted the

6699_support_large_columns

branch
July 7, 2023 05:48
@asmirnov82asmirnov82
mentioned this pull request
Jul 7, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Aug 6, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@JakeRadMSFTJakeRadMSFTJakeRadMSFT left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

community-contribution

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

DataFrame crashes on attempt to create column with size higher than 2,147,483,591 bytes


3 participants

@asmirnov82@michaelgsharp@JakeRadMSFT

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Fix DataFrame to allow to store columns with size more than 2 Gb - #6710
Merged
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Jul 6, 2023
Merged
Fix DataFrame to allow to store columns with size more than 2 Gb#6710
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Conversation
@asmirnov82
@asmirnov82asmirnov82
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#6699
  1. Decrease limit from Int.Max value on buffer size to the maximum amount of bytes supported by .Net Arrays
  2. Fix calculation of items amount for filling remaining capacity of fully allocated chunk
  3. Fix calculation of buffer size during default growing strategy (double the initial size)
@codecov
codecovBot
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #6710 (ee9cc4c) into main (184e661) will increase coverage by 0.00%.

The diff coverage is 100.00%.
Additional details and impacted files
@@ Coverage Diff @@## main #6710 +/- ##
=======================================
Coverage 68.87% 68.88% =======================================
Files 1216 1216 Lines 250915 250916 +1 Branches 26259 26259 =======================================
+ Hits 172825 172841 +16 + Misses 71265 71258 -7 + Partials 6825 6817 -8 
FlagCoverage Δ
Debug68.88% <100.00%> (+<0.01%)⬆️
production63.38% <100.00%> (+<0.01%)⬆️
test88.90% <ø> (ø)
Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted FilesCoverage Δ
test/Microsoft.Data.Analysis.Tests/BufferTests.cs100.00% <ø> (ø)
src/Microsoft.Data.Analysis/DataFrameBuffer.cs82.97% <100.00%> (+0.37%)⬆️
...icrosoft.Data.Analysis/PrimitiveColumnContainer.cs83.04% <100.00%> (ø)
...Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs47.36% <100.00%> (ø)
... and 7 files with indirect coverage changes
michaelgsharp
michaelgsharp
approved these changes
Jun 1, 2023
View reviewed changes
@michaelgsharpmichaelgsharp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@torronentorronen
mentioned this pull request
Jun 3, 2023

Closed

@michaelgsharp
michaelgsharp
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 13, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 22, 2023

Copy link


Copy Markdown


Commenter does not have sufficient privileges for PR 6710 in repo dotnet/machinelearning
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
@michaelgsharp, tests are failing due to "Unable to load DLL 'lib_lightgbm' or one of its dependencies: The specified module could not be found. (0x8007007E)" in ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2. It doesn't look to be somehow related to changes done in this PR. More over, I see that the same error happens from time to time during build in many other my PRs. Is it possible to disable this test temporaly (until the fix for the test is found)?
@JakeRadMSFT
JakeRadMSFT
commented
Jun 23, 2023

Copy link


Copy Markdown

Member
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
JakeRadMSFT
JakeRadMSFT
reviewed
Jul 6, 2023
View reviewed changes
Comment threadtest/Microsoft.Data.Analysis.Tests/BufferTests.cs
 Assert.Null(clone[i]);
 }

 /* Don't run tests during build as they fail, because build if build machine doesn't have enought memory
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 are you sure this is the reason?
Can we add a flag so these still run locally?
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to do a #if and then set a flag in build definition
@JakeRadMSFTJakeRadMSFTJul 6, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 can you open an issue. I don't want to block on this.
@tannergooding said this might actually be a netfx thing? There might need to be something set to support > 2gb. Comment below.
"For the more than 2GB one, it will be allowed but will then fail on .NET Framework anyways due to GC limitations
You have to opt in with gcAllowVeryLargeObjects iirc, and that's only for 64-bit"
@asmirnov82asmirnov82Jul 7, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakeRadMSFT I fixed tests using X64FactAttribute. Please take a look at #6758 and #6759
@JakeRadMSFT
JakeRadMSFT
merged commit 26c2446
into
dotnet:mainJul 6, 2023
@asmirnov82
asmirnov82
deleted the

6699_support_large_columns

branch
July 7, 2023 05:48
@asmirnov82asmirnov82
mentioned this pull request
Jul 7, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Aug 6, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@JakeRadMSFTJakeRadMSFTJakeRadMSFT left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

community-contribution

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

DataFrame crashes on attempt to create column with size higher than 2,147,483,591 bytes


3 participants

@asmirnov82@michaelgsharp@JakeRadMSFT

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Fix DataFrame to allow to store columns with size more than 2 Gb - #6710
Merged
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Jul 6, 2023
Merged
Fix DataFrame to allow to store columns with size more than 2 Gb#6710
JakeRadMSFT merged 5 commits into
dotnet:mainfrom 
asmirnov82:6699_support_large_columns
Conversation
@asmirnov82
@asmirnov82asmirnov82
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Fixes#6699
  1. Decrease limit from Int.Max value on buffer size to the maximum amount of bytes supported by .Net Arrays
  2. Fix calculation of items amount for filling remaining capacity of fully allocated chunk
  3. Fix calculation of buffer size during default growing strategy (double the initial size)
@codecov
codecovBot
commented
May 25, 2023•

edited

Loading

Copy link


Copy Markdown

Codecov Report
Merging #6710 (ee9cc4c) into main (184e661) will increase coverage by 0.00%.

The diff coverage is 100.00%.
Additional details and impacted files
@@ Coverage Diff @@## main #6710 +/- ##
=======================================
Coverage 68.87% 68.88% =======================================
Files 1216 1216 Lines 250915 250916 +1 Branches 26259 26259 =======================================
+ Hits 172825 172841 +16 + Misses 71265 71258 -7 + Partials 6825 6817 -8 
FlagCoverage Δ
Debug68.88% <100.00%> (+<0.01%)⬆️
production63.38% <100.00%> (+<0.01%)⬆️
test88.90% <ø> (ø)
Flags with carried forward coverage won't be shown. Click here to find out more.
Impacted FilesCoverage Δ
test/Microsoft.Data.Analysis.Tests/BufferTests.cs100.00% <ø> (ø)
src/Microsoft.Data.Analysis/DataFrameBuffer.cs82.97% <100.00%> (+0.37%)⬆️
...icrosoft.Data.Analysis/PrimitiveColumnContainer.cs83.04% <100.00%> (ø)
...Microsoft.Data.Analysis/ReadOnlyDataFrameBuffer.cs47.36% <100.00%> (ø)
... and 7 files with indirect coverage changes
michaelgsharp
michaelgsharp
approved these changes
Jun 1, 2023
View reviewed changes
@michaelgsharpmichaelgsharp
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
@torronentorronen
mentioned this pull request
Jun 3, 2023

Closed

@michaelgsharp
michaelgsharp
commented
Jun 13, 2023

Copy link


Copy Markdown

Contributor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 13, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 22, 2023

Copy link


Copy Markdown


Commenter does not have sufficient privileges for PR 6710 in repo dotnet/machinelearning
@asmirnov82
asmirnov82
commented
Jun 22, 2023

Copy link


Copy Markdown

ContributorAuthor
@michaelgsharp, tests are failing due to "Unable to load DLL 'lib_lightgbm' or one of its dependencies: The specified module could not be found. (0x8007007E)" in ML.Fairlearn.Tests.GridSearchTest.TestGridSearchTrialRunner2. It doesn't look to be somehow related to changes done in this PR. More over, I see that the same error happens from time to time during build in many other my PRs. Is it possible to disable this test temporaly (until the fix for the test is found)?
@JakeRadMSFT
JakeRadMSFT
commented
Jun 23, 2023

Copy link


Copy Markdown

Member
/azp run
@azure-pipelines
azure-pipelinesBot
commented
Jun 23, 2023

Copy link


Copy Markdown


Azure Pipelines successfully started running 2 pipeline(s).
JakeRadMSFT
JakeRadMSFT
reviewed
Jul 6, 2023
View reviewed changes
Comment threadtest/Microsoft.Data.Analysis.Tests/BufferTests.cs
 Assert.Null(clone[i]);
 }

 /* Don't run tests during build as they fail, because build if build machine doesn't have enought memory
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 are you sure this is the reason?
Can we add a flag so these still run locally?
@JakeRadMSFTJakeRadMSFTJul 6, 2023

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to do a #if and then set a flag in build definition
@JakeRadMSFTJakeRadMSFTJul 6, 2023•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asmirnov82 can you open an issue. I don't want to block on this.
@tannergooding said this might actually be a netfx thing? There might need to be something set to support > 2gb. Comment below.
"For the more than 2GB one, it will be allowed but will then fail on .NET Framework anyways due to GC limitations
You have to opt in with gcAllowVeryLargeObjects iirc, and that's only for 64-bit"
@asmirnov82asmirnov82Jul 7, 2023•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JakeRadMSFT I fixed tests using X64FactAttribute. Please take a look at #6758 and #6759
@JakeRadMSFT
JakeRadMSFT
merged commit 26c2446
into
dotnet:mainJul 6, 2023
@asmirnov82
asmirnov82
deleted the

6699_support_large_columns

branch
July 7, 2023 05:48
@asmirnov82asmirnov82
mentioned this pull request
Jul 7, 2023
 Merged

@ghostghost
locked as resolved and limited conversation to collaborators
Aug 6, 2023
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@JakeRadMSFTJakeRadMSFTJakeRadMSFT left review comments

+1 more reviewer

@michaelgsharpmichaelgsharpmichaelgsharp approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

community-contribution

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

DataFrame crashes on attempt to create column with size higher than 2,147,483,591 bytes


3 participants

@asmirnov82@michaelgsharp@JakeRadMSFT