ARROW-11243: [C++] Recognize time types in CSV files - #10782

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-11243-csv-times
Closed

ARROW-11243: [C++] Recognize time types in CSV files#10782
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-11243-csv-times

Conversation

@pitrou

@pitroupitrou commented Jul 22, 2021

Copy link
Copy Markdown
Member
  • Allow reading CSV columns as time32 and time64
  • Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]

@github-actions

github-actionsBot commented Jul 22, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-11243

@pitrou

pitrou commented Jul 22, 2021

Copy link
Copy Markdown
MemberAuthor

@ursabot please benchmark

@ursabot

ursabot commented Jul 22, 2021•
edited
Loading

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 169b249 and contender = 25b5c1e445f04b53a0beb40afabb3e8012c31f50. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2 (mimalloc)
[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x (mimalloc)
[Finished ⬇️0.91% ⬆️0.1%] ursa-thinkcentre-m75q (mimalloc)
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

bkietz
bkietz requested changes Jul 22, 2021
View reviewed changes

@bkietzbkietz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I'd just like the unit tests for inference to be expanded a bit

Comment threadcpp/src/arrow/csv/column_builder_test.cc Outdated
nealrichardson
nealrichardson reviewed Jul 26, 2021
View reviewed changes

@nealrichardsonnealrichardson left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me, just one question about time64

Comment threadcpp/src/arrow/csv/column_builder_test.cc Outdated

@nealrichardsonnealrichardsonJul 26, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be an inferring test that results in time64?

@pitroupitrouJul 26, 2021

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, there is no inference to time64. Should there be one (for times with nanosecond precision perhaps)?

@nealrichardsonnealrichardsonJul 26, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was looking at the wrong case statement, you're right. Maybe there should be one to catch any times with fractional seconds, though I don't feel strongly about it since you can explicitly declare the type/schema you want.

@pitrou
pitrouforce-pushed the ARROW-11243-csv-times branch from 25b5c1e to 301c1c1CompareJuly 26, 2021 13:10
@pitrou

pitrou commented Jul 26, 2021

Copy link
Copy Markdown
MemberAuthor

(rebased)

@westonpace

westonpace commented Jul 26, 2021

Copy link
Copy Markdown
Member

This could be a future JIRA or "wait until it's asked for" but technically ISO-8601 allows the omission of the colons if you add a leading T but I've never seen it in practice...

Thh
Thhmm
Thhmmss
Thhmmss.sss

pitrou added 2 commits July 27, 2021 09:57
@pitrou
ARROW-11243: [C++] Recognize time types in CSV files
e53da4a
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@pitrou
Add inference tests with mixed types
b103dd8
@pitrou
pitrouforce-pushed the ARROW-11243-csv-times branch from 301c1c1 to b103dd8CompareJuly 27, 2021 08:04
@pitrou

pitrou commented Jul 27, 2021•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

@westonpace I'd rather wait for someone to request it.

@pitrou
pitrou deleted the ARROW-11243-csv-times branch July 27, 2021 09:55
@asfimportasfimport mentioned this pull request Jul 27, 2021
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bkietzbkietzbkietz requested changes
@nealrichardsonnealrichardsonnealrichardson left review comments

Assignees

No one assigned

Labels

Component: C++Component: Python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@pitrou@ursabot@westonpace@bkietz@nealrichardson
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-11243: [C++] Recognize time types in CSV files - #10782
Closed
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Closed
ARROW-11243: [C++] Recognize time types in CSV files#10782
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Conversation
@pitrou
@pitroupitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

Member
  • Allow reading CSV columns as time32 and time64
  • Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@github-actions
github-actionsBot
commented
Jul 22, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-11243
@pitrou
pitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

MemberAuthor
@ursabot please benchmark
@ursabot
ursabot
commented
Jul 22, 2021•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 169b249 and contender = 25b5c1e445f04b53a0beb40afabb3e8012c31f50. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2 (mimalloc)

[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x (mimalloc)

[Finished ⬇️0.91% ⬆️0.1%] ursa-thinkcentre-m75q (mimalloc)

Supported benchmarks:

ursa-i9-9960x: langs = Python, R, JavaScript

ursa-thinkcentre-m75q: langs = C++, Java

ec2-t3-xlarge-us-east-2: cloud = True
bkietz
bkietz
requested changes
Jul 22, 2021
View reviewed changes
@bkietzbkietz
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I'd just like the unit tests for inference to be expanded a bit
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

nealrichardson
nealrichardson
reviewed
Jul 26, 2021
View reviewed changes
@nealrichardsonnealrichardson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me, just one question about time64
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be an inferring test that results in time64?
@pitroupitrouJul 26, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, there is no inference to time64. Should there be one (for times with nanosecond precision perhaps)?
@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was looking at the wrong case statement, you're right. Maybe there should be one to catch any times with fractional seconds, though I don't feel strongly about it since you can explicitly declare the type/schema you want.
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
25b5c1e to
301c1c1CompareJuly 26, 2021 13:10
@pitrou
pitrou
commented
Jul 26, 2021

Copy link


Copy Markdown

MemberAuthor
(rebased)
@westonpace
westonpace
commented
Jul 26, 2021

Copy link


Copy Markdown

Member
This could be a future JIRA or "wait until it's asked for" but technically ISO-8601 allows the omission of the colons if you add a leading T but I've never seen it in practice...
Thh
Thhmm
Thhmmss
Thhmmss.sss

pitrou
added 2 commits
July 27, 2021 09:57
@pitrou
ARROW-11243: [C++] Recognize time types in CSV files
e53da4a
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@pitrou
Add inference tests with mixed types
b103dd8
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
301c1c1 to
b103dd8CompareJuly 27, 2021 08:04
@pitrou
pitrou
commented
Jul 27, 2021•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
@westonpace I'd rather wait for someone to request it.
@pitrou
pitrou
deleted the

ARROW-11243-csv-times

branch
July 27, 2021 09:55
@asfimportasfimport
mentioned this pull request
Jul 27, 2021

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@bkietzbkietzbkietz requested changes
@nealrichardsonnealrichardsonnealrichardson left review comments

Assignees


No one assigned


Labels

Component: C++Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@pitrou@ursabot@westonpace@bkietz@nealrichardson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-11243: [C++] Recognize time types in CSV files - #10782
Closed
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Closed
ARROW-11243: [C++] Recognize time types in CSV files#10782
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Conversation
@pitrou
@pitroupitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

Member
  • Allow reading CSV columns as time32 and time64
  • Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@github-actions
github-actionsBot
commented
Jul 22, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-11243
@pitrou
pitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

MemberAuthor
@ursabot please benchmark
@ursabot
ursabot
commented
Jul 22, 2021•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 169b249 and contender = 25b5c1e445f04b53a0beb40afabb3e8012c31f50. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2 (mimalloc)

[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x (mimalloc)

[Finished ⬇️0.91% ⬆️0.1%] ursa-thinkcentre-m75q (mimalloc)

Supported benchmarks:

ursa-i9-9960x: langs = Python, R, JavaScript

ursa-thinkcentre-m75q: langs = C++, Java

ec2-t3-xlarge-us-east-2: cloud = True
bkietz
bkietz
requested changes
Jul 22, 2021
View reviewed changes
@bkietzbkietz
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I'd just like the unit tests for inference to be expanded a bit
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

nealrichardson
nealrichardson
reviewed
Jul 26, 2021
View reviewed changes
@nealrichardsonnealrichardson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me, just one question about time64
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be an inferring test that results in time64?
@pitroupitrouJul 26, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, there is no inference to time64. Should there be one (for times with nanosecond precision perhaps)?
@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was looking at the wrong case statement, you're right. Maybe there should be one to catch any times with fractional seconds, though I don't feel strongly about it since you can explicitly declare the type/schema you want.
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
25b5c1e to
301c1c1CompareJuly 26, 2021 13:10
@pitrou
pitrou
commented
Jul 26, 2021

Copy link


Copy Markdown

MemberAuthor
(rebased)
@westonpace
westonpace
commented
Jul 26, 2021

Copy link


Copy Markdown

Member
This could be a future JIRA or "wait until it's asked for" but technically ISO-8601 allows the omission of the colons if you add a leading T but I've never seen it in practice...
Thh
Thhmm
Thhmmss
Thhmmss.sss

pitrou
added 2 commits
July 27, 2021 09:57
@pitrou
ARROW-11243: [C++] Recognize time types in CSV files
e53da4a
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@pitrou
Add inference tests with mixed types
b103dd8
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
301c1c1 to
b103dd8CompareJuly 27, 2021 08:04
@pitrou
pitrou
commented
Jul 27, 2021•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
@westonpace I'd rather wait for someone to request it.
@pitrou
pitrou
deleted the

ARROW-11243-csv-times

branch
July 27, 2021 09:55
@asfimportasfimport
mentioned this pull request
Jul 27, 2021

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@bkietzbkietzbkietz requested changes
@nealrichardsonnealrichardsonnealrichardson left review comments

Assignees


No one assigned


Labels

Component: C++Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@pitrou@ursabot@westonpace@bkietz@nealrichardson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-11243: [C++] Recognize time types in CSV files - #10782
Closed
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Closed
ARROW-11243: [C++] Recognize time types in CSV files#10782
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Conversation
@pitrou
@pitroupitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

Member
  • Allow reading CSV columns as time32 and time64
  • Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@github-actions
github-actionsBot
commented
Jul 22, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-11243
@pitrou
pitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

MemberAuthor
@ursabot please benchmark
@ursabot
ursabot
commented
Jul 22, 2021•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 169b249 and contender = 25b5c1e445f04b53a0beb40afabb3e8012c31f50. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2 (mimalloc)

[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x (mimalloc)

[Finished ⬇️0.91% ⬆️0.1%] ursa-thinkcentre-m75q (mimalloc)

Supported benchmarks:

ursa-i9-9960x: langs = Python, R, JavaScript

ursa-thinkcentre-m75q: langs = C++, Java

ec2-t3-xlarge-us-east-2: cloud = True
bkietz
bkietz
requested changes
Jul 22, 2021
View reviewed changes
@bkietzbkietz
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I'd just like the unit tests for inference to be expanded a bit
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

nealrichardson
nealrichardson
reviewed
Jul 26, 2021
View reviewed changes
@nealrichardsonnealrichardson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me, just one question about time64
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be an inferring test that results in time64?
@pitroupitrouJul 26, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, there is no inference to time64. Should there be one (for times with nanosecond precision perhaps)?
@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was looking at the wrong case statement, you're right. Maybe there should be one to catch any times with fractional seconds, though I don't feel strongly about it since you can explicitly declare the type/schema you want.
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
25b5c1e to
301c1c1CompareJuly 26, 2021 13:10
@pitrou
pitrou
commented
Jul 26, 2021

Copy link


Copy Markdown

MemberAuthor
(rebased)
@westonpace
westonpace
commented
Jul 26, 2021

Copy link


Copy Markdown

Member
This could be a future JIRA or "wait until it's asked for" but technically ISO-8601 allows the omission of the colons if you add a leading T but I've never seen it in practice...
Thh
Thhmm
Thhmmss
Thhmmss.sss

pitrou
added 2 commits
July 27, 2021 09:57
@pitrou
ARROW-11243: [C++] Recognize time types in CSV files
e53da4a
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@pitrou
Add inference tests with mixed types
b103dd8
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
301c1c1 to
b103dd8CompareJuly 27, 2021 08:04
@pitrou
pitrou
commented
Jul 27, 2021•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
@westonpace I'd rather wait for someone to request it.
@pitrou
pitrou
deleted the

ARROW-11243-csv-times

branch
July 27, 2021 09:55
@asfimportasfimport
mentioned this pull request
Jul 27, 2021

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@bkietzbkietzbkietz requested changes
@nealrichardsonnealrichardsonnealrichardson left review comments

Assignees


No one assigned


Labels

Component: C++Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@pitrou@ursabot@westonpace@bkietz@nealrichardson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-11243: [C++] Recognize time types in CSV files - #10782
Closed
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Closed
ARROW-11243: [C++] Recognize time types in CSV files#10782
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Conversation
@pitrou
@pitroupitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

Member
  • Allow reading CSV columns as time32 and time64
  • Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@github-actions
github-actionsBot
commented
Jul 22, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-11243
@pitrou
pitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

MemberAuthor
@ursabot please benchmark
@ursabot
ursabot
commented
Jul 22, 2021•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 169b249 and contender = 25b5c1e445f04b53a0beb40afabb3e8012c31f50. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2 (mimalloc)

[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x (mimalloc)

[Finished ⬇️0.91% ⬆️0.1%] ursa-thinkcentre-m75q (mimalloc)

Supported benchmarks:

ursa-i9-9960x: langs = Python, R, JavaScript

ursa-thinkcentre-m75q: langs = C++, Java

ec2-t3-xlarge-us-east-2: cloud = True
bkietz
bkietz
requested changes
Jul 22, 2021
View reviewed changes
@bkietzbkietz
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I'd just like the unit tests for inference to be expanded a bit
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

nealrichardson
nealrichardson
reviewed
Jul 26, 2021
View reviewed changes
@nealrichardsonnealrichardson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me, just one question about time64
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be an inferring test that results in time64?
@pitroupitrouJul 26, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, there is no inference to time64. Should there be one (for times with nanosecond precision perhaps)?
@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was looking at the wrong case statement, you're right. Maybe there should be one to catch any times with fractional seconds, though I don't feel strongly about it since you can explicitly declare the type/schema you want.
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
25b5c1e to
301c1c1CompareJuly 26, 2021 13:10
@pitrou
pitrou
commented
Jul 26, 2021

Copy link


Copy Markdown

MemberAuthor
(rebased)
@westonpace
westonpace
commented
Jul 26, 2021

Copy link


Copy Markdown

Member
This could be a future JIRA or "wait until it's asked for" but technically ISO-8601 allows the omission of the colons if you add a leading T but I've never seen it in practice...
Thh
Thhmm
Thhmmss
Thhmmss.sss

pitrou
added 2 commits
July 27, 2021 09:57
@pitrou
ARROW-11243: [C++] Recognize time types in CSV files
e53da4a
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@pitrou
Add inference tests with mixed types
b103dd8
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
301c1c1 to
b103dd8CompareJuly 27, 2021 08:04
@pitrou
pitrou
commented
Jul 27, 2021•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
@westonpace I'd rather wait for someone to request it.
@pitrou
pitrou
deleted the

ARROW-11243-csv-times

branch
July 27, 2021 09:55
@asfimportasfimport
mentioned this pull request
Jul 27, 2021

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@bkietzbkietzbkietz requested changes
@nealrichardsonnealrichardsonnealrichardson left review comments

Assignees


No one assigned


Labels

Component: C++Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@pitrou@ursabot@westonpace@bkietz@nealrichardson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-11243: [C++] Recognize time types in CSV files - #10782
Closed
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Closed
ARROW-11243: [C++] Recognize time types in CSV files#10782
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Conversation
@pitrou
@pitroupitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

Member
  • Allow reading CSV columns as time32 and time64
  • Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@github-actions
github-actionsBot
commented
Jul 22, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-11243
@pitrou
pitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

MemberAuthor
@ursabot please benchmark
@ursabot
ursabot
commented
Jul 22, 2021•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 169b249 and contender = 25b5c1e445f04b53a0beb40afabb3e8012c31f50. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2 (mimalloc)

[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x (mimalloc)

[Finished ⬇️0.91% ⬆️0.1%] ursa-thinkcentre-m75q (mimalloc)

Supported benchmarks:

ursa-i9-9960x: langs = Python, R, JavaScript

ursa-thinkcentre-m75q: langs = C++, Java

ec2-t3-xlarge-us-east-2: cloud = True
bkietz
bkietz
requested changes
Jul 22, 2021
View reviewed changes
@bkietzbkietz
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I'd just like the unit tests for inference to be expanded a bit
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

nealrichardson
nealrichardson
reviewed
Jul 26, 2021
View reviewed changes
@nealrichardsonnealrichardson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me, just one question about time64
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be an inferring test that results in time64?
@pitroupitrouJul 26, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, there is no inference to time64. Should there be one (for times with nanosecond precision perhaps)?
@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was looking at the wrong case statement, you're right. Maybe there should be one to catch any times with fractional seconds, though I don't feel strongly about it since you can explicitly declare the type/schema you want.
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
25b5c1e to
301c1c1CompareJuly 26, 2021 13:10
@pitrou
pitrou
commented
Jul 26, 2021

Copy link


Copy Markdown

MemberAuthor
(rebased)
@westonpace
westonpace
commented
Jul 26, 2021

Copy link


Copy Markdown

Member
This could be a future JIRA or "wait until it's asked for" but technically ISO-8601 allows the omission of the colons if you add a leading T but I've never seen it in practice...
Thh
Thhmm
Thhmmss
Thhmmss.sss

pitrou
added 2 commits
July 27, 2021 09:57
@pitrou
ARROW-11243: [C++] Recognize time types in CSV files
e53da4a
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@pitrou
Add inference tests with mixed types
b103dd8
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
301c1c1 to
b103dd8CompareJuly 27, 2021 08:04
@pitrou
pitrou
commented
Jul 27, 2021•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
@westonpace I'd rather wait for someone to request it.
@pitrou
pitrou
deleted the

ARROW-11243-csv-times

branch
July 27, 2021 09:55
@asfimportasfimport
mentioned this pull request
Jul 27, 2021

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@bkietzbkietzbkietz requested changes
@nealrichardsonnealrichardsonnealrichardson left review comments

Assignees


No one assigned


Labels

Component: C++Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@pitrou@ursabot@westonpace@bkietz@nealrichardson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-11243: [C++] Recognize time types in CSV files - #10782
Closed
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Closed
ARROW-11243: [C++] Recognize time types in CSV files#10782
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Conversation
@pitrou
@pitroupitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

Member
  • Allow reading CSV columns as time32 and time64
  • Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@github-actions
github-actionsBot
commented
Jul 22, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-11243
@pitrou
pitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

MemberAuthor
@ursabot please benchmark
@ursabot
ursabot
commented
Jul 22, 2021•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 169b249 and contender = 25b5c1e445f04b53a0beb40afabb3e8012c31f50. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2 (mimalloc)

[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x (mimalloc)

[Finished ⬇️0.91% ⬆️0.1%] ursa-thinkcentre-m75q (mimalloc)

Supported benchmarks:

ursa-i9-9960x: langs = Python, R, JavaScript

ursa-thinkcentre-m75q: langs = C++, Java

ec2-t3-xlarge-us-east-2: cloud = True
bkietz
bkietz
requested changes
Jul 22, 2021
View reviewed changes
@bkietzbkietz
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I'd just like the unit tests for inference to be expanded a bit
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

nealrichardson
nealrichardson
reviewed
Jul 26, 2021
View reviewed changes
@nealrichardsonnealrichardson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me, just one question about time64
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be an inferring test that results in time64?
@pitroupitrouJul 26, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, there is no inference to time64. Should there be one (for times with nanosecond precision perhaps)?
@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was looking at the wrong case statement, you're right. Maybe there should be one to catch any times with fractional seconds, though I don't feel strongly about it since you can explicitly declare the type/schema you want.
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
25b5c1e to
301c1c1CompareJuly 26, 2021 13:10
@pitrou
pitrou
commented
Jul 26, 2021

Copy link


Copy Markdown

MemberAuthor
(rebased)
@westonpace
westonpace
commented
Jul 26, 2021

Copy link


Copy Markdown

Member
This could be a future JIRA or "wait until it's asked for" but technically ISO-8601 allows the omission of the colons if you add a leading T but I've never seen it in practice...
Thh
Thhmm
Thhmmss
Thhmmss.sss

pitrou
added 2 commits
July 27, 2021 09:57
@pitrou
ARROW-11243: [C++] Recognize time types in CSV files
e53da4a
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@pitrou
Add inference tests with mixed types
b103dd8
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
301c1c1 to
b103dd8CompareJuly 27, 2021 08:04
@pitrou
pitrou
commented
Jul 27, 2021•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
@westonpace I'd rather wait for someone to request it.
@pitrou
pitrou
deleted the

ARROW-11243-csv-times

branch
July 27, 2021 09:55
@asfimportasfimport
mentioned this pull request
Jul 27, 2021

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@bkietzbkietzbkietz requested changes
@nealrichardsonnealrichardsonnealrichardson left review comments

Assignees


No one assigned


Labels

Component: C++Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@pitrou@ursabot@westonpace@bkietz@nealrichardson

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-11243: [C++] Recognize time types in CSV files - #10782
Closed
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Closed
ARROW-11243: [C++] Recognize time types in CSV files#10782
pitrou wants to merge 2 commits into
apache:masterfrom 
pitrou:ARROW-11243-csv-times
Conversation
@pitrou
@pitroupitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

Member
  • Allow reading CSV columns as time32 and time64
  • Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@github-actions
github-actionsBot
commented
Jul 22, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-11243
@pitrou
pitrou
commented
Jul 22, 2021

Copy link


Copy Markdown

MemberAuthor
@ursabot please benchmark
@ursabot
ursabot
commented
Jul 22, 2021•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 169b249 and contender = 25b5c1e445f04b53a0beb40afabb3e8012c31f50. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2 (mimalloc)

[Finished ⬇️0.0% ⬆️1.53%] ursa-i9-9960x (mimalloc)

[Finished ⬇️0.91% ⬆️0.1%] ursa-thinkcentre-m75q (mimalloc)

Supported benchmarks:

ursa-i9-9960x: langs = Python, R, JavaScript

ursa-thinkcentre-m75q: langs = C++, Java

ec2-t3-xlarge-us-east-2: cloud = True
bkietz
bkietz
requested changes
Jul 22, 2021
View reviewed changes
@bkietzbkietz
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, I'd just like the unit tests for inference to be expanded a bit
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

nealrichardson
nealrichardson
reviewed
Jul 26, 2021
View reviewed changes
@nealrichardsonnealrichardson
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me, just one question about time64
Comment threadcpp/src/arrow/csv/column_builder_test.cc
Outdated

@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there also be an inferring test that results in time64?
@pitroupitrouJul 26, 2021

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, there is no inference to time64. Should there be one (for times with nanosecond precision perhaps)?
@nealrichardsonnealrichardsonJul 26, 2021

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see, I was looking at the wrong case statement, you're right. Maybe there should be one to catch any times with fractional seconds, though I don't feel strongly about it since you can explicitly declare the type/schema you want.
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
25b5c1e to
301c1c1CompareJuly 26, 2021 13:10
@pitrou
pitrou
commented
Jul 26, 2021

Copy link


Copy Markdown

MemberAuthor
(rebased)
@westonpace
westonpace
commented
Jul 26, 2021

Copy link


Copy Markdown

Member
This could be a future JIRA or "wait until it's asked for" but technically ISO-8601 allows the omission of the colons if you add a leading T but I've never seen it in practice...
Thh
Thhmm
Thhmmss
Thhmmss.sss

pitrou
added 2 commits
July 27, 2021 09:57
@pitrou
ARROW-11243: [C++] Recognize time types in CSV files
e53da4a
* Allow reading CSV columns as time32 and time64
* Automatically infer "hh:mm" and "hh:mm:ss" as time32[s]
@pitrou
Add inference tests with mixed types
b103dd8
@pitrou
pitrouforce-pushed
the
ARROW-11243-csv-times
branch
from
301c1c1 to
b103dd8CompareJuly 27, 2021 08:04
@pitrou
pitrou
commented
Jul 27, 2021•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
@westonpace I'd rather wait for someone to request it.
@pitrou
pitrou
deleted the

ARROW-11243-csv-times

branch
July 27, 2021 09:55
@asfimportasfimport
mentioned this pull request
Jul 27, 2021

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@bkietzbkietzbkietz requested changes
@nealrichardsonnealrichardsonnealrichardson left review comments

Assignees


No one assigned


Labels

Component: C++Component: Python

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@pitrou@ursabot@westonpace@bkietz@nealrichardson