ARROW-10924: [C++] Validate temporal data in ValidateArrayFull - #12014

Closed
JabariBooker wants to merge 21 commits into
apache:masterfrom
JabariBooker:ARROW-10924
Closed

ARROW-10924: [C++] Validate temporal data in ValidateArrayFull#12014
JabariBooker wants to merge 21 commits into
apache:masterfrom
JabariBooker:ARROW-10924

Conversation

@JabariBooker

@JabariBookerJabariBooker commented Dec 21, 2021

Copy link
Copy Markdown
Contributor

Validate that temporal data in arrays meet the restrictions of their data type specifications.

@github-actions

github-actionsBot commented Dec 21, 2021

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10924

@github-actions

github-actionsBot commented Dec 21, 2021

Copy link
Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

lidavidm
lidavidm reviewed Dec 22, 2021
View reviewed changes

@lidavidmlidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

There are quite a few failing tests. We'll have to go through each of them individually to decide what to do about them.

You will want to lint and format the code.

Comment threadcpp/src/arrow/array/validate.cc Outdated
Comment threadcpp/src/arrow/array/validate.cc Outdated
Comment threadcpp/src/arrow/array/validate.cc Outdated
lidavidm
lidavidm reviewed Dec 23, 2021
View reviewed changes

@lidavidmlidavidm left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.

You'll want to lint and format again (see the CI failure: https://github.com/apache/arrow/runs/4612596806?check_suite_focus=true)

Looking at test failures:

edponce
edponce suggested changes Dec 29, 2021
View reviewed changes
Comment threadcpp/src/arrow/array/validate.cc Outdated
Comment threadcpp/src/arrow/array/validate.cc Outdated
Comment threadcpp/src/arrow/array/validate.cc
Comment threadcpp/src/arrow/array/validate.cc Outdated
Comment threadcpp/src/arrow/array/validate.cc Outdated
@edponce

edponce commented Dec 29, 2021

Copy link
Copy Markdown
Contributor

@JabariBooker Thanks for working on this! I left some minor comments.
Also, we are missing validation for the following temporal types: Date32Type and TimestampType.

@pitrou

pitrou commented Dec 30, 2021

Copy link
Copy Markdown
Member

@edponce There are no limits on Date32Type and TimestampType AFAICT.

@edponce

edponce commented Jan 6, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

@pitrou So there is a limit for Date64Type but not for Date32Type? I now understand that TimestampType does not have limits.

@lidavidm

lidavidm commented Jan 6, 2022

Copy link
Copy Markdown
Member

Date32Type's unit is days and so there's nothing to check, but Date64Type's unit is milliseconds, hence it doesn't have a limit per se, but it must be a multiple of 86400000 (=1 day in milliseconds).

@lidavidm

lidavidm commented Jan 11, 2022

Copy link
Copy Markdown
Member

FYI, that test failure in GroupBy.MinMaxTypes is here:

arrow/cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

Line 1614 in d88e232

TEST(GroupBy, MinMaxTypes) {

The test uses one set of data to go through several data types, so you could just update the hardcoded data there too.

lidavidm
lidavidm approved these changes Jan 28, 2022
View reviewed changes

@lidavidmlidavidm left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I left some nits (we can check DataType::id() instead of the string name). And thanks for going through all the function tests; we should perhaps consider removing Date64 from the list of types to check in order to just split off those tests separately given the restrictions on value. (EDIT: not in this PR of course.)

Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
SCOPED_TRACE(ty->ToString());
auto in_schema = schema({field("argument0", ty), field("key", int64())});
auto table =
TableFromJSON(in_schema, (ty->name() == "date64") ? date64_table : default_table);

@lidavidmlidavidmJan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can check ty->id() == Type::DATE64

Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
[{"min": 3, "max": 5}, 4],
[{"min": 1, "max": 4}, null]
])"),
(ty->name() == "date64") ? date64_expected : default_expected),

@lidavidmlidavidmJan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ditto here

Comment threadcpp/src/arrow/compute/kernels/vector_hash_test.cc Outdated
// Sliced
CheckUnique(ArrayFromJSON(type, "[1, 2, null, 3, 2, null]")->Slice(1, 4),
ArrayFromJSON(type, "[2, null, 3]"));
if (type->name() == "date64") {

@lidavidmlidavidmJan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here and so on (sorry)

Comment threadcpp/src/arrow/testing/random.cc Outdated
}

case Type::type::TIME32: {
TimeUnit::type unit = std::dynamic_pointer_cast<Time32Type>(field.type())->unit();

@lidavidmlidavidmJan 28, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have internal::checked_[pointer_]cast which will use dynamic or static cast depending on debug/release build

@lidavidmlidavidm mentioned this pull request Jan 28, 2022
Closed
@JabariBooker

JabariBooker commented Jan 28, 2022

Copy link
Copy Markdown
ContributorAuthor

@lidavidm@edponce@pitrou There are still failing integration tests (all are also failing due to invalid values for date64). The data involved in these tests seem to be generated by some mechanism I'm unaware of.

edponce
edponce reviewed Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/array/array_test.cc
edponce
edponce suggested changes Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/testing/random.cc
options.GenerateData(buffers[1]->mutable_data(), size);
if (std::is_same<ArrowType, Date64Type>::value) {
GenerateFullDayMillisNoNan(buffers[1]->mutable_data(), size);
}

@edponceedponceJan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an if-else. buffers[1]->mutable_data() is modified twice if Date64Type.

@edponceedponceJan 28, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more suitable place for this type-specific generation of numeric arrays, is in GenerateTypeDataNoNan() from GenerateOptions. Make GenerateFullDayMillisNoNan as a private member to GenerateOptions and perform the if-else check in GenerateTypedDataNoNan().

@JabariBookerJabariBookerJan 28, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Since we need to generate multiples of 86400000 for Date64Type, random numbers are generated in GenerateData and ultimately multiplied by 86400000 in GenerateFullDayMillisNoNan. The behavior of GenerateFullDayMillisNoNan is separated from GenerateOptions since this class doesn't know the arrow type, only the underlying C type. However, behavior of GenerateFullDayMillisNoNan could be moved into the above if statement if needed.

Comment threadcpp/src/arrow/testing/random.cc
@pitrou

pitrou commented Jan 31, 2022

Copy link
Copy Markdown
Member

@JabariBooker Integration data is generated in Python by Archery here: https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py#L196

You can find more info about the JSON integration format here, and how to run the integration tests locally, here: https://arrow.apache.org/docs/format/Integration.html

lidavidm
lidavidm reviewed Feb 1, 2022
View reviewed changes

@lidavidmlidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a minor nit on the datagen (which we probably won't run into, but…)

Comment threaddev/archery/archery/integration/datagen.py Outdated
return super().generate_range(size, lower, upper, name)

full_day_millis = 1000 * 60 * 60 * 24
lower //= full_day_millis

@lidavidmlidavidmFeb 1, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python appears to round towards -Inf so a lower negative bound that is not exactly even will get rounded to an out-of-bound value

lidavidm
lidavidm reviewed Feb 1, 2022
View reviewed changes

@lidavidmlidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we should be good here. This turned out to be quite a bit more involved than it looked at first.

@pitrou@edponce any other comments?

Comment threaddev/archery/archery/integration/tester_cpp.py
@ursabot

ursabot commented Feb 2, 2022•
edited
Loading

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 9b53235 and contender = c715beb. c715beb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.13% ⬆️0.0%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@asfimportasfimport mentioned this pull request Feb 3, 2022
Closed
@JefffreyJefffrey mentioned this pull request Jan 7, 2024
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes
@pitroupitroupitrou left review comments
+1 more reviewer
@edponceedponceedponce approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@JabariBooker@edponce@pitrou@lidavidm@ursabot
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull - #12014
Closed
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Closed
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull#12014
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Conversation
@JabariBooker
@JabariBookerJabariBooker
commented
Dec 21, 2021

Copy link


Copy Markdown

Contributor
Validate that temporal data in arrays meet the restrictions of their data type specifications.
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10924
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
lidavidm
lidavidm
reviewed
Dec 22, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
There are quite a few failing tests. We'll have to go through each of them individually to decide what to do about them.
You will want to lint and format the code.
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

lidavidm
lidavidm
reviewed
Dec 23, 2021
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.
You'll want to lint and format again (see the CI failure: https://github.com/apache/arrow/runs/4612596806?check_suite_focus=true)
Looking at test failures:
edponce
edponce
suggested changes
Dec 29, 2021
View reviewed changes
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

@edponce
edponce
commented
Dec 29, 2021

Copy link


Copy Markdown

Contributor
@JabariBooker Thanks for working on this! I left some minor comments.

Also, we are missing validation for the following temporal types: Date32Type and TimestampType.
@pitrou
pitrou
commented
Dec 30, 2021

Copy link


Copy Markdown

Member
@edponce There are no limits on Date32Type and TimestampType AFAICT.
@edponce
edponce
commented
Jan 6, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
@pitrou So there is a limit for Date64Type but not for Date32Type? I now understand that TimestampType does not have limits.
@lidavidm
lidavidm
commented
Jan 6, 2022

Copy link


Copy Markdown

Member
Date32Type's unit is days and so there's nothing to check, but Date64Type's unit is milliseconds, hence it doesn't have a limit per se, but it must be a multiple of 86400000 (=1 day in milliseconds).
@lidavidm
lidavidm
commented
Jan 11, 2022

Copy link


Copy Markdown

Member
FYI, that test failure in GroupBy.MinMaxTypes is here: 
arrow/cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

Line 1614
in
d88e232
TEST(GroupBy, MinMaxTypes) { 
The test uses one set of data to go through several data types, so you could just update the hardcoded data there too.
lidavidm
lidavidm
approved these changes
Jan 28, 2022
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I left some nits (we can check DataType::id() instead of the string name). And thanks for going through all the function tests; we should perhaps consider removing Date64 from the list of types to check in order to just split off those tests separately given the restrictions on value. (EDIT: not in this PR of course.)
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 SCOPED_TRACE(ty->ToString());
 auto in_schema = schema({field("argument0", ty), field("key", int64())});
 auto table =
 TableFromJSON(in_schema, (ty->name() == "date64") ? date64_table : default_table);
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can check ty->id() == Type::DATE64
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 [{"min": 3, "max": 5}, 4],
 [{"min": 1, "max": 4}, null]
 ])"),
 (ty->name() == "date64") ? date64_expected : default_expected),
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ditto here
Comment threadcpp/src/arrow/compute/kernels/vector_hash_test.cc
Outdated

 // Sliced
 CheckUnique(ArrayFromJSON(type, "[1, 2, null, 3, 2, null]")->Slice(1, 4),
 ArrayFromJSON(type, "[2, null, 3]"));
 if (type->name() == "date64") {
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here and so on (sorry)
Comment threadcpp/src/arrow/testing/random.cc
Outdated

 }

 case Type::type::TIME32: {
 TimeUnit::type unit = std::dynamic_pointer_cast<Time32Type>(field.type())->unit();
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have internal::checked_[pointer_]cast which will use dynamic or static cast depending on debug/release build
@lidavidmlidavidm
mentioned this pull request
Jan 28, 2022
 Closed

@JabariBooker
JabariBooker
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@lidavidm@edponce@pitrou There are still failing integration tests (all are also failing due to invalid values for date64). The data involved in these tests seem to be generated by some mechanism I'm unaware of.
edponce
edponce
reviewed
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/array/array_test.cc
edponce
edponce
suggested changes
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/testing/random.cc
 options.GenerateData(buffers[1]->mutable_data(), size);
 if (std::is_same<ArrowType, Date64Type>::value) {
 GenerateFullDayMillisNoNan(buffers[1]->mutable_data(), size);
 }
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an if-else. buffers[1]->mutable_data() is modified twice if Date64Type.
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more suitable place for this type-specific generation of numeric arrays, is in GenerateTypeDataNoNan() from GenerateOptions. Make GenerateFullDayMillisNoNan as a private member to GenerateOptions and perform the if-else check in GenerateTypedDataNoNan().
@JabariBookerJabariBookerJan 28, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Since we need to generate multiples of 86400000 for Date64Type, random numbers are generated in GenerateData and ultimately multiplied by 86400000 in GenerateFullDayMillisNoNan. The behavior of GenerateFullDayMillisNoNan is separated from GenerateOptions since this class doesn't know the arrow type, only the underlying C type. However, behavior of GenerateFullDayMillisNoNan could be moved into the above if statement if needed.
Comment threadcpp/src/arrow/testing/random.cc
@pitrou
pitrou
commented
Jan 31, 2022

Copy link


Copy Markdown

Member
@JabariBooker Integration data is generated in Python by Archery here: https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py#L196
You can find more info about the JSON integration format here, and how to run the integration tests locally, here: https://arrow.apache.org/docs/format/Integration.html
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a minor nit on the datagen (which we probably won't run into, but…)
Comment threaddev/archery/archery/integration/datagen.py
Outdated

 return super().generate_range(size, lower, upper, name)

 full_day_millis = 1000 * 60 * 60 * 24
 lower //= full_day_millis
@lidavidmlidavidmFeb 1, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python appears to round towards -Inf so a lower negative bound that is not exactly even will get rounded to an out-of-bound value
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we should be good here. This turned out to be quite a bit more involved than it looked at first.
@pitrou@edponce any other comments?
Comment threaddev/archery/archery/integration/tester_cpp.py
@ursabot
ursabot
commented
Feb 2, 2022•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 9b53235 and contender = c715beb. c715beb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.13% ⬆️0.0%] ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Feb 3, 2022

Closed

@JefffreyJefffrey
mentioned this pull request
Jan 7, 2024

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes
@pitroupitroupitrou left review comments

+1 more reviewer

@edponceedponceedponce approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@JabariBooker@edponce@pitrou@lidavidm@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull - #12014
Closed
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Closed
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull#12014
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Conversation
@JabariBooker
@JabariBookerJabariBooker
commented
Dec 21, 2021

Copy link


Copy Markdown

Contributor
Validate that temporal data in arrays meet the restrictions of their data type specifications.
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10924
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
lidavidm
lidavidm
reviewed
Dec 22, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
There are quite a few failing tests. We'll have to go through each of them individually to decide what to do about them.
You will want to lint and format the code.
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

lidavidm
lidavidm
reviewed
Dec 23, 2021
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.
You'll want to lint and format again (see the CI failure: https://github.com/apache/arrow/runs/4612596806?check_suite_focus=true)
Looking at test failures:
edponce
edponce
suggested changes
Dec 29, 2021
View reviewed changes
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

@edponce
edponce
commented
Dec 29, 2021

Copy link


Copy Markdown

Contributor
@JabariBooker Thanks for working on this! I left some minor comments.

Also, we are missing validation for the following temporal types: Date32Type and TimestampType.
@pitrou
pitrou
commented
Dec 30, 2021

Copy link


Copy Markdown

Member
@edponce There are no limits on Date32Type and TimestampType AFAICT.
@edponce
edponce
commented
Jan 6, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
@pitrou So there is a limit for Date64Type but not for Date32Type? I now understand that TimestampType does not have limits.
@lidavidm
lidavidm
commented
Jan 6, 2022

Copy link


Copy Markdown

Member
Date32Type's unit is days and so there's nothing to check, but Date64Type's unit is milliseconds, hence it doesn't have a limit per se, but it must be a multiple of 86400000 (=1 day in milliseconds).
@lidavidm
lidavidm
commented
Jan 11, 2022

Copy link


Copy Markdown

Member
FYI, that test failure in GroupBy.MinMaxTypes is here: 
arrow/cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

Line 1614
in
d88e232
TEST(GroupBy, MinMaxTypes) { 
The test uses one set of data to go through several data types, so you could just update the hardcoded data there too.
lidavidm
lidavidm
approved these changes
Jan 28, 2022
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I left some nits (we can check DataType::id() instead of the string name). And thanks for going through all the function tests; we should perhaps consider removing Date64 from the list of types to check in order to just split off those tests separately given the restrictions on value. (EDIT: not in this PR of course.)
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 SCOPED_TRACE(ty->ToString());
 auto in_schema = schema({field("argument0", ty), field("key", int64())});
 auto table =
 TableFromJSON(in_schema, (ty->name() == "date64") ? date64_table : default_table);
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can check ty->id() == Type::DATE64
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 [{"min": 3, "max": 5}, 4],
 [{"min": 1, "max": 4}, null]
 ])"),
 (ty->name() == "date64") ? date64_expected : default_expected),
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ditto here
Comment threadcpp/src/arrow/compute/kernels/vector_hash_test.cc
Outdated

 // Sliced
 CheckUnique(ArrayFromJSON(type, "[1, 2, null, 3, 2, null]")->Slice(1, 4),
 ArrayFromJSON(type, "[2, null, 3]"));
 if (type->name() == "date64") {
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here and so on (sorry)
Comment threadcpp/src/arrow/testing/random.cc
Outdated

 }

 case Type::type::TIME32: {
 TimeUnit::type unit = std::dynamic_pointer_cast<Time32Type>(field.type())->unit();
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have internal::checked_[pointer_]cast which will use dynamic or static cast depending on debug/release build
@lidavidmlidavidm
mentioned this pull request
Jan 28, 2022
 Closed

@JabariBooker
JabariBooker
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@lidavidm@edponce@pitrou There are still failing integration tests (all are also failing due to invalid values for date64). The data involved in these tests seem to be generated by some mechanism I'm unaware of.
edponce
edponce
reviewed
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/array/array_test.cc
edponce
edponce
suggested changes
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/testing/random.cc
 options.GenerateData(buffers[1]->mutable_data(), size);
 if (std::is_same<ArrowType, Date64Type>::value) {
 GenerateFullDayMillisNoNan(buffers[1]->mutable_data(), size);
 }
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an if-else. buffers[1]->mutable_data() is modified twice if Date64Type.
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more suitable place for this type-specific generation of numeric arrays, is in GenerateTypeDataNoNan() from GenerateOptions. Make GenerateFullDayMillisNoNan as a private member to GenerateOptions and perform the if-else check in GenerateTypedDataNoNan().
@JabariBookerJabariBookerJan 28, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Since we need to generate multiples of 86400000 for Date64Type, random numbers are generated in GenerateData and ultimately multiplied by 86400000 in GenerateFullDayMillisNoNan. The behavior of GenerateFullDayMillisNoNan is separated from GenerateOptions since this class doesn't know the arrow type, only the underlying C type. However, behavior of GenerateFullDayMillisNoNan could be moved into the above if statement if needed.
Comment threadcpp/src/arrow/testing/random.cc
@pitrou
pitrou
commented
Jan 31, 2022

Copy link


Copy Markdown

Member
@JabariBooker Integration data is generated in Python by Archery here: https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py#L196
You can find more info about the JSON integration format here, and how to run the integration tests locally, here: https://arrow.apache.org/docs/format/Integration.html
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a minor nit on the datagen (which we probably won't run into, but…)
Comment threaddev/archery/archery/integration/datagen.py
Outdated

 return super().generate_range(size, lower, upper, name)

 full_day_millis = 1000 * 60 * 60 * 24
 lower //= full_day_millis
@lidavidmlidavidmFeb 1, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python appears to round towards -Inf so a lower negative bound that is not exactly even will get rounded to an out-of-bound value
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we should be good here. This turned out to be quite a bit more involved than it looked at first.
@pitrou@edponce any other comments?
Comment threaddev/archery/archery/integration/tester_cpp.py
@ursabot
ursabot
commented
Feb 2, 2022•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 9b53235 and contender = c715beb. c715beb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.13% ⬆️0.0%] ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Feb 3, 2022

Closed

@JefffreyJefffrey
mentioned this pull request
Jan 7, 2024

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes
@pitroupitroupitrou left review comments

+1 more reviewer

@edponceedponceedponce approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@JabariBooker@edponce@pitrou@lidavidm@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull - #12014
Closed
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Closed
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull#12014
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Conversation
@JabariBooker
@JabariBookerJabariBooker
commented
Dec 21, 2021

Copy link


Copy Markdown

Contributor
Validate that temporal data in arrays meet the restrictions of their data type specifications.
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10924
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
lidavidm
lidavidm
reviewed
Dec 22, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
There are quite a few failing tests. We'll have to go through each of them individually to decide what to do about them.
You will want to lint and format the code.
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

lidavidm
lidavidm
reviewed
Dec 23, 2021
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.
You'll want to lint and format again (see the CI failure: https://github.com/apache/arrow/runs/4612596806?check_suite_focus=true)
Looking at test failures:
edponce
edponce
suggested changes
Dec 29, 2021
View reviewed changes
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

@edponce
edponce
commented
Dec 29, 2021

Copy link


Copy Markdown

Contributor
@JabariBooker Thanks for working on this! I left some minor comments.

Also, we are missing validation for the following temporal types: Date32Type and TimestampType.
@pitrou
pitrou
commented
Dec 30, 2021

Copy link


Copy Markdown

Member
@edponce There are no limits on Date32Type and TimestampType AFAICT.
@edponce
edponce
commented
Jan 6, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
@pitrou So there is a limit for Date64Type but not for Date32Type? I now understand that TimestampType does not have limits.
@lidavidm
lidavidm
commented
Jan 6, 2022

Copy link


Copy Markdown

Member
Date32Type's unit is days and so there's nothing to check, but Date64Type's unit is milliseconds, hence it doesn't have a limit per se, but it must be a multiple of 86400000 (=1 day in milliseconds).
@lidavidm
lidavidm
commented
Jan 11, 2022

Copy link


Copy Markdown

Member
FYI, that test failure in GroupBy.MinMaxTypes is here: 
arrow/cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

Line 1614
in
d88e232
TEST(GroupBy, MinMaxTypes) { 
The test uses one set of data to go through several data types, so you could just update the hardcoded data there too.
lidavidm
lidavidm
approved these changes
Jan 28, 2022
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I left some nits (we can check DataType::id() instead of the string name). And thanks for going through all the function tests; we should perhaps consider removing Date64 from the list of types to check in order to just split off those tests separately given the restrictions on value. (EDIT: not in this PR of course.)
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 SCOPED_TRACE(ty->ToString());
 auto in_schema = schema({field("argument0", ty), field("key", int64())});
 auto table =
 TableFromJSON(in_schema, (ty->name() == "date64") ? date64_table : default_table);
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can check ty->id() == Type::DATE64
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 [{"min": 3, "max": 5}, 4],
 [{"min": 1, "max": 4}, null]
 ])"),
 (ty->name() == "date64") ? date64_expected : default_expected),
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ditto here
Comment threadcpp/src/arrow/compute/kernels/vector_hash_test.cc
Outdated

 // Sliced
 CheckUnique(ArrayFromJSON(type, "[1, 2, null, 3, 2, null]")->Slice(1, 4),
 ArrayFromJSON(type, "[2, null, 3]"));
 if (type->name() == "date64") {
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here and so on (sorry)
Comment threadcpp/src/arrow/testing/random.cc
Outdated

 }

 case Type::type::TIME32: {
 TimeUnit::type unit = std::dynamic_pointer_cast<Time32Type>(field.type())->unit();
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have internal::checked_[pointer_]cast which will use dynamic or static cast depending on debug/release build
@lidavidmlidavidm
mentioned this pull request
Jan 28, 2022
 Closed

@JabariBooker
JabariBooker
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@lidavidm@edponce@pitrou There are still failing integration tests (all are also failing due to invalid values for date64). The data involved in these tests seem to be generated by some mechanism I'm unaware of.
edponce
edponce
reviewed
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/array/array_test.cc
edponce
edponce
suggested changes
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/testing/random.cc
 options.GenerateData(buffers[1]->mutable_data(), size);
 if (std::is_same<ArrowType, Date64Type>::value) {
 GenerateFullDayMillisNoNan(buffers[1]->mutable_data(), size);
 }
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an if-else. buffers[1]->mutable_data() is modified twice if Date64Type.
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more suitable place for this type-specific generation of numeric arrays, is in GenerateTypeDataNoNan() from GenerateOptions. Make GenerateFullDayMillisNoNan as a private member to GenerateOptions and perform the if-else check in GenerateTypedDataNoNan().
@JabariBookerJabariBookerJan 28, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Since we need to generate multiples of 86400000 for Date64Type, random numbers are generated in GenerateData and ultimately multiplied by 86400000 in GenerateFullDayMillisNoNan. The behavior of GenerateFullDayMillisNoNan is separated from GenerateOptions since this class doesn't know the arrow type, only the underlying C type. However, behavior of GenerateFullDayMillisNoNan could be moved into the above if statement if needed.
Comment threadcpp/src/arrow/testing/random.cc
@pitrou
pitrou
commented
Jan 31, 2022

Copy link


Copy Markdown

Member
@JabariBooker Integration data is generated in Python by Archery here: https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py#L196
You can find more info about the JSON integration format here, and how to run the integration tests locally, here: https://arrow.apache.org/docs/format/Integration.html
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a minor nit on the datagen (which we probably won't run into, but…)
Comment threaddev/archery/archery/integration/datagen.py
Outdated

 return super().generate_range(size, lower, upper, name)

 full_day_millis = 1000 * 60 * 60 * 24
 lower //= full_day_millis
@lidavidmlidavidmFeb 1, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python appears to round towards -Inf so a lower negative bound that is not exactly even will get rounded to an out-of-bound value
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we should be good here. This turned out to be quite a bit more involved than it looked at first.
@pitrou@edponce any other comments?
Comment threaddev/archery/archery/integration/tester_cpp.py
@ursabot
ursabot
commented
Feb 2, 2022•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 9b53235 and contender = c715beb. c715beb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.13% ⬆️0.0%] ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Feb 3, 2022

Closed

@JefffreyJefffrey
mentioned this pull request
Jan 7, 2024

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes
@pitroupitroupitrou left review comments

+1 more reviewer

@edponceedponceedponce approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@JabariBooker@edponce@pitrou@lidavidm@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull - #12014
Closed
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Closed
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull#12014
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Conversation
@JabariBooker
@JabariBookerJabariBooker
commented
Dec 21, 2021

Copy link


Copy Markdown

Contributor
Validate that temporal data in arrays meet the restrictions of their data type specifications.
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10924
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
lidavidm
lidavidm
reviewed
Dec 22, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
There are quite a few failing tests. We'll have to go through each of them individually to decide what to do about them.
You will want to lint and format the code.
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

lidavidm
lidavidm
reviewed
Dec 23, 2021
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.
You'll want to lint and format again (see the CI failure: https://github.com/apache/arrow/runs/4612596806?check_suite_focus=true)
Looking at test failures:
edponce
edponce
suggested changes
Dec 29, 2021
View reviewed changes
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

@edponce
edponce
commented
Dec 29, 2021

Copy link


Copy Markdown

Contributor
@JabariBooker Thanks for working on this! I left some minor comments.

Also, we are missing validation for the following temporal types: Date32Type and TimestampType.
@pitrou
pitrou
commented
Dec 30, 2021

Copy link


Copy Markdown

Member
@edponce There are no limits on Date32Type and TimestampType AFAICT.
@edponce
edponce
commented
Jan 6, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
@pitrou So there is a limit for Date64Type but not for Date32Type? I now understand that TimestampType does not have limits.
@lidavidm
lidavidm
commented
Jan 6, 2022

Copy link


Copy Markdown

Member
Date32Type's unit is days and so there's nothing to check, but Date64Type's unit is milliseconds, hence it doesn't have a limit per se, but it must be a multiple of 86400000 (=1 day in milliseconds).
@lidavidm
lidavidm
commented
Jan 11, 2022

Copy link


Copy Markdown

Member
FYI, that test failure in GroupBy.MinMaxTypes is here: 
arrow/cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

Line 1614
in
d88e232
TEST(GroupBy, MinMaxTypes) { 
The test uses one set of data to go through several data types, so you could just update the hardcoded data there too.
lidavidm
lidavidm
approved these changes
Jan 28, 2022
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I left some nits (we can check DataType::id() instead of the string name). And thanks for going through all the function tests; we should perhaps consider removing Date64 from the list of types to check in order to just split off those tests separately given the restrictions on value. (EDIT: not in this PR of course.)
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 SCOPED_TRACE(ty->ToString());
 auto in_schema = schema({field("argument0", ty), field("key", int64())});
 auto table =
 TableFromJSON(in_schema, (ty->name() == "date64") ? date64_table : default_table);
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can check ty->id() == Type::DATE64
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 [{"min": 3, "max": 5}, 4],
 [{"min": 1, "max": 4}, null]
 ])"),
 (ty->name() == "date64") ? date64_expected : default_expected),
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ditto here
Comment threadcpp/src/arrow/compute/kernels/vector_hash_test.cc
Outdated

 // Sliced
 CheckUnique(ArrayFromJSON(type, "[1, 2, null, 3, 2, null]")->Slice(1, 4),
 ArrayFromJSON(type, "[2, null, 3]"));
 if (type->name() == "date64") {
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here and so on (sorry)
Comment threadcpp/src/arrow/testing/random.cc
Outdated

 }

 case Type::type::TIME32: {
 TimeUnit::type unit = std::dynamic_pointer_cast<Time32Type>(field.type())->unit();
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have internal::checked_[pointer_]cast which will use dynamic or static cast depending on debug/release build
@lidavidmlidavidm
mentioned this pull request
Jan 28, 2022
 Closed

@JabariBooker
JabariBooker
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@lidavidm@edponce@pitrou There are still failing integration tests (all are also failing due to invalid values for date64). The data involved in these tests seem to be generated by some mechanism I'm unaware of.
edponce
edponce
reviewed
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/array/array_test.cc
edponce
edponce
suggested changes
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/testing/random.cc
 options.GenerateData(buffers[1]->mutable_data(), size);
 if (std::is_same<ArrowType, Date64Type>::value) {
 GenerateFullDayMillisNoNan(buffers[1]->mutable_data(), size);
 }
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an if-else. buffers[1]->mutable_data() is modified twice if Date64Type.
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more suitable place for this type-specific generation of numeric arrays, is in GenerateTypeDataNoNan() from GenerateOptions. Make GenerateFullDayMillisNoNan as a private member to GenerateOptions and perform the if-else check in GenerateTypedDataNoNan().
@JabariBookerJabariBookerJan 28, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Since we need to generate multiples of 86400000 for Date64Type, random numbers are generated in GenerateData and ultimately multiplied by 86400000 in GenerateFullDayMillisNoNan. The behavior of GenerateFullDayMillisNoNan is separated from GenerateOptions since this class doesn't know the arrow type, only the underlying C type. However, behavior of GenerateFullDayMillisNoNan could be moved into the above if statement if needed.
Comment threadcpp/src/arrow/testing/random.cc
@pitrou
pitrou
commented
Jan 31, 2022

Copy link


Copy Markdown

Member
@JabariBooker Integration data is generated in Python by Archery here: https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py#L196
You can find more info about the JSON integration format here, and how to run the integration tests locally, here: https://arrow.apache.org/docs/format/Integration.html
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a minor nit on the datagen (which we probably won't run into, but…)
Comment threaddev/archery/archery/integration/datagen.py
Outdated

 return super().generate_range(size, lower, upper, name)

 full_day_millis = 1000 * 60 * 60 * 24
 lower //= full_day_millis
@lidavidmlidavidmFeb 1, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python appears to round towards -Inf so a lower negative bound that is not exactly even will get rounded to an out-of-bound value
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we should be good here. This turned out to be quite a bit more involved than it looked at first.
@pitrou@edponce any other comments?
Comment threaddev/archery/archery/integration/tester_cpp.py
@ursabot
ursabot
commented
Feb 2, 2022•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 9b53235 and contender = c715beb. c715beb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.13% ⬆️0.0%] ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Feb 3, 2022

Closed

@JefffreyJefffrey
mentioned this pull request
Jan 7, 2024

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes
@pitroupitroupitrou left review comments

+1 more reviewer

@edponceedponceedponce approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@JabariBooker@edponce@pitrou@lidavidm@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull - #12014
Closed
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Closed
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull#12014
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Conversation
@JabariBooker
@JabariBookerJabariBooker
commented
Dec 21, 2021

Copy link


Copy Markdown

Contributor
Validate that temporal data in arrays meet the restrictions of their data type specifications.
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10924
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
lidavidm
lidavidm
reviewed
Dec 22, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
There are quite a few failing tests. We'll have to go through each of them individually to decide what to do about them.
You will want to lint and format the code.
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

lidavidm
lidavidm
reviewed
Dec 23, 2021
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.
You'll want to lint and format again (see the CI failure: https://github.com/apache/arrow/runs/4612596806?check_suite_focus=true)
Looking at test failures:
edponce
edponce
suggested changes
Dec 29, 2021
View reviewed changes
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

@edponce
edponce
commented
Dec 29, 2021

Copy link


Copy Markdown

Contributor
@JabariBooker Thanks for working on this! I left some minor comments.

Also, we are missing validation for the following temporal types: Date32Type and TimestampType.
@pitrou
pitrou
commented
Dec 30, 2021

Copy link


Copy Markdown

Member
@edponce There are no limits on Date32Type and TimestampType AFAICT.
@edponce
edponce
commented
Jan 6, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
@pitrou So there is a limit for Date64Type but not for Date32Type? I now understand that TimestampType does not have limits.
@lidavidm
lidavidm
commented
Jan 6, 2022

Copy link


Copy Markdown

Member
Date32Type's unit is days and so there's nothing to check, but Date64Type's unit is milliseconds, hence it doesn't have a limit per se, but it must be a multiple of 86400000 (=1 day in milliseconds).
@lidavidm
lidavidm
commented
Jan 11, 2022

Copy link


Copy Markdown

Member
FYI, that test failure in GroupBy.MinMaxTypes is here: 
arrow/cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

Line 1614
in
d88e232
TEST(GroupBy, MinMaxTypes) { 
The test uses one set of data to go through several data types, so you could just update the hardcoded data there too.
lidavidm
lidavidm
approved these changes
Jan 28, 2022
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I left some nits (we can check DataType::id() instead of the string name). And thanks for going through all the function tests; we should perhaps consider removing Date64 from the list of types to check in order to just split off those tests separately given the restrictions on value. (EDIT: not in this PR of course.)
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 SCOPED_TRACE(ty->ToString());
 auto in_schema = schema({field("argument0", ty), field("key", int64())});
 auto table =
 TableFromJSON(in_schema, (ty->name() == "date64") ? date64_table : default_table);
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can check ty->id() == Type::DATE64
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 [{"min": 3, "max": 5}, 4],
 [{"min": 1, "max": 4}, null]
 ])"),
 (ty->name() == "date64") ? date64_expected : default_expected),
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ditto here
Comment threadcpp/src/arrow/compute/kernels/vector_hash_test.cc
Outdated

 // Sliced
 CheckUnique(ArrayFromJSON(type, "[1, 2, null, 3, 2, null]")->Slice(1, 4),
 ArrayFromJSON(type, "[2, null, 3]"));
 if (type->name() == "date64") {
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here and so on (sorry)
Comment threadcpp/src/arrow/testing/random.cc
Outdated

 }

 case Type::type::TIME32: {
 TimeUnit::type unit = std::dynamic_pointer_cast<Time32Type>(field.type())->unit();
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have internal::checked_[pointer_]cast which will use dynamic or static cast depending on debug/release build
@lidavidmlidavidm
mentioned this pull request
Jan 28, 2022
 Closed

@JabariBooker
JabariBooker
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@lidavidm@edponce@pitrou There are still failing integration tests (all are also failing due to invalid values for date64). The data involved in these tests seem to be generated by some mechanism I'm unaware of.
edponce
edponce
reviewed
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/array/array_test.cc
edponce
edponce
suggested changes
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/testing/random.cc
 options.GenerateData(buffers[1]->mutable_data(), size);
 if (std::is_same<ArrowType, Date64Type>::value) {
 GenerateFullDayMillisNoNan(buffers[1]->mutable_data(), size);
 }
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an if-else. buffers[1]->mutable_data() is modified twice if Date64Type.
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more suitable place for this type-specific generation of numeric arrays, is in GenerateTypeDataNoNan() from GenerateOptions. Make GenerateFullDayMillisNoNan as a private member to GenerateOptions and perform the if-else check in GenerateTypedDataNoNan().
@JabariBookerJabariBookerJan 28, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Since we need to generate multiples of 86400000 for Date64Type, random numbers are generated in GenerateData and ultimately multiplied by 86400000 in GenerateFullDayMillisNoNan. The behavior of GenerateFullDayMillisNoNan is separated from GenerateOptions since this class doesn't know the arrow type, only the underlying C type. However, behavior of GenerateFullDayMillisNoNan could be moved into the above if statement if needed.
Comment threadcpp/src/arrow/testing/random.cc
@pitrou
pitrou
commented
Jan 31, 2022

Copy link


Copy Markdown

Member
@JabariBooker Integration data is generated in Python by Archery here: https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py#L196
You can find more info about the JSON integration format here, and how to run the integration tests locally, here: https://arrow.apache.org/docs/format/Integration.html
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a minor nit on the datagen (which we probably won't run into, but…)
Comment threaddev/archery/archery/integration/datagen.py
Outdated

 return super().generate_range(size, lower, upper, name)

 full_day_millis = 1000 * 60 * 60 * 24
 lower //= full_day_millis
@lidavidmlidavidmFeb 1, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python appears to round towards -Inf so a lower negative bound that is not exactly even will get rounded to an out-of-bound value
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we should be good here. This turned out to be quite a bit more involved than it looked at first.
@pitrou@edponce any other comments?
Comment threaddev/archery/archery/integration/tester_cpp.py
@ursabot
ursabot
commented
Feb 2, 2022•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 9b53235 and contender = c715beb. c715beb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.13% ⬆️0.0%] ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Feb 3, 2022

Closed

@JefffreyJefffrey
mentioned this pull request
Jan 7, 2024

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes
@pitroupitroupitrou left review comments

+1 more reviewer

@edponceedponceedponce approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@JabariBooker@edponce@pitrou@lidavidm@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull - #12014
Closed
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Closed
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull#12014
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Conversation
@JabariBooker
@JabariBookerJabariBooker
commented
Dec 21, 2021

Copy link


Copy Markdown

Contributor
Validate that temporal data in arrays meet the restrictions of their data type specifications.
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10924
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
lidavidm
lidavidm
reviewed
Dec 22, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
There are quite a few failing tests. We'll have to go through each of them individually to decide what to do about them.
You will want to lint and format the code.
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

lidavidm
lidavidm
reviewed
Dec 23, 2021
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.
You'll want to lint and format again (see the CI failure: https://github.com/apache/arrow/runs/4612596806?check_suite_focus=true)
Looking at test failures:
edponce
edponce
suggested changes
Dec 29, 2021
View reviewed changes
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

@edponce
edponce
commented
Dec 29, 2021

Copy link


Copy Markdown

Contributor
@JabariBooker Thanks for working on this! I left some minor comments.

Also, we are missing validation for the following temporal types: Date32Type and TimestampType.
@pitrou
pitrou
commented
Dec 30, 2021

Copy link


Copy Markdown

Member
@edponce There are no limits on Date32Type and TimestampType AFAICT.
@edponce
edponce
commented
Jan 6, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
@pitrou So there is a limit for Date64Type but not for Date32Type? I now understand that TimestampType does not have limits.
@lidavidm
lidavidm
commented
Jan 6, 2022

Copy link


Copy Markdown

Member
Date32Type's unit is days and so there's nothing to check, but Date64Type's unit is milliseconds, hence it doesn't have a limit per se, but it must be a multiple of 86400000 (=1 day in milliseconds).
@lidavidm
lidavidm
commented
Jan 11, 2022

Copy link


Copy Markdown

Member
FYI, that test failure in GroupBy.MinMaxTypes is here: 
arrow/cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

Line 1614
in
d88e232
TEST(GroupBy, MinMaxTypes) { 
The test uses one set of data to go through several data types, so you could just update the hardcoded data there too.
lidavidm
lidavidm
approved these changes
Jan 28, 2022
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I left some nits (we can check DataType::id() instead of the string name). And thanks for going through all the function tests; we should perhaps consider removing Date64 from the list of types to check in order to just split off those tests separately given the restrictions on value. (EDIT: not in this PR of course.)
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 SCOPED_TRACE(ty->ToString());
 auto in_schema = schema({field("argument0", ty), field("key", int64())});
 auto table =
 TableFromJSON(in_schema, (ty->name() == "date64") ? date64_table : default_table);
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can check ty->id() == Type::DATE64
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 [{"min": 3, "max": 5}, 4],
 [{"min": 1, "max": 4}, null]
 ])"),
 (ty->name() == "date64") ? date64_expected : default_expected),
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ditto here
Comment threadcpp/src/arrow/compute/kernels/vector_hash_test.cc
Outdated

 // Sliced
 CheckUnique(ArrayFromJSON(type, "[1, 2, null, 3, 2, null]")->Slice(1, 4),
 ArrayFromJSON(type, "[2, null, 3]"));
 if (type->name() == "date64") {
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here and so on (sorry)
Comment threadcpp/src/arrow/testing/random.cc
Outdated

 }

 case Type::type::TIME32: {
 TimeUnit::type unit = std::dynamic_pointer_cast<Time32Type>(field.type())->unit();
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have internal::checked_[pointer_]cast which will use dynamic or static cast depending on debug/release build
@lidavidmlidavidm
mentioned this pull request
Jan 28, 2022
 Closed

@JabariBooker
JabariBooker
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@lidavidm@edponce@pitrou There are still failing integration tests (all are also failing due to invalid values for date64). The data involved in these tests seem to be generated by some mechanism I'm unaware of.
edponce
edponce
reviewed
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/array/array_test.cc
edponce
edponce
suggested changes
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/testing/random.cc
 options.GenerateData(buffers[1]->mutable_data(), size);
 if (std::is_same<ArrowType, Date64Type>::value) {
 GenerateFullDayMillisNoNan(buffers[1]->mutable_data(), size);
 }
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an if-else. buffers[1]->mutable_data() is modified twice if Date64Type.
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more suitable place for this type-specific generation of numeric arrays, is in GenerateTypeDataNoNan() from GenerateOptions. Make GenerateFullDayMillisNoNan as a private member to GenerateOptions and perform the if-else check in GenerateTypedDataNoNan().
@JabariBookerJabariBookerJan 28, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Since we need to generate multiples of 86400000 for Date64Type, random numbers are generated in GenerateData and ultimately multiplied by 86400000 in GenerateFullDayMillisNoNan. The behavior of GenerateFullDayMillisNoNan is separated from GenerateOptions since this class doesn't know the arrow type, only the underlying C type. However, behavior of GenerateFullDayMillisNoNan could be moved into the above if statement if needed.
Comment threadcpp/src/arrow/testing/random.cc
@pitrou
pitrou
commented
Jan 31, 2022

Copy link


Copy Markdown

Member
@JabariBooker Integration data is generated in Python by Archery here: https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py#L196
You can find more info about the JSON integration format here, and how to run the integration tests locally, here: https://arrow.apache.org/docs/format/Integration.html
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a minor nit on the datagen (which we probably won't run into, but…)
Comment threaddev/archery/archery/integration/datagen.py
Outdated

 return super().generate_range(size, lower, upper, name)

 full_day_millis = 1000 * 60 * 60 * 24
 lower //= full_day_millis
@lidavidmlidavidmFeb 1, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python appears to round towards -Inf so a lower negative bound that is not exactly even will get rounded to an out-of-bound value
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we should be good here. This turned out to be quite a bit more involved than it looked at first.
@pitrou@edponce any other comments?
Comment threaddev/archery/archery/integration/tester_cpp.py
@ursabot
ursabot
commented
Feb 2, 2022•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 9b53235 and contender = c715beb. c715beb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.13% ⬆️0.0%] ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Feb 3, 2022

Closed

@JefffreyJefffrey
mentioned this pull request
Jan 7, 2024

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes
@pitroupitroupitrou left review comments

+1 more reviewer

@edponceedponceedponce approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@JabariBooker@edponce@pitrou@lidavidm@ursabot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull - #12014
Closed
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Closed
ARROW-10924: [C++] Validate temporal data in ValidateArrayFull#12014
JabariBooker wants to merge 21 commits into
apache:masterfrom 
JabariBooker:ARROW-10924
Conversation
@JabariBooker
@JabariBookerJabariBooker
commented
Dec 21, 2021

Copy link


Copy Markdown

Contributor
Validate that temporal data in arrays meet the restrictions of their data type specifications.
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10924
@github-actions
github-actionsBot
commented
Dec 21, 2021

Copy link


Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
lidavidm
lidavidm
reviewed
Dec 22, 2021
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
There are quite a few failing tests. We'll have to go through each of them individually to decide what to do about them.
You will want to lint and format the code.
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

lidavidm
lidavidm
reviewed
Dec 23, 2021
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the updates.
You'll want to lint and format again (see the CI failure: https://github.com/apache/arrow/runs/4612596806?check_suite_focus=true)
Looking at test failures:
edponce
edponce
suggested changes
Dec 29, 2021
View reviewed changes
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Comment threadcpp/src/arrow/array/validate.cc
Outdated

Comment threadcpp/src/arrow/array/validate.cc
Outdated

@edponce
edponce
commented
Dec 29, 2021

Copy link


Copy Markdown

Contributor
@JabariBooker Thanks for working on this! I left some minor comments.

Also, we are missing validation for the following temporal types: Date32Type and TimestampType.
@pitrou
pitrou
commented
Dec 30, 2021

Copy link


Copy Markdown

Member
@edponce There are no limits on Date32Type and TimestampType AFAICT.
@edponce
edponce
commented
Jan 6, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
@pitrou So there is a limit for Date64Type but not for Date32Type? I now understand that TimestampType does not have limits.
@lidavidm
lidavidm
commented
Jan 6, 2022

Copy link


Copy Markdown

Member
Date32Type's unit is days and so there's nothing to check, but Date64Type's unit is milliseconds, hence it doesn't have a limit per se, but it must be a multiple of 86400000 (=1 day in milliseconds).
@lidavidm
lidavidm
commented
Jan 11, 2022

Copy link


Copy Markdown

Member
FYI, that test failure in GroupBy.MinMaxTypes is here: 
arrow/cpp/src/arrow/compute/kernels/hash_aggregate_test.cc

Line 1614
in
d88e232
TEST(GroupBy, MinMaxTypes) { 
The test uses one set of data to go through several data types, so you could just update the hardcoded data there too.
lidavidm
lidavidm
approved these changes
Jan 28, 2022
View reviewed changes
@lidavidmlidavidm
left a comment
•

edited

Loading

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM. I left some nits (we can check DataType::id() instead of the string name). And thanks for going through all the function tests; we should perhaps consider removing Date64 from the list of types to check in order to just split off those tests separately given the restrictions on value. (EDIT: not in this PR of course.)
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 SCOPED_TRACE(ty->ToString());
 auto in_schema = schema({field("argument0", ty), field("key", int64())});
 auto table =
 TableFromJSON(in_schema, (ty->name() == "date64") ? date64_table : default_table);
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can check ty->id() == Type::DATE64
Comment threadcpp/src/arrow/compute/kernels/hash_aggregate_test.cc
 [{"min": 3, "max": 5}, 4],
 [{"min": 1, "max": 4}, null]
 ])"),
 (ty->name() == "date64") ? date64_expected : default_expected),
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: ditto here
Comment threadcpp/src/arrow/compute/kernels/vector_hash_test.cc
Outdated

 // Sliced
 CheckUnique(ArrayFromJSON(type, "[1, 2, null, 3, 2, null]")->Slice(1, 4),
 ArrayFromJSON(type, "[2, null, 3]"));
 if (type->name() == "date64") {
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and here and so on (sorry)
Comment threadcpp/src/arrow/testing/random.cc
Outdated

 }

 case Type::type::TIME32: {
 TimeUnit::type unit = std::dynamic_pointer_cast<Time32Type>(field.type())->unit();
@lidavidmlidavidmJan 28, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we have internal::checked_[pointer_]cast which will use dynamic or static cast depending on debug/release build
@lidavidmlidavidm
mentioned this pull request
Jan 28, 2022
 Closed

@JabariBooker
JabariBooker
commented
Jan 28, 2022

Copy link


Copy Markdown

ContributorAuthor
@lidavidm@edponce@pitrou There are still failing integration tests (all are also failing due to invalid values for date64). The data involved in these tests seem to be generated by some mechanism I'm unaware of.
edponce
edponce
reviewed
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/array/array_test.cc
edponce
edponce
suggested changes
Jan 28, 2022
View reviewed changes
Comment threadcpp/src/arrow/testing/random.cc
 options.GenerateData(buffers[1]->mutable_data(), size);
 if (std::is_same<ArrowType, Date64Type>::value) {
 GenerateFullDayMillisNoNan(buffers[1]->mutable_data(), size);
 }
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an if-else. buffers[1]->mutable_data() is modified twice if Date64Type.
@edponceedponceJan 28, 2022

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a more suitable place for this type-specific generation of numeric arrays, is in GenerateTypeDataNoNan() from GenerateOptions. Make GenerateFullDayMillisNoNan as a private member to GenerateOptions and perform the if-else check in GenerateTypedDataNoNan().
@JabariBookerJabariBookerJan 28, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intentional. Since we need to generate multiples of 86400000 for Date64Type, random numbers are generated in GenerateData and ultimately multiplied by 86400000 in GenerateFullDayMillisNoNan. The behavior of GenerateFullDayMillisNoNan is separated from GenerateOptions since this class doesn't know the arrow type, only the underlying C type. However, behavior of GenerateFullDayMillisNoNan could be moved into the above if statement if needed.
Comment threadcpp/src/arrow/testing/random.cc
@pitrou
pitrou
commented
Jan 31, 2022

Copy link


Copy Markdown

Member
@JabariBooker Integration data is generated in Python by Archery here: https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/datagen.py#L196
You can find more info about the JSON integration format here, and how to run the integration tests locally, here: https://arrow.apache.org/docs/format/Integration.html
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just a minor nit on the datagen (which we probably won't run into, but…)
Comment threaddev/archery/archery/integration/datagen.py
Outdated

 return super().generate_range(size, lower, upper, name)

 full_day_millis = 1000 * 60 * 60 * 24
 lower //= full_day_millis
@lidavidmlidavidmFeb 1, 2022

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python appears to round towards -Inf so a lower negative bound that is not exactly even will get rounded to an out-of-bound value
lidavidm
lidavidm
reviewed
Feb 1, 2022
View reviewed changes
@lidavidmlidavidm
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I think we should be good here. This turned out to be quite a bit more involved than it looked at first.
@pitrou@edponce any other comments?
Comment threaddev/archery/archery/integration/tester_cpp.py
@ursabot
ursabot
commented
Feb 2, 2022•

edited

Loading

Copy link


Copy Markdown

Benchmark runs are scheduled for baseline = 9b53235 and contender = c715beb. c715beb is a master commit associated with this PR. Results will be available as each benchmark for each run completes.

Conbench compare runs links:

[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2

[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x

[Finished ⬇️0.13% ⬆️0.0%] ursa-thinkcentre-m75q

Supported benchmarks:

ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python. Runs only benchmarks with cloud = True

ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript

ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
@asfimportasfimport
mentioned this pull request
Feb 3, 2022

Closed

@JefffreyJefffrey
mentioned this pull request
Jan 7, 2024

Closed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@lidavidmlidavidmlidavidm approved these changes
@pitroupitroupitrou left review comments

+1 more reviewer

@edponceedponceedponce approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

Component: C++

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@JabariBooker@edponce@pitrou@lidavidm@ursabot