ARROW-13168: [C++][R] Enable runtime timezone database for Windows

[wjones127]

This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.

For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of strftime() and friends.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13168

[wjones127]

So timezone database seems to work, but methods that rely on std::local currently error with:

Error (test-dplyr-funcs-datetime.R:344:3): extract month from timestamp
Error: Invalid: Cannot find locale 'English_United States.1252': locale::facet::_S_create_c_locale name not valid
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1085 GetLocale(options.locale)
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1105 Make(ctx, *in.type)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:700 kernel_->exec(kernel_ctx_, batch, &out)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:641 ExecuteBatch(batch, listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:547 executor->Execute(arguments, &listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:533 ExecuteScalarExpression(call->arguments[i], input, exec_context)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/project_node.cc:91 ExecuteScalarExpression(simplified_expr, target, plan()->exec_context())
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:484 iterator_.Next()
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)

These do work if you set Sys.setlocale("LC_TIME", "C"). If I don't find a fix for this, I may consider only supporting the "C" locale on Windows.

[wjones127]

CI failure is unrelated Flight error.

[wjones127]

cc @pitrou

[rok]

Will this always be the same DB that R and Python will be using?

[wjones127]

No that 2021e is a version, which won't necessarily align with the R and Python ones in the future. I don't think it matters that we update it, unless the format changes somehow. This is just for testing that it works, and we don't ship it.

But the R unit tests use the one provided by the tzdb package, so we are testing that as well.

[rok]

Currently US and EU have DST, but they plan to abolish it soon. At that point we will have Python and R DBs with fresh DSTless times and arrow c++ using DST for tests. In isolation that's ok, but we do have tests comparing pandas and pyarrow results and similar for lubridate. It's not a big problem for sure, but if there is like a tzdata-latest.tar.gz that would be great.

[jorisvandenbossche]

The times in the past won't change, so unless we test against a random "now", updates to the timezone database (such as for DST policy changes) shouldn't impact those tests?

[wjones127]

Yeah this should be fine as is. We will never be testing between timezone databases.

[rok]

We currently test arrow vs pandas tzdb in CI. We have a test for a timestamp in 2033 and if it is in DST and DST is abolished it will error if we'll be using a pre-abolishment db with arrow and post-abolishment db with pandas. This is super irrelevant and as it's a simple fix and I'm sorry for wasting your time :).

[wjones127]

Ah got it. We'll address in the follow up PR where we'll have PyArrow use tzdb. For now the python timestamp tests are skipped on Windows

arrow/python/pyarrow/tests/test_compute.py

Lines 1915 to 1917 in d327f69

	ifsys.platform=='win32':
	# TODO: We should test on windows once ARROW-13168 is resolved.
	pytest.skip('Timezone database is not available on Windows yet')

[pitrou]

Thanks a lot for this @wjones127 . Here are some comments.

[pitrou]

I'd rather if you copied the relevant snippet here, we shouldn't ideally rely on the contents of CI scripts (which may contain specific quirks that irrelevant to normal user setups) for the public docs.

[wjones127]

Since we are pulling a specific part of the script, we should easily be able to avoid quirks? I like the idea of having our build instructions tested in CI. And we reference the CI scripts regularly when providing build instructions.

At the very least, if we get to a point where the public instructions and CI instructions diverge, we can easily separate them.

[pitrou]

Is this on Windows or specifically MinGW? i.e., would the non-MinGW Windows CI pass if you remove this skip? I'm wondering if we can narrow the condition.

[wjones127]

If failed on Windows 2019 C++17, which uses MSVC. But seems like it was actually passing on MinGW. So maybe I can try skipping for just MSVC?

[pitrou]

That's a bit weird, MinGW is just a different compiler but targetting the same runtime libraries...

[wjones127]

Actually, I think the test is likely skipping on the LocaleExists("fr_FR.UTF-8") condition; IIRC MinGW doesn't support locales apart from "C" and "POSIX". Sadly, no indication in CI whether the test was skipped: https://github.com/apache/arrow/runs/5504310182?check_suite_focus=true

[pitrou]

Is this a reliable location? Is there a risk that the downloads folder gets cleared from time to time?

[wjones127]

This is the behavior of the vendored datetime library, not something we chose. I think it's a fine default for testing and in production applications I expect users will manually specify a more appropriate path at runtime.

[jonkeane]

A few things I noticed in CI

[DavisVaughan]

FWIW, they just released 2022a a few days ago

[jonkeane]

Are there any outstanding comments that we need to resolve before merging?

The failures in CI both look unrelated. I'm happy to merge if no one objects

[ursabot]

Benchmark runs are scheduled for baseline = 919d113 and contender = f4dfd6c. f4dfd6c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.36% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.02% ⬆️0.81%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[wjones127]

Follow-up Jira created: https://issues.apache.org/jira/browse/ARROW-16054

[wjones127]

This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.

For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of strftime() and friends.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13168

[wjones127]

So timezone database seems to work, but methods that rely on std::local currently error with:

Error (test-dplyr-funcs-datetime.R:344:3): extract month from timestamp
Error: Invalid: Cannot find locale 'English_United States.1252': locale::facet::_S_create_c_locale name not valid
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1085 GetLocale(options.locale)
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1105 Make(ctx, *in.type)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:700 kernel_->exec(kernel_ctx_, batch, &out)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:641 ExecuteBatch(batch, listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:547 executor->Execute(arguments, &listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:533 ExecuteScalarExpression(call->arguments[i], input, exec_context)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/project_node.cc:91 ExecuteScalarExpression(simplified_expr, target, plan()->exec_context())
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:484 iterator_.Next()
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)

These do work if you set Sys.setlocale("LC_TIME", "C"). If I don't find a fix for this, I may consider only supporting the "C" locale on Windows.

[wjones127]

CI failure is unrelated Flight error.

[wjones127]

cc @pitrou

[rok]

Will this always be the same DB that R and Python will be using?

[wjones127]

No that 2021e is a version, which won't necessarily align with the R and Python ones in the future. I don't think it matters that we update it, unless the format changes somehow. This is just for testing that it works, and we don't ship it.

But the R unit tests use the one provided by the tzdb package, so we are testing that as well.

[rok]

Currently US and EU have DST, but they plan to abolish it soon. At that point we will have Python and R DBs with fresh DSTless times and arrow c++ using DST for tests. In isolation that's ok, but we do have tests comparing pandas and pyarrow results and similar for lubridate. It's not a big problem for sure, but if there is like a tzdata-latest.tar.gz that would be great.

[jorisvandenbossche]

The times in the past won't change, so unless we test against a random "now", updates to the timezone database (such as for DST policy changes) shouldn't impact those tests?

[wjones127]

Yeah this should be fine as is. We will never be testing between timezone databases.

[rok]

We currently test arrow vs pandas tzdb in CI. We have a test for a timestamp in 2033 and if it is in DST and DST is abolished it will error if we'll be using a pre-abolishment db with arrow and post-abolishment db with pandas. This is super irrelevant and as it's a simple fix and I'm sorry for wasting your time :).

[wjones127]

Ah got it. We'll address in the follow up PR where we'll have PyArrow use tzdb. For now the python timestamp tests are skipped on Windows

arrow/python/pyarrow/tests/test_compute.py

Lines 1915 to 1917 in d327f69

	ifsys.platform=='win32':
	# TODO: We should test on windows once ARROW-13168 is resolved.
	pytest.skip('Timezone database is not available on Windows yet')

[pitrou]

Thanks a lot for this @wjones127 . Here are some comments.

[pitrou]

I'd rather if you copied the relevant snippet here, we shouldn't ideally rely on the contents of CI scripts (which may contain specific quirks that irrelevant to normal user setups) for the public docs.

[wjones127]

Since we are pulling a specific part of the script, we should easily be able to avoid quirks? I like the idea of having our build instructions tested in CI. And we reference the CI scripts regularly when providing build instructions.

At the very least, if we get to a point where the public instructions and CI instructions diverge, we can easily separate them.

[pitrou]

Is this on Windows or specifically MinGW? i.e., would the non-MinGW Windows CI pass if you remove this skip? I'm wondering if we can narrow the condition.

[wjones127]

If failed on Windows 2019 C++17, which uses MSVC. But seems like it was actually passing on MinGW. So maybe I can try skipping for just MSVC?

[pitrou]

That's a bit weird, MinGW is just a different compiler but targetting the same runtime libraries...

[wjones127]

Actually, I think the test is likely skipping on the LocaleExists("fr_FR.UTF-8") condition; IIRC MinGW doesn't support locales apart from "C" and "POSIX". Sadly, no indication in CI whether the test was skipped: https://github.com/apache/arrow/runs/5504310182?check_suite_focus=true

[pitrou]

Is this a reliable location? Is there a risk that the downloads folder gets cleared from time to time?

[wjones127]

This is the behavior of the vendored datetime library, not something we chose. I think it's a fine default for testing and in production applications I expect users will manually specify a more appropriate path at runtime.

[jonkeane]

A few things I noticed in CI

[DavisVaughan]

FWIW, they just released 2022a a few days ago

[jonkeane]

Are there any outstanding comments that we need to resolve before merging?

The failures in CI both look unrelated. I'm happy to merge if no one objects

[ursabot]

Benchmark runs are scheduled for baseline = 919d113 and contender = f4dfd6c. f4dfd6c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.36% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.02% ⬆️0.81%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[wjones127]

Follow-up Jira created: https://issues.apache.org/jira/browse/ARROW-16054

[wjones127]

This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.

For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of strftime() and friends.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13168

[wjones127]

So timezone database seems to work, but methods that rely on std::local currently error with:

Error (test-dplyr-funcs-datetime.R:344:3): extract month from timestamp
Error: Invalid: Cannot find locale 'English_United States.1252': locale::facet::_S_create_c_locale name not valid
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1085 GetLocale(options.locale)
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1105 Make(ctx, *in.type)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:700 kernel_->exec(kernel_ctx_, batch, &out)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:641 ExecuteBatch(batch, listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:547 executor->Execute(arguments, &listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:533 ExecuteScalarExpression(call->arguments[i], input, exec_context)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/project_node.cc:91 ExecuteScalarExpression(simplified_expr, target, plan()->exec_context())
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:484 iterator_.Next()
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)

These do work if you set Sys.setlocale("LC_TIME", "C"). If I don't find a fix for this, I may consider only supporting the "C" locale on Windows.

[wjones127]

CI failure is unrelated Flight error.

[wjones127]

cc @pitrou

[rok]

Will this always be the same DB that R and Python will be using?

[wjones127]

No that 2021e is a version, which won't necessarily align with the R and Python ones in the future. I don't think it matters that we update it, unless the format changes somehow. This is just for testing that it works, and we don't ship it.

But the R unit tests use the one provided by the tzdb package, so we are testing that as well.

[rok]

Currently US and EU have DST, but they plan to abolish it soon. At that point we will have Python and R DBs with fresh DSTless times and arrow c++ using DST for tests. In isolation that's ok, but we do have tests comparing pandas and pyarrow results and similar for lubridate. It's not a big problem for sure, but if there is like a tzdata-latest.tar.gz that would be great.

[jorisvandenbossche]

The times in the past won't change, so unless we test against a random "now", updates to the timezone database (such as for DST policy changes) shouldn't impact those tests?

[wjones127]

Yeah this should be fine as is. We will never be testing between timezone databases.

[rok]

We currently test arrow vs pandas tzdb in CI. We have a test for a timestamp in 2033 and if it is in DST and DST is abolished it will error if we'll be using a pre-abolishment db with arrow and post-abolishment db with pandas. This is super irrelevant and as it's a simple fix and I'm sorry for wasting your time :).

[wjones127]

Ah got it. We'll address in the follow up PR where we'll have PyArrow use tzdb. For now the python timestamp tests are skipped on Windows

arrow/python/pyarrow/tests/test_compute.py

Lines 1915 to 1917 in d327f69

	ifsys.platform=='win32':
	# TODO: We should test on windows once ARROW-13168 is resolved.
	pytest.skip('Timezone database is not available on Windows yet')

[pitrou]

Thanks a lot for this @wjones127 . Here are some comments.

[pitrou]

I'd rather if you copied the relevant snippet here, we shouldn't ideally rely on the contents of CI scripts (which may contain specific quirks that irrelevant to normal user setups) for the public docs.

[wjones127]

Since we are pulling a specific part of the script, we should easily be able to avoid quirks? I like the idea of having our build instructions tested in CI. And we reference the CI scripts regularly when providing build instructions.

At the very least, if we get to a point where the public instructions and CI instructions diverge, we can easily separate them.

[pitrou]

Is this on Windows or specifically MinGW? i.e., would the non-MinGW Windows CI pass if you remove this skip? I'm wondering if we can narrow the condition.

[wjones127]

If failed on Windows 2019 C++17, which uses MSVC. But seems like it was actually passing on MinGW. So maybe I can try skipping for just MSVC?

[pitrou]

That's a bit weird, MinGW is just a different compiler but targetting the same runtime libraries...

[wjones127]

Actually, I think the test is likely skipping on the LocaleExists("fr_FR.UTF-8") condition; IIRC MinGW doesn't support locales apart from "C" and "POSIX". Sadly, no indication in CI whether the test was skipped: https://github.com/apache/arrow/runs/5504310182?check_suite_focus=true

[pitrou]

Is this a reliable location? Is there a risk that the downloads folder gets cleared from time to time?

[wjones127]

This is the behavior of the vendored datetime library, not something we chose. I think it's a fine default for testing and in production applications I expect users will manually specify a more appropriate path at runtime.

[jonkeane]

A few things I noticed in CI

[DavisVaughan]

FWIW, they just released 2022a a few days ago

[jonkeane]

Are there any outstanding comments that we need to resolve before merging?

The failures in CI both look unrelated. I'm happy to merge if no one objects

[ursabot]

Benchmark runs are scheduled for baseline = 919d113 and contender = f4dfd6c. f4dfd6c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.36% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.02% ⬆️0.81%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[wjones127]

Follow-up Jira created: https://issues.apache.org/jira/browse/ARROW-16054

[wjones127]

This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.

For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of strftime() and friends.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13168

[wjones127]

So timezone database seems to work, but methods that rely on std::local currently error with:

Error (test-dplyr-funcs-datetime.R:344:3): extract month from timestamp
Error: Invalid: Cannot find locale 'English_United States.1252': locale::facet::_S_create_c_locale name not valid
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1085 GetLocale(options.locale)
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1105 Make(ctx, *in.type)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:700 kernel_->exec(kernel_ctx_, batch, &out)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:641 ExecuteBatch(batch, listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:547 executor->Execute(arguments, &listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:533 ExecuteScalarExpression(call->arguments[i], input, exec_context)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/project_node.cc:91 ExecuteScalarExpression(simplified_expr, target, plan()->exec_context())
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:484 iterator_.Next()
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)

These do work if you set Sys.setlocale("LC_TIME", "C"). If I don't find a fix for this, I may consider only supporting the "C" locale on Windows.

[wjones127]

CI failure is unrelated Flight error.

[wjones127]

cc @pitrou

[rok]

Will this always be the same DB that R and Python will be using?

[wjones127]

No that 2021e is a version, which won't necessarily align with the R and Python ones in the future. I don't think it matters that we update it, unless the format changes somehow. This is just for testing that it works, and we don't ship it.

But the R unit tests use the one provided by the tzdb package, so we are testing that as well.

[rok]

Currently US and EU have DST, but they plan to abolish it soon. At that point we will have Python and R DBs with fresh DSTless times and arrow c++ using DST for tests. In isolation that's ok, but we do have tests comparing pandas and pyarrow results and similar for lubridate. It's not a big problem for sure, but if there is like a tzdata-latest.tar.gz that would be great.

[jorisvandenbossche]

The times in the past won't change, so unless we test against a random "now", updates to the timezone database (such as for DST policy changes) shouldn't impact those tests?

[wjones127]

Yeah this should be fine as is. We will never be testing between timezone databases.

[rok]

We currently test arrow vs pandas tzdb in CI. We have a test for a timestamp in 2033 and if it is in DST and DST is abolished it will error if we'll be using a pre-abolishment db with arrow and post-abolishment db with pandas. This is super irrelevant and as it's a simple fix and I'm sorry for wasting your time :).

[wjones127]

Ah got it. We'll address in the follow up PR where we'll have PyArrow use tzdb. For now the python timestamp tests are skipped on Windows

arrow/python/pyarrow/tests/test_compute.py

Lines 1915 to 1917 in d327f69

	ifsys.platform=='win32':
	# TODO: We should test on windows once ARROW-13168 is resolved.
	pytest.skip('Timezone database is not available on Windows yet')

[pitrou]

Thanks a lot for this @wjones127 . Here are some comments.

[pitrou]

I'd rather if you copied the relevant snippet here, we shouldn't ideally rely on the contents of CI scripts (which may contain specific quirks that irrelevant to normal user setups) for the public docs.

[wjones127]

Since we are pulling a specific part of the script, we should easily be able to avoid quirks? I like the idea of having our build instructions tested in CI. And we reference the CI scripts regularly when providing build instructions.

At the very least, if we get to a point where the public instructions and CI instructions diverge, we can easily separate them.

[pitrou]

Is this on Windows or specifically MinGW? i.e., would the non-MinGW Windows CI pass if you remove this skip? I'm wondering if we can narrow the condition.

[wjones127]

If failed on Windows 2019 C++17, which uses MSVC. But seems like it was actually passing on MinGW. So maybe I can try skipping for just MSVC?

[pitrou]

That's a bit weird, MinGW is just a different compiler but targetting the same runtime libraries...

[wjones127]

Actually, I think the test is likely skipping on the LocaleExists("fr_FR.UTF-8") condition; IIRC MinGW doesn't support locales apart from "C" and "POSIX". Sadly, no indication in CI whether the test was skipped: https://github.com/apache/arrow/runs/5504310182?check_suite_focus=true

[pitrou]

Is this a reliable location? Is there a risk that the downloads folder gets cleared from time to time?

[wjones127]

This is the behavior of the vendored datetime library, not something we chose. I think it's a fine default for testing and in production applications I expect users will manually specify a more appropriate path at runtime.

[jonkeane]

A few things I noticed in CI

[DavisVaughan]

FWIW, they just released 2022a a few days ago

[jonkeane]

Are there any outstanding comments that we need to resolve before merging?

The failures in CI both look unrelated. I'm happy to merge if no one objects

[ursabot]

Benchmark runs are scheduled for baseline = 919d113 and contender = f4dfd6c. f4dfd6c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.36% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.02% ⬆️0.81%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[wjones127]

Follow-up Jira created: https://issues.apache.org/jira/browse/ARROW-16054

[wjones127]

This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.

For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of strftime() and friends.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13168

[wjones127]

So timezone database seems to work, but methods that rely on std::local currently error with:

Error (test-dplyr-funcs-datetime.R:344:3): extract month from timestamp
Error: Invalid: Cannot find locale 'English_United States.1252': locale::facet::_S_create_c_locale name not valid
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1085 GetLocale(options.locale)
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1105 Make(ctx, *in.type)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:700 kernel_->exec(kernel_ctx_, batch, &out)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:641 ExecuteBatch(batch, listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:547 executor->Execute(arguments, &listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:533 ExecuteScalarExpression(call->arguments[i], input, exec_context)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/project_node.cc:91 ExecuteScalarExpression(simplified_expr, target, plan()->exec_context())
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:484 iterator_.Next()
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)

These do work if you set Sys.setlocale("LC_TIME", "C"). If I don't find a fix for this, I may consider only supporting the "C" locale on Windows.

[wjones127]

CI failure is unrelated Flight error.

[wjones127]

cc @pitrou

[rok]

Will this always be the same DB that R and Python will be using?

[wjones127]

No that 2021e is a version, which won't necessarily align with the R and Python ones in the future. I don't think it matters that we update it, unless the format changes somehow. This is just for testing that it works, and we don't ship it.

But the R unit tests use the one provided by the tzdb package, so we are testing that as well.

[rok]

Currently US and EU have DST, but they plan to abolish it soon. At that point we will have Python and R DBs with fresh DSTless times and arrow c++ using DST for tests. In isolation that's ok, but we do have tests comparing pandas and pyarrow results and similar for lubridate. It's not a big problem for sure, but if there is like a tzdata-latest.tar.gz that would be great.

[jorisvandenbossche]

The times in the past won't change, so unless we test against a random "now", updates to the timezone database (such as for DST policy changes) shouldn't impact those tests?

[wjones127]

Yeah this should be fine as is. We will never be testing between timezone databases.

[rok]

We currently test arrow vs pandas tzdb in CI. We have a test for a timestamp in 2033 and if it is in DST and DST is abolished it will error if we'll be using a pre-abolishment db with arrow and post-abolishment db with pandas. This is super irrelevant and as it's a simple fix and I'm sorry for wasting your time :).

[wjones127]

Ah got it. We'll address in the follow up PR where we'll have PyArrow use tzdb. For now the python timestamp tests are skipped on Windows

arrow/python/pyarrow/tests/test_compute.py

Lines 1915 to 1917 in d327f69

	ifsys.platform=='win32':
	# TODO: We should test on windows once ARROW-13168 is resolved.
	pytest.skip('Timezone database is not available on Windows yet')

[pitrou]

Thanks a lot for this @wjones127 . Here are some comments.

[pitrou]

I'd rather if you copied the relevant snippet here, we shouldn't ideally rely on the contents of CI scripts (which may contain specific quirks that irrelevant to normal user setups) for the public docs.

[wjones127]

Since we are pulling a specific part of the script, we should easily be able to avoid quirks? I like the idea of having our build instructions tested in CI. And we reference the CI scripts regularly when providing build instructions.

At the very least, if we get to a point where the public instructions and CI instructions diverge, we can easily separate them.

[pitrou]

Is this on Windows or specifically MinGW? i.e., would the non-MinGW Windows CI pass if you remove this skip? I'm wondering if we can narrow the condition.

[wjones127]

If failed on Windows 2019 C++17, which uses MSVC. But seems like it was actually passing on MinGW. So maybe I can try skipping for just MSVC?

[pitrou]

That's a bit weird, MinGW is just a different compiler but targetting the same runtime libraries...

[wjones127]

Actually, I think the test is likely skipping on the LocaleExists("fr_FR.UTF-8") condition; IIRC MinGW doesn't support locales apart from "C" and "POSIX". Sadly, no indication in CI whether the test was skipped: https://github.com/apache/arrow/runs/5504310182?check_suite_focus=true

[pitrou]

Is this a reliable location? Is there a risk that the downloads folder gets cleared from time to time?

[wjones127]

This is the behavior of the vendored datetime library, not something we chose. I think it's a fine default for testing and in production applications I expect users will manually specify a more appropriate path at runtime.

[jonkeane]

A few things I noticed in CI

[DavisVaughan]

FWIW, they just released 2022a a few days ago

[jonkeane]

Are there any outstanding comments that we need to resolve before merging?

The failures in CI both look unrelated. I'm happy to merge if no one objects

[ursabot]

Benchmark runs are scheduled for baseline = 919d113 and contender = f4dfd6c. f4dfd6c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.36% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.02% ⬆️0.81%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[wjones127]

Follow-up Jira created: https://issues.apache.org/jira/browse/ARROW-16054

[wjones127]

This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.

For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of strftime() and friends.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13168

[wjones127]

So timezone database seems to work, but methods that rely on std::local currently error with:

Error (test-dplyr-funcs-datetime.R:344:3): extract month from timestamp
Error: Invalid: Cannot find locale 'English_United States.1252': locale::facet::_S_create_c_locale name not valid
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1085 GetLocale(options.locale)
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1105 Make(ctx, *in.type)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:700 kernel_->exec(kernel_ctx_, batch, &out)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:641 ExecuteBatch(batch, listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:547 executor->Execute(arguments, &listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:533 ExecuteScalarExpression(call->arguments[i], input, exec_context)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/project_node.cc:91 ExecuteScalarExpression(simplified_expr, target, plan()->exec_context())
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:484 iterator_.Next()
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)

These do work if you set Sys.setlocale("LC_TIME", "C"). If I don't find a fix for this, I may consider only supporting the "C" locale on Windows.

[wjones127]

CI failure is unrelated Flight error.

[wjones127]

cc @pitrou

[rok]

Will this always be the same DB that R and Python will be using?

[wjones127]

No that 2021e is a version, which won't necessarily align with the R and Python ones in the future. I don't think it matters that we update it, unless the format changes somehow. This is just for testing that it works, and we don't ship it.

But the R unit tests use the one provided by the tzdb package, so we are testing that as well.

[rok]

Currently US and EU have DST, but they plan to abolish it soon. At that point we will have Python and R DBs with fresh DSTless times and arrow c++ using DST for tests. In isolation that's ok, but we do have tests comparing pandas and pyarrow results and similar for lubridate. It's not a big problem for sure, but if there is like a tzdata-latest.tar.gz that would be great.

[jorisvandenbossche]

The times in the past won't change, so unless we test against a random "now", updates to the timezone database (such as for DST policy changes) shouldn't impact those tests?

[wjones127]

Yeah this should be fine as is. We will never be testing between timezone databases.

[rok]

We currently test arrow vs pandas tzdb in CI. We have a test for a timestamp in 2033 and if it is in DST and DST is abolished it will error if we'll be using a pre-abolishment db with arrow and post-abolishment db with pandas. This is super irrelevant and as it's a simple fix and I'm sorry for wasting your time :).

[wjones127]

Ah got it. We'll address in the follow up PR where we'll have PyArrow use tzdb. For now the python timestamp tests are skipped on Windows

arrow/python/pyarrow/tests/test_compute.py

Lines 1915 to 1917 in d327f69

	ifsys.platform=='win32':
	# TODO: We should test on windows once ARROW-13168 is resolved.
	pytest.skip('Timezone database is not available on Windows yet')

[pitrou]

Thanks a lot for this @wjones127 . Here are some comments.

[pitrou]

I'd rather if you copied the relevant snippet here, we shouldn't ideally rely on the contents of CI scripts (which may contain specific quirks that irrelevant to normal user setups) for the public docs.

[wjones127]

Since we are pulling a specific part of the script, we should easily be able to avoid quirks? I like the idea of having our build instructions tested in CI. And we reference the CI scripts regularly when providing build instructions.

At the very least, if we get to a point where the public instructions and CI instructions diverge, we can easily separate them.

[pitrou]

Is this on Windows or specifically MinGW? i.e., would the non-MinGW Windows CI pass if you remove this skip? I'm wondering if we can narrow the condition.

[wjones127]

If failed on Windows 2019 C++17, which uses MSVC. But seems like it was actually passing on MinGW. So maybe I can try skipping for just MSVC?

[pitrou]

That's a bit weird, MinGW is just a different compiler but targetting the same runtime libraries...

[wjones127]

Actually, I think the test is likely skipping on the LocaleExists("fr_FR.UTF-8") condition; IIRC MinGW doesn't support locales apart from "C" and "POSIX". Sadly, no indication in CI whether the test was skipped: https://github.com/apache/arrow/runs/5504310182?check_suite_focus=true

[pitrou]

Is this a reliable location? Is there a risk that the downloads folder gets cleared from time to time?

[wjones127]

This is the behavior of the vendored datetime library, not something we chose. I think it's a fine default for testing and in production applications I expect users will manually specify a more appropriate path at runtime.

[jonkeane]

A few things I noticed in CI

[DavisVaughan]

FWIW, they just released 2022a a few days ago

[jonkeane]

Are there any outstanding comments that we need to resolve before merging?

The failures in CI both look unrelated. I'm happy to merge if no one objects

[ursabot]

Benchmark runs are scheduled for baseline = 919d113 and contender = f4dfd6c. f4dfd6c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.36% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.02% ⬆️0.81%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[wjones127]

Follow-up Jira created: https://issues.apache.org/jira/browse/ARROW-16054

[wjones127]

This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.

For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of strftime() and friends.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13168

[wjones127]

So timezone database seems to work, but methods that rely on std::local currently error with:

Error (test-dplyr-funcs-datetime.R:344:3): extract month from timestamp
Error: Invalid: Cannot find locale 'English_United States.1252': locale::facet::_S_create_c_locale name not valid
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1085 GetLocale(options.locale)
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1105 Make(ctx, *in.type)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:700 kernel_->exec(kernel_ctx_, batch, &out)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:641 ExecuteBatch(batch, listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:547 executor->Execute(arguments, &listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:533 ExecuteScalarExpression(call->arguments[i], input, exec_context)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/project_node.cc:91 ExecuteScalarExpression(simplified_expr, target, plan()->exec_context())
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:484 iterator_.Next()
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)

These do work if you set Sys.setlocale("LC_TIME", "C"). If I don't find a fix for this, I may consider only supporting the "C" locale on Windows.

[wjones127]

CI failure is unrelated Flight error.

[wjones127]

cc @pitrou

[rok]

Will this always be the same DB that R and Python will be using?

[wjones127]

No that 2021e is a version, which won't necessarily align with the R and Python ones in the future. I don't think it matters that we update it, unless the format changes somehow. This is just for testing that it works, and we don't ship it.

But the R unit tests use the one provided by the tzdb package, so we are testing that as well.

[rok]

Currently US and EU have DST, but they plan to abolish it soon. At that point we will have Python and R DBs with fresh DSTless times and arrow c++ using DST for tests. In isolation that's ok, but we do have tests comparing pandas and pyarrow results and similar for lubridate. It's not a big problem for sure, but if there is like a tzdata-latest.tar.gz that would be great.

[jorisvandenbossche]

The times in the past won't change, so unless we test against a random "now", updates to the timezone database (such as for DST policy changes) shouldn't impact those tests?

[wjones127]

Yeah this should be fine as is. We will never be testing between timezone databases.

[rok]

We currently test arrow vs pandas tzdb in CI. We have a test for a timestamp in 2033 and if it is in DST and DST is abolished it will error if we'll be using a pre-abolishment db with arrow and post-abolishment db with pandas. This is super irrelevant and as it's a simple fix and I'm sorry for wasting your time :).

[wjones127]

Ah got it. We'll address in the follow up PR where we'll have PyArrow use tzdb. For now the python timestamp tests are skipped on Windows

arrow/python/pyarrow/tests/test_compute.py

Lines 1915 to 1917 in d327f69

	ifsys.platform=='win32':
	# TODO: We should test on windows once ARROW-13168 is resolved.
	pytest.skip('Timezone database is not available on Windows yet')

[pitrou]

Thanks a lot for this @wjones127 . Here are some comments.

[pitrou]

I'd rather if you copied the relevant snippet here, we shouldn't ideally rely on the contents of CI scripts (which may contain specific quirks that irrelevant to normal user setups) for the public docs.

[wjones127]

Since we are pulling a specific part of the script, we should easily be able to avoid quirks? I like the idea of having our build instructions tested in CI. And we reference the CI scripts regularly when providing build instructions.

At the very least, if we get to a point where the public instructions and CI instructions diverge, we can easily separate them.

[pitrou]

Is this on Windows or specifically MinGW? i.e., would the non-MinGW Windows CI pass if you remove this skip? I'm wondering if we can narrow the condition.

[wjones127]

If failed on Windows 2019 C++17, which uses MSVC. But seems like it was actually passing on MinGW. So maybe I can try skipping for just MSVC?

[pitrou]

That's a bit weird, MinGW is just a different compiler but targetting the same runtime libraries...

[wjones127]

Actually, I think the test is likely skipping on the LocaleExists("fr_FR.UTF-8") condition; IIRC MinGW doesn't support locales apart from "C" and "POSIX". Sadly, no indication in CI whether the test was skipped: https://github.com/apache/arrow/runs/5504310182?check_suite_focus=true

[pitrou]

Is this a reliable location? Is there a risk that the downloads folder gets cleared from time to time?

[wjones127]

This is the behavior of the vendored datetime library, not something we chose. I think it's a fine default for testing and in production applications I expect users will manually specify a more appropriate path at runtime.

[jonkeane]

A few things I noticed in CI

[DavisVaughan]

FWIW, they just released 2022a a few days ago

[jonkeane]

Are there any outstanding comments that we need to resolve before merging?

The failures in CI both look unrelated. I'm happy to merge if no one objects

[ursabot]

Benchmark runs are scheduled for baseline = 919d113 and contender = f4dfd6c. f4dfd6c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.36% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.02% ⬆️0.81%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[wjones127]

Follow-up Jira created: https://issues.apache.org/jira/browse/ARROW-16054

[wjones127]

This allows for runtime configuration of the timezone database on Windows for C++ and R. Python will be handled later because it's available timezone libraries use the binary rather than text format, which is not yet supported the vendored date library.

For R, Windows will only support the "C" locale, since (as far as I can tell) that's the only locale supported by the MingW std::locale implementation. I think R itself gets around this by implementing a completely custom version of strftime() and friends.

[github-actions]

https://issues.apache.org/jira/browse/ARROW-13168

[wjones127]

So timezone database seems to work, but methods that rely on std::local currently error with:

Error (test-dplyr-funcs-datetime.R:344:3): extract month from timestamp
Error: Invalid: Cannot find locale 'English_United States.1252': locale::facet::_S_create_c_locale name not valid
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1085 GetLocale(options.locale)
C:/Users/voltron/arrow/cpp/src/arrow/compute/kernels/scalar_temporal_unary.cc:1105 Make(ctx, *in.type)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:700 kernel_->exec(kernel_ctx_, batch, &out)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec.cc:641 ExecuteBatch(batch, listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:547 executor->Execute(arguments, &listener)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/expression.cc:533 ExecuteScalarExpression(call->arguments[i], input, exec_context)
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/project_node.cc:91 ExecuteScalarExpression(simplified_expr, target, plan()->exec_context())
C:/Users/voltron/arrow/cpp/src/arrow/compute/exec/exec_plan.cc:484 iterator_.Next()
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:336 ReadNext(&batch)
C:/Users/voltron/arrow/cpp/src/arrow/record_batch.cc:347 ReadAll(&batches)

These do work if you set Sys.setlocale("LC_TIME", "C"). If I don't find a fix for this, I may consider only supporting the "C" locale on Windows.

[wjones127]

CI failure is unrelated Flight error.

[wjones127]

cc @pitrou

[rok]

Will this always be the same DB that R and Python will be using?

[wjones127]

No that 2021e is a version, which won't necessarily align with the R and Python ones in the future. I don't think it matters that we update it, unless the format changes somehow. This is just for testing that it works, and we don't ship it.

But the R unit tests use the one provided by the tzdb package, so we are testing that as well.

[rok]

Currently US and EU have DST, but they plan to abolish it soon. At that point we will have Python and R DBs with fresh DSTless times and arrow c++ using DST for tests. In isolation that's ok, but we do have tests comparing pandas and pyarrow results and similar for lubridate. It's not a big problem for sure, but if there is like a tzdata-latest.tar.gz that would be great.

[jorisvandenbossche]

The times in the past won't change, so unless we test against a random "now", updates to the timezone database (such as for DST policy changes) shouldn't impact those tests?

[wjones127]

Yeah this should be fine as is. We will never be testing between timezone databases.

[rok]

We currently test arrow vs pandas tzdb in CI. We have a test for a timestamp in 2033 and if it is in DST and DST is abolished it will error if we'll be using a pre-abolishment db with arrow and post-abolishment db with pandas. This is super irrelevant and as it's a simple fix and I'm sorry for wasting your time :).

[wjones127]

Ah got it. We'll address in the follow up PR where we'll have PyArrow use tzdb. For now the python timestamp tests are skipped on Windows

arrow/python/pyarrow/tests/test_compute.py

Lines 1915 to 1917 in d327f69

	ifsys.platform=='win32':
	# TODO: We should test on windows once ARROW-13168 is resolved.
	pytest.skip('Timezone database is not available on Windows yet')

[pitrou]

Thanks a lot for this @wjones127 . Here are some comments.

[pitrou]

I'd rather if you copied the relevant snippet here, we shouldn't ideally rely on the contents of CI scripts (which may contain specific quirks that irrelevant to normal user setups) for the public docs.

[wjones127]

Since we are pulling a specific part of the script, we should easily be able to avoid quirks? I like the idea of having our build instructions tested in CI. And we reference the CI scripts regularly when providing build instructions.

At the very least, if we get to a point where the public instructions and CI instructions diverge, we can easily separate them.

[pitrou]

Is this on Windows or specifically MinGW? i.e., would the non-MinGW Windows CI pass if you remove this skip? I'm wondering if we can narrow the condition.

[wjones127]

If failed on Windows 2019 C++17, which uses MSVC. But seems like it was actually passing on MinGW. So maybe I can try skipping for just MSVC?

[pitrou]

That's a bit weird, MinGW is just a different compiler but targetting the same runtime libraries...

[wjones127]

Actually, I think the test is likely skipping on the LocaleExists("fr_FR.UTF-8") condition; IIRC MinGW doesn't support locales apart from "C" and "POSIX". Sadly, no indication in CI whether the test was skipped: https://github.com/apache/arrow/runs/5504310182?check_suite_focus=true

[pitrou]

Is this a reliable location? Is there a risk that the downloads folder gets cleared from time to time?

[wjones127]

This is the behavior of the vendored datetime library, not something we chose. I think it's a fine default for testing and in production applications I expect users will manually specify a more appropriate path at runtime.

[jonkeane]

A few things I noticed in CI

[DavisVaughan]

FWIW, they just released 2022a a few days ago

[jonkeane]

Are there any outstanding comments that we need to resolve before merging?

The failures in CI both look unrelated. I'm happy to merge if no one objects

[ursabot]

Benchmark runs are scheduled for baseline = 919d113 and contender = f4dfd6c. f4dfd6c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.34% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.36% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.02% ⬆️0.81%] ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[wjones127]

Follow-up Jira created: https://issues.apache.org/jira/browse/ARROW-16054