ARROW-10797: [C++] Vendor and use PCG random generator library - #8879

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library#8879
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg

Conversation

@pitrou

@pitroupitrou commented Dec 9, 2020

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actionsBot commented Dec 9, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10797

@github-actionsgithub-actionsBot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou
pitrouforce-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147CompareApril 21, 2021 10:14
@pitrou
pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou
pitrou requested a review from cyb70289April 21, 2021 12:57
@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>>a1011966660.249961>>>b1011966660.2499604>>>np.testing.assert_array_max_ulp(a, b)
Traceback (mostrecentcalllast):
File"<ipython-input-15-48fada135e2b>", line1, in<module>np.testing.assert_array_max_ulp(a, b)
File"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line1643, inassert_array_max_ulp
(maxulp, np.max(ret)))
AssertionError: Arraysarenotalmostequalupto1ULP (maxdifferenceis5ULP)

@cyb70289

cyb70289 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes

@cyb70289cyb70289 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly

@cyb70289cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions

github-actionsBot commented Apr 22, 2021

Copy link
Copy Markdown

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Drone
conda-linux-gcc-py36-cpu-r36Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Drone
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Drone
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Drone
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r36Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r36Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
gandiva-jar-osxGithub Actions
gandiva-jar-ubuntuGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-hdfs-3.2Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-pandas-masterGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-dask-masterGithub Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-jpypeGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-releaseAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-osx-high-sierra-cp36Github Actions
wheel-osx-high-sierra-cp37Github Actions
wheel-osx-high-sierra-cp38Github Actions
wheel-osx-high-sierra-cp39Github Actions
wheel-osx-mavericks-cp36Github Actions
wheel-osx-mavericks-cp37Github Actions
wheel-osx-mavericks-cp38Github Actions
wheel-osx-mavericks-cp39Github Actions
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou
pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimportasfimport mentioned this pull request Jul 21, 2021
Closed
@koukou mentioned this pull request Oct 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@cyb70289
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all \u003cpre\u003e\u003ccode\u003e blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks"); } } catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); } })(); (function(){ try { var __m = "github.com"; var __re = new RegExp('^' + "github\\.com" + '
Skip to content

ARROW-10797: [C++] Vendor and use PCG random generator library - #8879

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library#8879
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg

Conversation

@pitrou

@pitroupitrou commented Dec 9, 2020

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actionsBot commented Dec 9, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10797

@github-actionsgithub-actionsBot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou
pitrouforce-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147CompareApril 21, 2021 10:14
@pitrou
pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou
pitrou requested a review from cyb70289April 21, 2021 12:57
@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>>a1011966660.249961>>>b1011966660.2499604>>>np.testing.assert_array_max_ulp(a, b)
Traceback (mostrecentcalllast):
File"<ipython-input-15-48fada135e2b>", line1, in<module>np.testing.assert_array_max_ulp(a, b)
File"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line1643, inassert_array_max_ulp
(maxulp, np.max(ret)))
AssertionError: Arraysarenotalmostequalupto1ULP (maxdifferenceis5ULP)

@cyb70289

cyb70289 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes

@cyb70289cyb70289 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly

@cyb70289cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions

github-actionsBot commented Apr 22, 2021

Copy link
Copy Markdown

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Drone
conda-linux-gcc-py36-cpu-r36Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Drone
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Drone
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Drone
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r36Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r36Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
gandiva-jar-osxGithub Actions
gandiva-jar-ubuntuGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-hdfs-3.2Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-pandas-masterGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-dask-masterGithub Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-jpypeGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-releaseAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-osx-high-sierra-cp36Github Actions
wheel-osx-high-sierra-cp37Github Actions
wheel-osx-high-sierra-cp38Github Actions
wheel-osx-high-sierra-cp39Github Actions
wheel-osx-mavericks-cp36Github Actions
wheel-osx-mavericks-cp37Github Actions
wheel-osx-mavericks-cp38Github Actions
wheel-osx-mavericks-cp39Github Actions
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou
pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimportasfimport mentioned this pull request Jul 21, 2021
Closed
@koukou mentioned this pull request Oct 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@cyb70289
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-10797: [C++] Vendor and use PCG random generator library - #8879

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library#8879
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg

Conversation

@pitrou

@pitroupitrou commented Dec 9, 2020

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actionsBot commented Dec 9, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10797

@github-actionsgithub-actionsBot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou
pitrouforce-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147CompareApril 21, 2021 10:14
@pitrou
pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou
pitrou requested a review from cyb70289April 21, 2021 12:57
@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>>a1011966660.249961>>>b1011966660.2499604>>>np.testing.assert_array_max_ulp(a, b)
Traceback (mostrecentcalllast):
File"<ipython-input-15-48fada135e2b>", line1, in<module>np.testing.assert_array_max_ulp(a, b)
File"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line1643, inassert_array_max_ulp
(maxulp, np.max(ret)))
AssertionError: Arraysarenotalmostequalupto1ULP (maxdifferenceis5ULP)

@cyb70289

cyb70289 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes

@cyb70289cyb70289 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly

@cyb70289cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions

github-actionsBot commented Apr 22, 2021

Copy link
Copy Markdown

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Drone
conda-linux-gcc-py36-cpu-r36Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Drone
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Drone
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Drone
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r36Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r36Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
gandiva-jar-osxGithub Actions
gandiva-jar-ubuntuGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-hdfs-3.2Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-pandas-masterGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-dask-masterGithub Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-jpypeGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-releaseAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-osx-high-sierra-cp36Github Actions
wheel-osx-high-sierra-cp37Github Actions
wheel-osx-high-sierra-cp38Github Actions
wheel-osx-high-sierra-cp39Github Actions
wheel-osx-mavericks-cp36Github Actions
wheel-osx-mavericks-cp37Github Actions
wheel-osx-mavericks-cp38Github Actions
wheel-osx-mavericks-cp39Github Actions
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou
pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimportasfimport mentioned this pull request Jul 21, 2021
Closed
@koukou mentioned this pull request Oct 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@cyb70289
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length \u003e 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-10797: [C++] Vendor and use PCG random generator library - #8879

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library#8879
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg

Conversation

@pitrou

@pitroupitrou commented Dec 9, 2020

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actionsBot commented Dec 9, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10797

@github-actionsgithub-actionsBot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou
pitrouforce-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147CompareApril 21, 2021 10:14
@pitrou
pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou
pitrou requested a review from cyb70289April 21, 2021 12:57
@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>>a1011966660.249961>>>b1011966660.2499604>>>np.testing.assert_array_max_ulp(a, b)
Traceback (mostrecentcalllast):
File"<ipython-input-15-48fada135e2b>", line1, in<module>np.testing.assert_array_max_ulp(a, b)
File"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line1643, inassert_array_max_ulp
(maxulp, np.max(ret)))
AssertionError: Arraysarenotalmostequalupto1ULP (maxdifferenceis5ULP)

@cyb70289

cyb70289 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes

@cyb70289cyb70289 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly

@cyb70289cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions

github-actionsBot commented Apr 22, 2021

Copy link
Copy Markdown

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Drone
conda-linux-gcc-py36-cpu-r36Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Drone
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Drone
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Drone
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r36Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r36Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
gandiva-jar-osxGithub Actions
gandiva-jar-ubuntuGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-hdfs-3.2Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-pandas-masterGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-dask-masterGithub Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-jpypeGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-releaseAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-osx-high-sierra-cp36Github Actions
wheel-osx-high-sierra-cp37Github Actions
wheel-osx-high-sierra-cp38Github Actions
wheel-osx-high-sierra-cp39Github Actions
wheel-osx-mavericks-cp36Github Actions
wheel-osx-mavericks-cp37Github Actions
wheel-osx-mavericks-cp38Github Actions
wheel-osx-mavericks-cp39Github Actions
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou
pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimportasfimport mentioned this pull request Jul 21, 2021
Closed
@koukou mentioned this pull request Oct 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@cyb70289
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

ARROW-10797: [C++] Vendor and use PCG random generator library - #8879

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library#8879
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg

Conversation

@pitrou

@pitroupitrou commented Dec 9, 2020

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actionsBot commented Dec 9, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10797

@github-actionsgithub-actionsBot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou
pitrouforce-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147CompareApril 21, 2021 10:14
@pitrou
pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou
pitrou requested a review from cyb70289April 21, 2021 12:57
@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>>a1011966660.249961>>>b1011966660.2499604>>>np.testing.assert_array_max_ulp(a, b)
Traceback (mostrecentcalllast):
File"<ipython-input-15-48fada135e2b>", line1, in<module>np.testing.assert_array_max_ulp(a, b)
File"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line1643, inassert_array_max_ulp
(maxulp, np.max(ret)))
AssertionError: Arraysarenotalmostequalupto1ULP (maxdifferenceis5ULP)

@cyb70289

cyb70289 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes

@cyb70289cyb70289 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly

@cyb70289cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions

github-actionsBot commented Apr 22, 2021

Copy link
Copy Markdown

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Drone
conda-linux-gcc-py36-cpu-r36Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Drone
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Drone
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Drone
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r36Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r36Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
gandiva-jar-osxGithub Actions
gandiva-jar-ubuntuGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-hdfs-3.2Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-pandas-masterGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-dask-masterGithub Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-jpypeGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-releaseAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-osx-high-sierra-cp36Github Actions
wheel-osx-high-sierra-cp37Github Actions
wheel-osx-high-sierra-cp38Github Actions
wheel-osx-high-sierra-cp39Github Actions
wheel-osx-mavericks-cp36Github Actions
wheel-osx-mavericks-cp37Github Actions
wheel-osx-mavericks-cp38Github Actions
wheel-osx-mavericks-cp39Github Actions
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou
pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimportasfimport mentioned this pull request Jul 21, 2021
Closed
@koukou mentioned this pull request Oct 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@cyb70289
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-10797: [C++] Vendor and use PCG random generator library - #8879

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library#8879
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg

Conversation

@pitrou

@pitroupitrou commented Dec 9, 2020

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actionsBot commented Dec 9, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10797

@github-actionsgithub-actionsBot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou
pitrouforce-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147CompareApril 21, 2021 10:14
@pitrou
pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou
pitrou requested a review from cyb70289April 21, 2021 12:57
@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>>a1011966660.249961>>>b1011966660.2499604>>>np.testing.assert_array_max_ulp(a, b)
Traceback (mostrecentcalllast):
File"<ipython-input-15-48fada135e2b>", line1, in<module>np.testing.assert_array_max_ulp(a, b)
File"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line1643, inassert_array_max_ulp
(maxulp, np.max(ret)))
AssertionError: Arraysarenotalmostequalupto1ULP (maxdifferenceis5ULP)

@cyb70289

cyb70289 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes

@cyb70289cyb70289 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly

@cyb70289cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions

github-actionsBot commented Apr 22, 2021

Copy link
Copy Markdown

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Drone
conda-linux-gcc-py36-cpu-r36Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Drone
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Drone
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Drone
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r36Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r36Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
gandiva-jar-osxGithub Actions
gandiva-jar-ubuntuGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-hdfs-3.2Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-pandas-masterGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-dask-masterGithub Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-jpypeGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-releaseAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-osx-high-sierra-cp36Github Actions
wheel-osx-high-sierra-cp37Github Actions
wheel-osx-high-sierra-cp38Github Actions
wheel-osx-high-sierra-cp39Github Actions
wheel-osx-mavericks-cp36Github Actions
wheel-osx-mavericks-cp37Github Actions
wheel-osx-mavericks-cp38Github Actions
wheel-osx-mavericks-cp39Github Actions
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou
pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimportasfimport mentioned this pull request Jul 21, 2021
Closed
@koukou mentioned this pull request Oct 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@cyb70289
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

ARROW-10797: [C++] Vendor and use PCG random generator library - #8879

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library#8879
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg

Conversation

@pitrou

@pitroupitrou commented Dec 9, 2020

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actionsBot commented Dec 9, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10797

@github-actionsgithub-actionsBot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou
pitrouforce-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147CompareApril 21, 2021 10:14
@pitrou
pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou
pitrou requested a review from cyb70289April 21, 2021 12:57
@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>>a1011966660.249961>>>b1011966660.2499604>>>np.testing.assert_array_max_ulp(a, b)
Traceback (mostrecentcalllast):
File"<ipython-input-15-48fada135e2b>", line1, in<module>np.testing.assert_array_max_ulp(a, b)
File"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line1643, inassert_array_max_ulp
(maxulp, np.max(ret)))
AssertionError: Arraysarenotalmostequalupto1ULP (maxdifferenceis5ULP)

@cyb70289

cyb70289 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes

@cyb70289cyb70289 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly

@cyb70289cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions

github-actionsBot commented Apr 22, 2021

Copy link
Copy Markdown

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Drone
conda-linux-gcc-py36-cpu-r36Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Drone
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Drone
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Drone
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r36Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r36Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
gandiva-jar-osxGithub Actions
gandiva-jar-ubuntuGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-hdfs-3.2Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-pandas-masterGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-dask-masterGithub Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-jpypeGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-releaseAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-osx-high-sierra-cp36Github Actions
wheel-osx-high-sierra-cp37Github Actions
wheel-osx-high-sierra-cp38Github Actions
wheel-osx-high-sierra-cp39Github Actions
wheel-osx-mavericks-cp36Github Actions
wheel-osx-mavericks-cp37Github Actions
wheel-osx-mavericks-cp38Github Actions
wheel-osx-mavericks-cp39Github Actions
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou
pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimportasfimport mentioned this pull request Jul 21, 2021
Closed
@koukou mentioned this pull request Oct 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@cyb70289
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

ARROW-10797: [C++] Vendor and use PCG random generator library - #8879

Closed
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library#8879
pitrou wants to merge 2 commits into
apache:masterfrom
pitrou:ARROW-10797-pcg

Conversation

@pitrou

@pitroupitrou commented Dec 9, 2020

Copy link
Copy Markdown
Member

No description provided.

@github-actions

github-actionsBot commented Dec 9, 2020

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/ARROW-10797

@github-actionsgithub-actionsBot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou
pitrouforce-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147CompareApril 21, 2021 10:14
@pitrou
pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou
pitrou requested a review from cyb70289April 21, 2021 12:57
@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>>a1011966660.249961>>>b1011966660.2499604>>>np.testing.assert_array_max_ulp(a, b)
Traceback (mostrecentcalllast):
File"<ipython-input-15-48fada135e2b>", line1, in<module>np.testing.assert_array_max_ulp(a, b)
File"/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line1643, inassert_array_max_ulp
(maxulp, np.max(ret)))
AssertionError: Arraysarenotalmostequalupto1ULP (maxdifferenceis5ULP)

@cyb70289

cyb70289 commented Apr 21, 2021

Copy link
Copy Markdown
Contributor

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou

pitrou commented Apr 21, 2021

Copy link
Copy Markdown
MemberAuthor

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes

@cyb70289cyb70289 left a comment •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

@github-actions crossbow submit -g nightly

@cyb70289cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions

github-actionsBot commented Apr 22, 2021

Copy link
Copy Markdown

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

TaskStatus
centos-7-amd64Github Actions
centos-8-amd64Github Actions
centos-8-arm64TravisCI
conda-cleanAzure
conda-linux-gcc-py36-arm64Drone
conda-linux-gcc-py36-cpu-r36Azure
conda-linux-gcc-py36-cudaAzure
conda-linux-gcc-py37-arm64Drone
conda-linux-gcc-py37-cpu-r40Azure
conda-linux-gcc-py37-cudaAzure
conda-linux-gcc-py38-arm64Drone
conda-linux-gcc-py38-cpuAzure
conda-linux-gcc-py38-cudaAzure
conda-linux-gcc-py39-arm64Drone
conda-linux-gcc-py39-cpuAzure
conda-linux-gcc-py39-cudaAzure
conda-osx-arm64-clang-py38Azure
conda-osx-arm64-clang-py39Azure
conda-osx-clang-py36-r36Azure
conda-osx-clang-py37-r40Azure
conda-osx-clang-py38Azure
conda-osx-clang-py39Azure
conda-win-vs2017-py36-r36Azure
conda-win-vs2017-py37-r40Azure
conda-win-vs2017-py38Azure
conda-win-vs2017-py39Azure
debian-bullseye-amd64Github Actions
debian-bullseye-arm64TravisCI
debian-buster-amd64Github Actions
debian-buster-arm64TravisCI
example-cpp-minimal-build-staticGithub Actions
example-cpp-minimal-build-static-system-dependencyGithub Actions
gandiva-jar-osxGithub Actions
gandiva-jar-ubuntuGithub Actions
homebrew-cppGithub Actions
homebrew-r-autobrewGithub Actions
nugetGithub Actions
python-sdistGithub Actions
test-build-vcpkg-winGithub Actions
test-conda-cppGithub Actions
test-conda-cpp-valgrindGithub Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-dask-latestGithub Actions
test-conda-python-3.7-hdfs-3.2Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-pandas-masterGithub Actions
test-conda-python-3.7-spark-branch-3.0Github Actions
test-conda-python-3.7-turbodbc-latestGithub Actions
test-conda-python-3.7-turbodbc-masterGithub Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-dask-masterGithub Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-jpypeGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-masterGithub Actions
test-conda-python-3.9Github Actions
test-debian-10-cppGithub Actions
test-debian-10-go-1.15Azure
test-debian-10-python-3Azure
test-debian-c-glibGithub Actions
test-debian-rubyGithub Actions
test-fedora-33-cppGithub Actions
test-fedora-33-python-3Azure
test-r-devdocsGithub Actions
test-r-install-localGithub Actions
test-r-linux-as-cranGithub Actions
test-r-minimal-buildAzure
test-r-rhub-ubuntu-gcc-releaseAzure
test-r-rocker-r-base-latestAzure
test-r-rstudio-r-base-3.6-bionicAzure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8Azure
test-r-rstudio-r-base-3.6-centos8Azure
test-r-rstudio-r-base-3.6-opensuse15Azure
test-r-rstudio-r-base-3.6-opensuse42Azure
test-r-version-compatibilityGithub Actions
test-r-versionsGithub Actions
test-r-without-arrowAzure
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-18.04-python-3Azure
test-ubuntu-18.04-r-sanitizerAzure
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions
test-ubuntu-20.10-docsAzure
test-ubuntu-c-glibGithub Actions
test-ubuntu-rubyGithub Actions
ubuntu-bionic-amd64Github Actions
ubuntu-bionic-arm64TravisCI
ubuntu-focal-amd64Github Actions
ubuntu-focal-arm64TravisCI
ubuntu-groovy-amd64Github Actions
ubuntu-groovy-arm64TravisCI
wheel-manylinux2010-cp36-amd64Github Actions
wheel-manylinux2010-cp37-amd64Github Actions
wheel-manylinux2010-cp38-amd64Github Actions
wheel-manylinux2010-cp39-amd64Github Actions
wheel-manylinux2014-cp36-amd64Github Actions
wheel-manylinux2014-cp36-arm64TravisCI
wheel-manylinux2014-cp37-amd64Github Actions
wheel-manylinux2014-cp37-arm64TravisCI
wheel-manylinux2014-cp38-amd64Github Actions
wheel-manylinux2014-cp38-arm64TravisCI
wheel-manylinux2014-cp39-amd64Github Actions
wheel-manylinux2014-cp39-arm64TravisCI
wheel-osx-high-sierra-cp36Github Actions
wheel-osx-high-sierra-cp37Github Actions
wheel-osx-high-sierra-cp38Github Actions
wheel-osx-high-sierra-cp39Github Actions
wheel-osx-mavericks-cp36Github Actions
wheel-osx-mavericks-cp37Github Actions
wheel-osx-mavericks-cp38Github Actions
wheel-osx-mavericks-cp39Github Actions
wheel-windows-cp36Github Actions
wheel-windows-cp37Github Actions
wheel-windows-cp38Github Actions
wheel-windows-cp39Github Actions

@cyb70289

cyb70289 commented Apr 22, 2021

Copy link
Copy Markdown
Contributor

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou
pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimportasfimport mentioned this pull request Jul 21, 2021
Closed
@koukou mentioned this pull request Oct 11, 2025
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289cyb70289cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou@cyb70289