Skip to content

ARROW-10797: [C++] Vendor and use PCG random generator library #8879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pitrou wants to merge 2 commits into from
Closed

ARROW-10797: [C++] Vendor and use PCG random generator library #8879

pitrou wants to merge 2 commits into from

Conversation

@pitrou
Copy link
Member

@pitrou pitrou commented Dec 9, 2020

No description provided.

@github-actions
Copy link

github-actions bot commented Dec 9, 2020

https://issues.apache.org/jira/browse/ARROW-10797

@github-actions github-actions bot added the needs-rebase A PR that needs to be rebased by the author label Dec 14, 2020
@pitrou pitrou force-pushed the ARROW-10797-pcg branch 2 times, most recently from 2557213 to f7b1147 Compare April 21, 2021 10:14
@pitrou pitrou marked this pull request as ready for review April 21, 2021 12:57
@pitrou pitrou requested a review from cyb70289 April 21, 2021 12:57
@pitrou
Copy link
Member Author

pitrou commented Apr 21, 2021

This exposes a small precision issue in the variance kernel tests in MinGW32 on 32-bit floats (because of rounding issues perhaps?):
https://github.com/pitrou/arrow/runs/2400347814?check_suite_focus=true#step:10:831

According to Numpy, the two float64 values differ by 5 ULPs:

>>> a
1011966660.249961
>>> b
1011966660.2499604
>>> np.testing.assert_array_max_ulp(a, b)
Traceback (most recent call last):
  File "<ipython-input-15-48fada135e2b>", line 1, in <module>
    np.testing.assert_array_max_ulp(a, b)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/numpy/testing/_private/utils.py", line 1643, in assert_array_max_ulp
    (maxulp, np.max(ret)))
AssertionError: Arrays are not almost equal up to 1 ULP (max difference is 5 ULP)

@cyb70289
Copy link
Contributor

cyb70289 commented Apr 21, 2021

Should be the MinGW32 rounding issue we met with many times before.
Guess PCG generator produces different random numbers than std::default_random_engine. So this issue is always there, just not revealed by previous test data.
I think we can disable the random test for MinGW32. ASSERT_DOUBLE_EQ expects at most 4ULP diff, it's almost there.

@pitrou
Copy link
Member Author

pitrou commented Apr 21, 2021

Ok, I added a skip on MinGW32.

cyb70289
cyb70289 approved these changes Apr 22, 2021
View reviewed changes
Copy link
Contributor

@cyb70289 cyb70289 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. CI failures looks not related.

@cyb70289
Copy link
Contributor

cyb70289 commented Apr 22, 2021

@github-actions crossbow submit -g nightly

@cyb70289 cyb70289 removed the needs-rebase A PR that needs to be rebased by the author label Apr 22, 2021
@github-actions
Copy link

github-actions bot commented Apr 22, 2021

Revision: c62dc304163a68844e76217da6f5ca4555da34bb

Submitted crossbow builds: ursacomputing/crossbow @ actions-341

Task Status
centos-7-amd64 Github Actions
centos-8-amd64 Github Actions
centos-8-arm64 TravisCI
conda-clean Azure
conda-linux-gcc-py36-arm64 Drone
conda-linux-gcc-py36-cpu-r36 Azure
conda-linux-gcc-py36-cuda Azure
conda-linux-gcc-py37-arm64 Drone
conda-linux-gcc-py37-cpu-r40 Azure
conda-linux-gcc-py37-cuda Azure
conda-linux-gcc-py38-arm64 Drone
conda-linux-gcc-py38-cpu Azure
conda-linux-gcc-py38-cuda Azure
conda-linux-gcc-py39-arm64 Drone
conda-linux-gcc-py39-cpu Azure
conda-linux-gcc-py39-cuda Azure
conda-osx-arm64-clang-py38 Azure
conda-osx-arm64-clang-py39 Azure
conda-osx-clang-py36-r36 Azure
conda-osx-clang-py37-r40 Azure
conda-osx-clang-py38 Azure
conda-osx-clang-py39 Azure
conda-win-vs2017-py36-r36 Azure
conda-win-vs2017-py37-r40 Azure
conda-win-vs2017-py38 Azure
conda-win-vs2017-py39 Azure
debian-bullseye-amd64 Github Actions
debian-bullseye-arm64 TravisCI
debian-buster-amd64 Github Actions
debian-buster-arm64 TravisCI
example-cpp-minimal-build-static Github Actions
example-cpp-minimal-build-static-system-dependency Github Actions
gandiva-jar-osx Github Actions
gandiva-jar-ubuntu Github Actions
homebrew-cpp Github Actions
homebrew-r-autobrew Github Actions
nuget Github Actions
python-sdist Github Actions
test-build-vcpkg-win Github Actions
test-conda-cpp Github Actions
test-conda-cpp-valgrind Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-dask-latest Github Actions
test-conda-python-3.7-hdfs-3.2 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-pandas-master Github Actions
test-conda-python-3.7-spark-branch-3.0 Github Actions
test-conda-python-3.7-turbodbc-latest Github Actions
test-conda-python-3.7-turbodbc-master Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-dask-master Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-jpype Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-master Github Actions
test-conda-python-3.9 Github Actions
test-debian-10-cpp Github Actions
test-debian-10-go-1.15 Azure
test-debian-10-python-3 Azure
test-debian-c-glib Github Actions
test-debian-ruby Github Actions
test-fedora-33-cpp Github Actions
test-fedora-33-python-3 Azure
test-r-devdocs Github Actions
test-r-install-local Github Actions
test-r-linux-as-cran Github Actions
test-r-minimal-build Azure
test-r-rhub-ubuntu-gcc-release Azure
test-r-rocker-r-base-latest Azure
test-r-rstudio-r-base-3.6-bionic Azure
test-r-rstudio-r-base-3.6-centos7-devtoolset-8 Azure
test-r-rstudio-r-base-3.6-centos8 Azure
test-r-rstudio-r-base-3.6-opensuse15 Azure
test-r-rstudio-r-base-3.6-opensuse42 Azure
test-r-version-compatibility Github Actions
test-r-versions Github Actions
test-r-without-arrow Azure
test-ubuntu-18.04-cpp Github Actions
test-ubuntu-18.04-cpp-release Github Actions
test-ubuntu-18.04-cpp-static Github Actions
test-ubuntu-18.04-python-3 Azure
test-ubuntu-18.04-r-sanitizer Azure
test-ubuntu-20.04-cpp Github Actions
test-ubuntu-20.04-cpp-14 Github Actions
test-ubuntu-20.04-cpp-17 Github Actions
test-ubuntu-20.04-cpp-thread-sanitizer Github Actions
test-ubuntu-20.10-docs Azure
test-ubuntu-c-glib Github Actions
test-ubuntu-ruby Github Actions
ubuntu-bionic-amd64 Github Actions
ubuntu-bionic-arm64 TravisCI
ubuntu-focal-amd64 Github Actions
ubuntu-focal-arm64 TravisCI
ubuntu-groovy-amd64 Github Actions
ubuntu-groovy-arm64 TravisCI
wheel-manylinux2010-cp36-amd64 Github Actions
wheel-manylinux2010-cp37-amd64 Github Actions
wheel-manylinux2010-cp38-amd64 Github Actions
wheel-manylinux2010-cp39-amd64 Github Actions
wheel-manylinux2014-cp36-amd64 Github Actions
wheel-manylinux2014-cp36-arm64 TravisCI
wheel-manylinux2014-cp37-amd64 Github Actions
wheel-manylinux2014-cp37-arm64 TravisCI
wheel-manylinux2014-cp38-amd64 Github Actions
wheel-manylinux2014-cp38-arm64 TravisCI
wheel-manylinux2014-cp39-amd64 Github Actions
wheel-manylinux2014-cp39-arm64 TravisCI
wheel-osx-high-sierra-cp36 Github Actions
wheel-osx-high-sierra-cp37 Github Actions
wheel-osx-high-sierra-cp38 Github Actions
wheel-osx-high-sierra-cp39 Github Actions
wheel-osx-mavericks-cp36 Github Actions
wheel-osx-mavericks-cp37 Github Actions
wheel-osx-mavericks-cp38 Github Actions
wheel-osx-mavericks-cp39 Github Actions
wheel-windows-cp36 Github Actions
wheel-windows-cp37 Github Actions
wheel-windows-cp38 Github Actions
wheel-windows-cp39 Github Actions

@cyb70289
Copy link
Contributor

cyb70289 commented Apr 22, 2021

CentOS error may be caused by gcc-4.8 build failure from earlier pr.

@pitrou pitrou closed this in 29a9a3e Apr 22, 2021
@pitrou pitrou deleted the ARROW-10797-pcg branch April 22, 2021 11:09
@asfimport asfimport mentioned this pull request Jul 21, 2021
Closed
@kou kou mentioned this pull request Oct 11, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyb70289 cyb70289 cyb70289 approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@pitrou @cyb70289