[C++] Checksum failed for boost 1.75.0

[zhztheplayer]

Describe the bug, including details regarding any error messages, version, and platform.

-- verifying file...
 file='/opt/gluten/ep/build-arrow/build/arrow_ep/cpp/build/boost_ep-prefix/src/boost_1_75_0.tar.gz'
-- SHA256 hash of
 /opt/gluten/ep/build-arrow/build/arrow_ep/cpp/build/boost_ep-prefix/src/boost_1_75_0.tar.gz
 does not match expected value
 expected: '267e04a7c0bfe85daf796dedc789c3a27a76707e1c968f0a2a87bb96331e2b61'
 actual: 'aeb26f80e80945e82ee93e5939baebdca47b9dee80a07d3144be1e1a6a66dd6a'
-- Hash mismatch, removing...

I bumped into this this morning when I was building Arrow 11.0.0 's source code.

It's weird that the build could succeed before. I manually downloaded the library then found that it matches the "actual" sha256sum:

[root@localhost ~]# wget https://boostorg.jfrog.io/artifactory/main/release/1.75.0/source/boost_1_75_0.tar.gz
--2023-03-22 11:47:48-- https://boostorg.jfrog.io/artifactory/main/release/1.75.0/source/boost_1_75_0.tar.gz
Resolving boostorg.jfrog.io (boostorg.jfrog.io)... 34.211.121.239, 54.189.250.86, 52.40.5.152, ...
Connecting to boostorg.jfrog.io (boostorg.jfrog.io)|34.211.121.239|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-boostorg/filestore/68/68be4a43b73c66370c8d3fd94723b3913217ce1b?X-Artifactory-username=anonymous&X-Artifactory-repositoryKey=main&X-Artifactory-artifactPath=release%2F1.75.0%2Fsource%2Fboost_1_75_0.tar.gz&X-Artifactory-projectKey=default&x-jf-traceId=78bc8128fa9857a2&response-content-disposition=attachment%3Bfilename%3D%22boost_1_75_0.tar.gz%22&response-content-type=application%2Fx-gzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjENT%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJGMEQCIDNsJ6mm3ED2qEBKXXogmKqSUgCqmDiHytrUp5fRakRrAiBBzsixGsd2akUqjqrrWzgMe8HxdEP1LIAAoPsz8AXLTSqPBQid%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAAaDDE1MjE1MzA2MjE0MSIMtL8rAWkfo90CvBSaKuMEzSfcNizP9ia5lyp%2BY%2FpmQyRtUe8i%2BPMNHxt1qeXLxlPDNDAdqNH64C5PGZ%2BoDYu1Y2ofhzbgf%2BjMA3pq1MalfvnJ33rVOrngAhwdSKROzg6qWIz3KtMc9vPpGyT1MjBVAZ9nnn9%2F2BpI99rNPobAX0jci6la4k9GDYrl%2FadSvR1VsW977fwlIGvfzm%2FjQih2RrSArGIHLMXmMSO4n61FxVvO6IoYJsks5cRf53dpJR0PbO2hN9tWidlTapcV00P0F2mMBnZK1dTFwicUjw4lHOf2mFVlFaV2%2FhbWgQV6ap0Xe3tCnCyhZR0GJWGDwOweGyf2oN8eX9Wg5kEDuFpw3TcnaqJ11Y%2B9Rrel%2F1NA8oRTGVGgQL5Ztcn5o7gexSi4LQWnWXaYRUFlUrmm7ovM04afpPiHscmjL1lqqn7p3ncFFRmluT91pX5nLE2cRjeZvyFcwiaeywrNfFM0azeg52aZNfdj2UfHV3PghQcxmi9oEO39FYHPFmVLVqDTurKrg3nWa%2FLAAuSQmW3AeO9zBjQvj7f21Sl%2BIGEgkgRYIKKla4iXVYPFSb4RtF9g%2BfdvBk1%2FhK%2BG6V%2BNW0j4vxeYrtyokIX4TJFBI6VdoM19AsHFPi6D4HJRhVzCOQAjFq1TgleBkt49J%2FPCT1a1BT8IBE8k%2BYownnvrtqOMZsi9qYWinypQBykqS0BztRmi6cUvNcUiEtyrUauSoVTYJJj4nj38nCh%2F7%2FO69x5SSCd3HOVaetg4a%2Br9w8Zwb9K8HRPTzTEHFjJ0GIea9GeqYViXY1MUSBgpUDj7PJI00AFq22%2FK8lQw9%2B7poAY6mwFSw6jCZhWYDjTNq%2F8XDXNmwdqaOSwjg%2FOasApDbXfv3cHi32kMFQYWGkd%2FmsupakPqJd1Y47uzn6c5YiZ8yb6s2IMCtjlxICvg4DlafPyr7K3dDS7S5LU6Im2VZZp0siN5I4RTx4SezK9xMffluU9XsDp9RylolhWb7Gq6e71ZOslcSwPaBNLt7KRbSq%2Fpqk6cbSLhwI%2BGf%2FafhQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230322T034750Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=ASIASG3IHPL6UQV3T6HY%2F20230322%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=f3589f0642bc1a1df364baaf54956b3df111ef51902673b92a6a4ac3e9ebe043 [following]
--2023-03-22 11:47:50-- https://jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com/aol-boostorg/filestore/68/68be4a43b73c66370c8d3fd94723b3913217ce1b?X-Artifactory-username=anonymous&X-Artifactory-repositoryKey=main&X-Artifactory-artifactPath=release%2F1.75.0%2Fsource%2Fboost_1_75_0.tar.gz&X-Artifactory-projectKey=default&x-jf-traceId=78bc8128fa9857a2&response-content-disposition=attachment%3Bfilename%3D%22boost_1_75_0.tar.gz%22&response-content-type=application%2Fx-gzip&X-Amz-Security-Token=IQoJb3JpZ2luX2VjENT%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLXdlc3QtMiJGMEQCIDNsJ6mm3ED2qEBKXXogmKqSUgCqmDiHytrUp5fRakRrAiBBzsixGsd2akUqjqrrWzgMe8HxdEP1LIAAoPsz8AXLTSqPBQid%2F%2F%2F%2F%2F%2F%2F%2F%2F%2F8BEAAaDDE1MjE1MzA2MjE0MSIMtL8rAWkfo90CvBSaKuMEzSfcNizP9ia5lyp%2BY%2FpmQyRtUe8i%2BPMNHxt1qeXLxlPDNDAdqNH64C5PGZ%2BoDYu1Y2ofhzbgf%2BjMA3pq1MalfvnJ33rVOrngAhwdSKROzg6qWIz3KtMc9vPpGyT1MjBVAZ9nnn9%2F2BpI99rNPobAX0jci6la4k9GDYrl%2FadSvR1VsW977fwlIGvfzm%2FjQih2RrSArGIHLMXmMSO4n61FxVvO6IoYJsks5cRf53dpJR0PbO2hN9tWidlTapcV00P0F2mMBnZK1dTFwicUjw4lHOf2mFVlFaV2%2FhbWgQV6ap0Xe3tCnCyhZR0GJWGDwOweGyf2oN8eX9Wg5kEDuFpw3TcnaqJ11Y%2B9Rrel%2F1NA8oRTGVGgQL5Ztcn5o7gexSi4LQWnWXaYRUFlUrmm7ovM04afpPiHscmjL1lqqn7p3ncFFRmluT91pX5nLE2cRjeZvyFcwiaeywrNfFM0azeg52aZNfdj2UfHV3PghQcxmi9oEO39FYHPFmVLVqDTurKrg3nWa%2FLAAuSQmW3AeO9zBjQvj7f21Sl%2BIGEgkgRYIKKla4iXVYPFSb4RtF9g%2BfdvBk1%2FhK%2BG6V%2BNW0j4vxeYrtyokIX4TJFBI6VdoM19AsHFPi6D4HJRhVzCOQAjFq1TgleBkt49J%2FPCT1a1BT8IBE8k%2BYownnvrtqOMZsi9qYWinypQBykqS0BztRmi6cUvNcUiEtyrUauSoVTYJJj4nj38nCh%2F7%2FO69x5SSCd3HOVaetg4a%2Br9w8Zwb9K8HRPTzTEHFjJ0GIea9GeqYViXY1MUSBgpUDj7PJI00AFq22%2FK8lQw9%2B7poAY6mwFSw6jCZhWYDjTNq%2F8XDXNmwdqaOSwjg%2FOasApDbXfv3cHi32kMFQYWGkd%2FmsupakPqJd1Y47uzn6c5YiZ8yb6s2IMCtjlxICvg4DlafPyr7K3dDS7S5LU6Im2VZZp0siN5I4RTx4SezK9xMffluU9XsDp9RylolhWb7Gq6e71ZOslcSwPaBNLt7KRbSq%2Fpqk6cbSLhwI%2BGf%2FafhQ%3D%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20230322T034750Z&X-Amz-SignedHeaders=host&X-Amz-Expires=60&X-Amz-Credential=ASIASG3IHPL6UQV3T6HY%2F20230322%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Signature=f3589f0642bc1a1df364baaf54956b3df111ef51902673b92a6a4ac3e9ebe043
Resolving jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)... 52.92.136.225, 52.92.165.97, 52.92.209.41, ...
Connecting to jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com (jfrog-prod-usw2-shared-oregon-main.s3.amazonaws.com)|52.92.136.225|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 143817536 (137M) [application/x-gzip]
Saving to: ‘boost_1_75_0.tar.gz’

boost_1_75_0.tar.gz 100%[=====================================================================================================================================================================================================>] 137.15M 13.2MB/s in 12s 

2023-03-22 11:48:03 (11.2 MB/s) - ‘boost_1_75_0.tar.gz’ saved [143817536/143817536]

[root@localhost ~]# sha256sum boost_1_75_0.tar.gz 
aeb26f80e80945e82ee93e5939baebdca47b9dee80a07d3144be1e1a6a66dd6a boost_1_75_0.tar.gz

But in Arrow 11's code it uses 267e04a7c0bfe85daf796dedc789c3a27a76707e1c968f0a2a87bb96331e2b61. Why?

arrow/cpp/thirdparty/versions.txt

Lines 36 to 37 in f10f5cf

	ARROW_BOOST_BUILD_VERSION=1.75.0
	ARROW_BOOST_BUILD_SHA256_CHECKSUM=267e04a7c0bfe85daf796dedc789c3a27a76707e1c968f0a2a87bb96331e2b61

Component(s)

C++

[zhztheplayer]

@kszucs @paleolimbot

[assignUser]

This might be an issue with the apache jfrog instance we use as the primary mirror which seems to be down/disabled at the moment. My guess is that the expected sha is for the trimmed boost source we host on jfrog and does not match the full sources published by boostorg.

We should probably have a second checksom in the versions.txt/cmake to account for this case.

[zhztheplayer]

This might be an issue with the apache jfrog instance we use as the primary mirror which seems to be down/disabled at the moment.

I see, thanks. And I also found that from official Boost site aeb26f80e80945e82ee93e5939baebdca47b9dee80a07d3144be1e1a6a66dd6a is suggested.

I'm going to change the checksum from our forked Arrow repo, but I assume the already-released Arrow 11 cpp code is currently not able to build. That might be hard to fix then.

[mapleFU]

Also meet the problem on ut and boost 1.81: https://github.com/apache/arrow/actions/runs/4486679773/jobs/7889390019?pr=34676

[assignUser]

I assume the already-released Arrow 11 cpp code is currently not able to build. That might be hard to fix then.

We can't change the sha in the 11.0.0 release but you can just install boost on the system you are building on and instruct cmake to use that instead of building from source (-DBoost_SOURCE=SYSTEM). That should be a workaround for this issue.

[mapleFU]

No we have:

if(DEFINED ENV{ARROW_BOOST_URL})
 set(BOOST_SOURCE_URL "$ENV{ARROW_BOOST_URL}")
else()
 string(REPLACE "." "_" ARROW_BOOST_BUILD_VERSION_UNDERSCORES
 ${ARROW_BOOST_BUILD_VERSION})
 set_urls(BOOST_SOURCE_URL
 # These are trimmed boost bundles we maintain.
 # See cpp/build-support/trim-boost.sh
 # FIXME(ARROW-6407) automate uploading this archive to ensure it reflects
 # our currently used packages and doesn't fall out of sync with
 # ${ARROW_BOOST_BUILD_VERSION_UNDERSCORES}
 "${THIRDPARTY_MIRROR_URL}/boost_${ARROW_BOOST_BUILD_VERSION_UNDERSCORES}.tar.gz"
 "https://boostorg.jfrog.io/artifactory/main/release/${ARROW_BOOST_BUILD_VERSION}/source/boost_${ARROW_BOOST_BUILD_VERSION_UNDERSCORES}.tar.gz"
 "https://sourceforge.net/projects/boost/files/boost/${ARROW_BOOST_BUILD_VERSION}/boost_${ARROW_BOOST_BUILD_VERSION_UNDERSCORES}.tar.gz"
 )
endif()

I guess maybe they have different hash tags. We can check it later

[kou]

Yes. We have 2 Boost source archives. One is a trimmed version by us and another is the official version.

Our checksum is for the trimmed version. So the checksum isn't matched with the official version.

A solution was provided by @assignUser in #34675 (comment) :

We should probably have a second checksom in the versions.txt/cmake to account for this case.

[benz0li]

Installation of R package fails, too: https://gitlab.b-data.ch/r/tidyverse/-/pipelines/17678.

---

Steps to reproduce

On a machine with docker:

docker run --rm -e ARROW_R_DEV=true -ti glcr.b-data.ch/r/base bash

In the container:

apt-get update \
 && apt-get -y install --no-install-recommends \
 cmake \
 default-libmysqlclient-dev \
 libfribidi-dev \
 libgit2-dev \
 libharfbuzz-dev \
 libpq-dev \
 libsasl2-dev \
 libsqlite3-dev \
 libssh2-1-dev \
 libtiff-dev \
 libxtst6 \
 unixodbc-dev \
 && install2.r --error --skipinstalled -n $NCPUS arrow

👉 Logfile.txt

Crucial part:

-- verifying file...
 file='/tmp/RtmpligywJ/file104d3dd49f1b/boost_ep-prefix/src/boost_1_81_0.tar.gz'
-- SHA256 hash of
 /tmp/RtmpligywJ/file104d3dd49f1b/boost_ep-prefix/src/boost_1_81_0.tar.gz
 does not match expected value
 expected: '9e0ffae35528c35f90468997bc8d99500bf179cbae355415a89a600c38e13574'
 actual: '205666dea9f6a7cfed87c7a6dfbeb52a2c1b9de55712c9c1a87735d7181452b6'
-- Hash mismatch, removing...

-- verifying file...
 file='/tmp/RtmpligywJ/file104d3dd49f1b/boost_ep-prefix/src/boost_1_81_0.tar.gz'
-- SHA256 hash of
 /tmp/RtmpligywJ/file104d3dd49f1b/boost_ep-prefix/src/boost_1_81_0.tar.gz
 does not match expected value
 expected: '9e0ffae35528c35f90468997bc8d99500bf179cbae355415a89a600c38e13574'
 actual: '8c7cee4bed8b23faf8a60e682a03655ad28c80fabd5d4004176d756f0120e101'
-- Hash mismatch, removing...

Download locations:

1. https://apache.jfrog.io/artifactory/arrow/thirdparty/7.0.0/boost_1_81_0.tar.gz
   
   (8c7cee4bed8b23faf8a60e682a03655ad28c80fabd5d4004176d756f0120e101 is the checksum of the JFrog landing page...)
2. https://boostorg.jfrog.io/artifactory/main/release/1.81.0/source/boost_1_81_0.tar.gz
3. https://sourceforge.net/projects/boost/files/boost/1.81.0/boost_1_81_0.tar.gz

---

FYI @eitsupi