Uh oh!
There was an error while loading. Please reload this page.
[fix](thirdparty) fix four ways the thirdparty download step fails or ships corruption - #66650
Merged
morningman merged 5 commits intoAug 11, 2026
Conversation
…d build `build-thirdparty.sh arrow_adbc` passes the package name through to download-thirdparty.sh, which narrows TP_ARCHIVES to exactly the named entries. ARROW_ADBC_FLIGHTSQL is a companion archive of arrow_adbc rather than a package of its own -- it has no build function, build_arrow_adbc() only copies the prebuilt driver out of it -- so its name never appears on a command line and the narrowing drops it. The archive is then neither downloaded nor unpacked, and the build dies on thirdparty/src/adbc_driver_flightsql does not exist. A full `build-thirdparty.sh` run is unaffected. The targeted run fails on a clean tree, after --clean, and -- less obviously -- on every run after the first: cleanup_package_source() removes the unpacked driver once arrow_adbc is built, and the narrowed download list never restores it, so the first run succeeding is what makes the second one fail. Pull the companion archive in at the narrowing site, guarded on the platforms that register it. Upstream publishes no prebuilt driver for macOS or other architectures, where ARROW_ADBC_FLIGHTSQL_SOURCE stays unset and the archive is not part of TP_ARCHIVES to begin with. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TY1QjAasVmxqqQj8LPueBA
The shebang only takes effect when these scripts are executed directly. `sh build-thirdparty.sh` hands them to /bin/sh instead, which is dash on Debian and Ubuntu and parses none of the `[[ ]]`, arrays and here-strings they are built on. dash does not stop at the first of them either, it keeps going and runs a mangled version of the script: build-thirdparty.sh: 34: Bad substitution build-thirdparty.sh: 42: [[: not found build-thirdparty.sh: 120: Syntax error: redirection unexpected Re-exec under bash so that the way the script was invoked cannot decide whether the build works. The block is kept POSIX so the shell that is about to be replaced can parse it, and it is a no-op under bash, which always sets BASH_VERSION. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TY1QjAasVmxqqQj8LPueBA
hello-stephen
commented
Aug 11, 2026
Contributor
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
The doris-thirdparty source bundle the CI jobs unpack currently ships a 0-byte `src/tsan_interface_atomic.h`, so every job fails its md5 check and falls back to fetching the header from gcc.gnu.org. That host is not reachable from every runner -- in run 31463316509 the Linux and macOS jobs got it, macOS-arm64 did not and the build died with Failed to download tsan_interface_atomic.h `download_func()` loops twice, but when a corrupt file is already on disk the first pass is spent detecting and removing it, so an unreachable host gets one single attempt. Point a fallback at the GitHub gcc mirror: the gcc-7 branch is closed and the mirrored file hashes to the md5 already recorded here, so the two sources cannot drift apart. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TY1QjAasVmxqqQj8LPueBA
download-thirdparty.sh disabled every checksum when GNU md5sum was absent, which is the normal state on macOS. That is not a neutral loss of coverage: `wget -q URL -O file` creates its output file before the transfer, so a failed download leaves a 0-byte file behind, and the second pass of the retry loop in `download_func()` then finds a readable file, "verifies" it against the disabled checksum, reports `Archive ... already exist` and lets the script exit 0 with a corrupt archive on disk. That is how a 0-byte `src/tsan_interface_atomic.h` reached the published doris-thirdparty source bundle on 2026-08-06: the job that builds it runs on macOS, so nothing there could notice. Fall back to BSD md5, and compare digests rather than whole output lines since the two tools agree on nothing but the digest. This also makes `md5sum_bin` the command that actually runs -- it was detected and then ignored in favour of a literal `md5sum`. Every macOS consumer gains verification it silently lacked: the macOS jobs of build-thirdparty.yml and be-ut-mac.yml, the doris-thirdparty release jobs, and anyone building Doris on a Mac. The test imitates macOS with a PATH that holds no md5sum and a BSD md5 stub, so it runs anywhere, and feeds the script an empty response returned with a success status -- a case only a checksum can catch. Dropping the md5 fallback makes it fail. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TY1QjAasVmxqqQj8LPueBA
thirdparty/test/ has held download-thirdparty-fallback-test.sh and juicefs-default-mirror-test.sh for a while, but no workflow referenced them and neither carried an executable bit, so neither had ever run. Give them a job of their own next to the lifecycle test, together with the md5 test added in the previous commit. They drive download-thirdparty.sh against stubbed downloads, so they need none of the toolchain the build jobs spend an hour preparing, and finish in seconds. Guarding this file with tests that never run is part of why the checksum gap stayed invisible for as long as it did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TY1QjAasVmxqqQj8LPueBA
morningman
commented
Aug 11, 2026
ContributorAuthor
skip buildall |
hello-stephen
approved these changes
Aug 11, 2026
Uh oh!
There was an error while loading. Please reload this page.
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 17, 2026
… ships corruption (apache#66650) ### What problem does this PR solve? Related PR: apache#66358 Problem Summary: `sh build-thirdparty.sh arrow_adbc` fails with ``` -- Installing: .../thirdparty/installed/lib64/pkgconfig/adbc-driver-sqlite.pc .../thirdparty/src/adbc_driver_flightsql does not exist. ``` `build-thirdparty.sh` passes the package names straight through to `download-thirdparty.sh`, which narrows `TP_ARCHIVES` to exactly the named entries. `ARROW_ADBC_FLIGHTSQL` is a companion archive of `arrow_adbc` rather than a package of its own -- it has no build function, `build_arrow_adbc()` only copies the prebuilt driver out of it -- so its name never appears on a command line and the narrowing drops it. The wheel is then neither downloaded nor unpacked, and `check_if_source_exist` kills the build. A full `build-thirdparty.sh` run is unaffected: | invocation | result | | --- | --- | | `build-thirdparty.sh` (no package argument) | ok | | `build-thirdparty.sh arrow_adbc` on a clean tree | **fails** | | `build-thirdparty.sh arrow_adbc` twice in a row | first ok, **second fails** | | `build-thirdparty.sh --clean arrow_adbc` | **fails** | The third row is the confusing one: `cleanup_package_source()` deletes the unpacked driver once `arrow_adbc` is built, and the narrowed download list never restores it, so the first run succeeding is exactly what makes the second one fail. The first commit pulls the companion archive in at the narrowing site, guarded on the platforms that register it -- upstream publishes no prebuilt driver for macOS or other architectures, where `ARROW_ADBC_FLIGHTSQL_SOURCE` stays unset and the archive is not part of `TP_ARCHIVES` to begin with. The second commit is unrelated to the failure above, but was noticed while debugging it. The thirdparty scripts carry a `#!/usr/bin/env bash` shebang, which is ignored when they are started as `sh build-thirdparty.sh`. On Debian and Ubuntu `/bin/sh` is dash, which parses none of the `[[ ]]`, arrays and here-strings these scripts are built on -- and does not stop at the first one, it runs a mangled version of the script. They now re-exec themselves under bash. The third commit fixes a CI failure this PR ran into, which predates it and hits every thirdparty build. The `doris-thirdparty` source bundle the jobs unpack currently ships a 0-byte `src/tsan_interface_atomic.h` (it was still intact on 2026-08-01, run 30679875385), so every job now fails its md5 check and falls back to fetching the header from `gcc.gnu.org`. That host is not reachable from every runner: in run 31463316509 the Linux and macOS jobs got it, macOS-arm64 did not and the build died with `Failed to download tsan_interface_atomic.h`. `download_func()` loops twice, but when a corrupt file is already on disk the first pass is spent detecting and removing it, so an unreachable host gets a single attempt. A fallback now points at the GitHub gcc mirror. The fourth commit fixes why that 0-byte header was ever published. The job that builds the bundle runs on macOS, and `download-thirdparty.sh` disabled every checksum when GNU `md5sum` was absent, which is the normal state there. Combined with `wget -q URL -O file` creating its output file before the transfer, a failed download left a 0-byte file that the unverified retry then reported as `Archive ... already exist`, and the script exited 0 with a corrupt archive on disk. It now falls back to BSD `md5`. This is the widest of the four: every macOS consumer gains verification it silently lacked -- the macOS jobs here, `be-ut-mac.yml`, the `doris-thirdparty` release jobs, and anyone building Doris on a Mac, where none of the ~87 archives were being checked. Note that the already-published bundle is still corrupt. It is rebuilt whenever `git diff` reports changes under `thirdparty/`, so merging this PR is itself the trigger -- and by then the rebuild will run with the checksum fix in place. The fifth commit gives `thirdparty/test/` a CI job. Two of the three scripts there were referenced by no workflow and carried no executable bit, so they had never run; guarding `download-thirdparty.sh` with tests that never run is part of why the checksum gap stayed invisible for as long as it did. They stub out the downloads, so the job needs none of the toolchain the build jobs spend an hour preparing and finishes in seconds. ### Release note None ### Check List (For Author) - Test - [x] Manual test (add detailed scripts or steps below) <details> <summary>Manual test details</summary> Linux / x86_64, with both archives already in `thirdparty/src` so nothing is downloaded. **Targeted download, before and after** ``` # before (upstream master) $ bash download-thirdparty.sh arrow_adbc Download and build specified libs only: ARROW_ADBC $ ls src/ | grep '^adbc_driver_flightsql$' (nothing -- this is the state build_arrow_adbc() then dies on) # after $ bash download-thirdparty.sh arrow_adbc Download and build specified libs only: ARROW_ADBC ARROW_ADBC_FLIGHTSQL $ ls -la src/adbc_driver_flightsql/libadbc_driver_flightsql.so -rwxr-xr-x 1 ... 33693080 ... src/adbc_driver_flightsql/libadbc_driver_flightsql.so ``` **Full targeted build, run twice from a tree that only holds the archives** ``` $ sh build-thirdparty.sh arrow_adbc # RUN1_EXIT=0 $ sh build-thirdparty.sh arrow_adbc # RUN2_EXIT=0 $ ls installed/lib64/ | grep adbc libadbc_driver_flightsql.so libadbc_driver_jni.so libadbc_driver_manager.a ... ``` **Edge cases** ``` $ bash download-thirdparty.sh arrow_adbc_flightsql Download and build specified libs only: ARROW_ADBC_FLIGHTSQL # companion does not drag its owner in $ bash download-thirdparty.sh arrow_adbc arrow_adbc_flightsql Download and build specified libs only: ARROW_ADBC ARROW_ADBC_FLIGHTSQL # not duplicated # with `uname -s` stubbed to Darwin, so ARROW_ADBC_FLIGHTSQL_SOURCE stays unset $ bash download-thirdparty.sh arrow_adbc Download and build specified libs only: ARROW_ADBC # companion correctly not added ``` Removing the `-n "${ARROW_ADBC_FLIGHTSQL_SOURCE}"` guard makes that last case fail with `Failed to download ` (empty archive name), so the guard is load bearing rather than decorative. **Second commit, under a real dash** (`python:3.11-slim`, `/bin/sh` -> `/usr/bin/dash`) ``` # before $ sh build-thirdparty.sh -h ; echo $? build-thirdparty.sh: 34: Bad substitution build-thirdparty.sh: 42: [[: not found build-thirdparty.sh: 72: [[: not found build-thirdparty.sh: 114: [[: not found build-thirdparty.sh: 120: Syntax error: redirection unexpected 2 # after -- re-execs under bash and reaches the real toolchain check # (the container has no LDB toolchain, which is why it stops there) $ sh build-thirdparty.sh -h ; echo $? Target system: Linux; Target arch: x86_64 Python 3.11.15 Error: wrong directory DORIS_GCC_HOME=./.. 1 ``` Under bash the guard is a no-op: `BASH_VERSION` is set, so no re-exec happens and behaviour is unchanged. **Third commit, reproducing the exact CI failure locally** A 0-byte `tsan_interface_atomic.h` on disk plus an unreachable primary host, which is what the macOS-arm64 runner saw: ``` $ printf '' > src/tsan_interface_atomic.h # same md5 as the bundle's copy $ bash download-thirdparty.sh tsan_header # TSAN_HEADER_DOWNLOAD temporarily broken src/tsan_interface_atomic.h md5sum check failed! Archive tsan_interface_atomic.h will be removed and download again. Downloading tsan_interface_atomic.h from https://gcc.gnu.org.invalid/unreachable-on-purpose ... Failed to download tsan_interface_atomic.h. attemp: 2 Failed to download tsan_interface_atomic.h src/tsan_interface_atomic.h md5sum check failed! Archive tsan_interface_atomic.h will be removed and download again. Downloading tsan_interface_atomic.h from https://raw.githubusercontent.com/gcc-mirror/gcc/... Success to download tsan_interface_atomic.h $ md5sum src/tsan_interface_atomic.h d72679bea167d6a513d959f5abd149dc # matches TSAN_HEADER_MD5SUM ``` **Fourth commit** `thirdparty/test/download-thirdparty-md5-test.sh` is added alongside it. It imitates macOS with a PATH that holds no `md5sum` and a BSD `md5` stub, so it runs on any host, and feeds the script an empty body returned with a success status -- a case only a checksum can catch. All three scripts under `thirdparty/test/` pass: ``` [PASS rc=0] download-thirdparty-fallback-test.sh [PASS rc=0] juicefs-default-mirror-test.sh [PASS rc=0] download-thirdparty-md5-test.sh ``` The first two are the pre-existing ones, run to confirm that changing the comparison in `md5sum_func()` from whole output lines to bare digests did not break them. Dropping the `md5` fallback again makes the new test fail, so it is not a test that cannot fail: ``` FAIL: download-thirdparty.sh accepted an empty archive ```
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #66358
Problem Summary:
sh build-thirdparty.sh arrow_adbcfails withbuild-thirdparty.shpasses the package names straight through todownload-thirdparty.sh, which narrowsTP_ARCHIVESto exactly the named entries.ARROW_ADBC_FLIGHTSQLis a companion archive ofarrow_adbcrather than a package ofits own -- it has no build function,
build_arrow_adbc()only copies the prebuiltdriver out of it -- so its name never appears on a command line and the narrowing drops
it. The wheel is then neither downloaded nor unpacked, and
check_if_source_existkillsthe build.
A full
build-thirdparty.shrun is unaffected:build-thirdparty.sh(no package argument)build-thirdparty.sh arrow_adbcon a clean treebuild-thirdparty.sh arrow_adbctwice in a rowbuild-thirdparty.sh --clean arrow_adbcThe third row is the confusing one:
cleanup_package_source()deletes the unpackeddriver once
arrow_adbcis built, and the narrowed download list never restores it, sothe first run succeeding is exactly what makes the second one fail.
The first commit pulls the companion archive in at the narrowing site, guarded on the
platforms that register it -- upstream publishes no prebuilt driver for macOS or other
architectures, where
ARROW_ADBC_FLIGHTSQL_SOURCEstays unset and the archive is notpart of
TP_ARCHIVESto begin with.The second commit is unrelated to the failure above, but was noticed while debugging it.
The thirdparty scripts carry a
#!/usr/bin/env bashshebang, which is ignored when theyare started as
sh build-thirdparty.sh. On Debian and Ubuntu/bin/shis dash, whichparses none of the
[[ ]], arrays and here-strings these scripts are built on -- anddoes not stop at the first one, it runs a mangled version of the script. They now
re-exec themselves under bash.
The third commit fixes a CI failure this PR ran into, which predates it and hits every
thirdparty build. The
doris-thirdpartysource bundle the jobs unpack currently ships a0-byte
src/tsan_interface_atomic.h(it was still intact on 2026-08-01, run30679875385), so every job now fails its md5 check and falls back to fetching the header
from
gcc.gnu.org. That host is not reachable from every runner: in run 31463316509 theLinux and macOS jobs got it, macOS-arm64 did not and the build died with
Failed to download tsan_interface_atomic.h.download_func()loops twice, but when a corrupt fileis already on disk the first pass is spent detecting and removing it, so an unreachable
host gets a single attempt. A fallback now points at the GitHub gcc mirror.
The fourth commit fixes why that 0-byte header was ever published. The job that builds
the bundle runs on macOS, and
download-thirdparty.shdisabled every checksum when GNUmd5sumwas absent, which is the normal state there. Combined withwget -q URL -O filecreating its output file before the transfer, a failed download left a 0-byte file that
the unverified retry then reported as
Archive ... already exist, and the script exited0 with a corrupt archive on disk. It now falls back to BSD
md5.This is the widest of the four: every macOS consumer gains verification it silently
lacked -- the macOS jobs here,
be-ut-mac.yml, thedoris-thirdpartyrelease jobs, andanyone building Doris on a Mac, where none of the ~87 archives were being checked.
Note that the already-published bundle is still corrupt. It is rebuilt whenever
git diffreports changes under
thirdparty/, so merging this PR is itself the trigger -- and bythen the rebuild will run with the checksum fix in place.
The fifth commit gives
thirdparty/test/a CI job. Two of the three scripts there werereferenced by no workflow and carried no executable bit, so they had never run; guarding
download-thirdparty.shwith tests that never run is part of why the checksum gap stayedinvisible for as long as it did. They stub out the downloads, so the job needs none of the
toolchain the build jobs spend an hour preparing and finishes in seconds.
Release note
None
Check List (For Author)
Manual test details
Linux / x86_64, with both archives already in
thirdparty/srcso nothing is downloaded.Targeted download, before and after
Full targeted build, run twice from a tree that only holds the archives
Edge cases
Removing the
-n "${ARROW_ADBC_FLIGHTSQL_SOURCE}"guard makes that last case fail withFailed to download(empty archive name), so the guard is load bearing rather thandecorative.
Second commit, under a real dash (
python:3.11-slim,/bin/sh->/usr/bin/dash)Under bash the guard is a no-op:
BASH_VERSIONis set, so no re-exec happens andbehaviour is unchanged.
Third commit, reproducing the exact CI failure locally
A 0-byte
tsan_interface_atomic.hon disk plus an unreachable primary host, which iswhat the macOS-arm64 runner saw:
Fourth commit
thirdparty/test/download-thirdparty-md5-test.shis added alongside it. It imitates macOSwith a PATH that holds no
md5sumand a BSDmd5stub, so it runs on any host, and feedsthe script an empty body returned with a success status -- a case only a checksum can
catch. All three scripts under
thirdparty/test/pass:The first two are the pre-existing ones, run to confirm that changing the comparison in
md5sum_func()from whole output lines to bare digests did not break them.Dropping the
md5fallback again makes the new test fail, so it is not a test that cannotfail:
Behavior changed:
Does this need documentation?