Skip to content

ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure - #11583

Closed
pitrou wants to merge 3 commits into
apache:masterfrom
pitrou:oss-fuzz-gtest
Closed

ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure#11583
pitrou wants to merge 3 commits into
apache:masterfrom
pitrou:oss-fuzz-gtest

Conversation

@pitrou

@pitroupitrou commented Nov 1, 2021

Copy link
Copy Markdown
Member

ARROW-14393 downgraded the bundled GTest version to a version that doesn't build on the OSS-Fuzz setup
(and there doesn't seem to be an obvious fix or workaround around those build errors).

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Also, add a Crossbow task for the OSS-Fuzz build check, since the Github Actions cron job failure went unnoticed.

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g cpp

@github-actions

Copy link
Copy Markdown

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@pitroupitrou changed the title [CI] [WIP] Always compile GTest from source[CI] Always compile GTest from sourceNov 1, 2021
@pitrou
pitrou marked this pull request as draft November 1, 2021 15:39
@github-actions

Copy link
Copy Markdown

Revision: 209b1e6382c4fb4bc81e4f6a3e725fe2d1faddb9

Submitted crossbow builds: ursacomputing/crossbow @ actions-1063

TaskStatus
test-conda-cppGithub Actions
test-conda-cpp-valgrindAzure
test-debian-10-cppGithub Actions
test-debian-11-cppGithub Actions
test-fedora-33-cppGithub Actions
test-ubuntu-18.04-cppGithub Actions
test-ubuntu-18.04-cpp-releaseGithub Actions
test-ubuntu-18.04-cpp-staticGithub Actions
test-ubuntu-20.04-cppGithub Actions
test-ubuntu-20.04-cpp-14Github Actions
test-ubuntu-20.04-cpp-17Github Actions
test-ubuntu-20.04-cpp-bundledGithub Actions
test-ubuntu-20.04-cpp-thread-sanitizerGithub Actions

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-*

@github-actions

Copy link
Copy Markdown

Revision: 4da0482af9451acbbd5c7efef6d83040e5d48d97

Submitted crossbow builds: ursacomputing/crossbow @ actions-1064

TaskStatus
test-conda-cppGithub Actions
test-conda-cpp-valgrindAzure
test-conda-python-3.10Github Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-masterGithub Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-conda-python-3.9-spark-masterGithub Actions

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-*

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g verify-rc-source

@github-actions

Copy link
Copy Markdown

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1065

TaskStatus
test-conda-cppGithub Actions
test-conda-cpp-valgrindAzure
test-conda-python-3.10Github Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-masterGithub Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-conda-python-3.9-spark-masterGithub Actions

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit vcpkg

@github-actions

Copy link
Copy Markdown

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1066

TaskStatus
verify-rc-source-cpp-linux-amd64Github Actions
verify-rc-source-cpp-macos-amd64Github Actions
verify-rc-source-cpp-macos-arm64Github Actions
verify-rc-source-csharp-linux-amd64Github Actions
verify-rc-source-csharp-macos-amd64Github Actions
verify-rc-source-csharp-macos-arm64Github Actions
verify-rc-source-go-linux-amd64Github Actions
verify-rc-source-go-macos-amd64Github Actions
verify-rc-source-go-macos-arm64Github Actions
verify-rc-source-integration-linux-amd64Github Actions
verify-rc-source-integration-macos-amd64Github Actions
verify-rc-source-integration-macos-arm64Github Actions
verify-rc-source-java-linux-amd64Github Actions
verify-rc-source-java-macos-amd64Github Actions
verify-rc-source-js-linux-amd64Github Actions
verify-rc-source-js-macos-amd64Github Actions
verify-rc-source-js-macos-arm64Github Actions
verify-rc-source-python-linux-amd64Github Actions
verify-rc-source-python-macos-amd64Github Actions
verify-rc-source-python-macos-arm64Github Actions
verify-rc-source-ruby-linux-amd64Github Actions
verify-rc-source-ruby-macos-amd64Github Actions
verify-rc-source-ruby-macos-arm64Github Actions
verify-rc-source-windowsGithub Actions

@github-actions

Copy link
Copy Markdown

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1067

TaskStatus
test-build-vcpkg-winGithub Actions

@pitrou

Copy link
Copy Markdown
MemberAuthor

Hmm, it appears we cannot exercise the release verification script using crossbow (it tries to fetch Arrow 1.0.0)? @kszucs

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-*

@github-actions

Copy link
Copy Markdown

Revision: a9cd09f34907e8183ebcdbca0f44d3145e17b11c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1068

TaskStatus
test-conda-cppGithub Actions
test-conda-cpp-valgrindAzure
test-conda-python-3.10Github Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-masterGithub Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-conda-python-3.9-spark-masterGithub Actions

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g fuzz

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g fuzz

@pitroupitrou changed the title [CI] Always compile GTest from sourceARROW-14554: [C++][CI] Fix OSS-Fuzz build failureNov 2, 2021
@github-actions

Copy link
Copy Markdown

@github-actions

Copy link
Copy Markdown

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g fuzz

@github-actions

Copy link
Copy Markdown

Revision: 7cf7cd67bf5f9748a95787eb0b824bae6d787036

Submitted crossbow builds: ursacomputing/crossbow @ actions-1069

TaskStatus
test-build-cpp-fuzzGithub Actions

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g fuzz

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g fuzz

@github-actions

Copy link
Copy Markdown

Revision: 860317fead19201a884c7b2029a4ba5d9b8eb62f

Submitted crossbow builds: ursacomputing/crossbow @ actions-1072

TaskStatus
test-build-cpp-fuzzGithub Actions

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit -g fuzz

@github-actions

Copy link
Copy Markdown

Revision: 90971770ecc6e54fef48de05e50aa9e8f44c4713

Submitted crossbow builds: ursacomputing/crossbow @ actions-1073

TaskStatus
test-build-cpp-fuzzGithub Actions

ARROW-14393 downgraded the bundled GTest version to a version that doesn't build on the OSS-Fuzz setup
(and there doesn't seem to be an obvious fix or workaround around those build errors).
Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.
Also, add a Crossbow task for the OSS-Fuzz build check, since the Github Actions cron job failure went unnoticed.
@pitrou
pitrou marked this pull request as ready for review November 2, 2021 15:52
@pitrou
pitrou requested a review from kszucsNovember 2, 2021 15:52
@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-*

@github-actions

Copy link
Copy Markdown

Revision: 0bc448e

Submitted crossbow builds: ursacomputing/crossbow @ actions-1074

TaskStatus
test-conda-cppGithub Actions
test-conda-cpp-valgrindAzure
test-conda-python-3.10Github Actions
test-conda-python-3.6Github Actions
test-conda-python-3.6-pandas-0.23Github Actions
test-conda-python-3.7Github Actions
test-conda-python-3.7-hdfs-2.9.2Github Actions
test-conda-python-3.7-hdfs-3.2.1Github Actions
test-conda-python-3.7-kartothek-latestGithub Actions
test-conda-python-3.7-kartothek-masterGithub Actions
test-conda-python-3.7-pandas-0.24Github Actions
test-conda-python-3.7-pandas-latestGithub Actions
test-conda-python-3.7-spark-v3.1.2Github Actions
test-conda-python-3.8Github Actions
test-conda-python-3.8-hypothesisGithub Actions
test-conda-python-3.8-pandas-latestGithub Actions
test-conda-python-3.8-pandas-nightlyGithub Actions
test-conda-python-3.8-spark-v3.2.0Github Actions
test-conda-python-3.9Github Actions
test-conda-python-3.9-dask-latestGithub Actions
test-conda-python-3.9-dask-masterGithub Actions
test-conda-python-3.9-pandas-masterGithub Actions
test-conda-python-3.9-spark-masterGithub Actions

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-*

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-cpp-valgrind

@github-actions

Copy link
Copy Markdown

Revision: e035ed5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1075

TaskStatus
test-conda-cpp-valgrindAzure

@pitrou

Copy link
Copy Markdown
MemberAuthor

@github-actions crossbow submit test-conda-cpp-valgrind

@github-actions

Copy link
Copy Markdown

Revision: e035ed5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1077

TaskStatus
test-conda-cpp-valgrindAzure

@pitrou

Copy link
Copy Markdown
MemberAuthor

@kszucs Can we merge this? I'd like to unblock the OSS-Fuzz builds.

@kszucs

Copy link
Copy Markdown
Member

Hmm, it appears we cannot exercise the release verification script using crossbow (it tries to fetch Arrow 1.0.0)? @kszucs

We can only build releases with those tasks by providing additional parameters: #11511 (comment)

@kszucs

Copy link
Copy Markdown
Member

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Do we have a JIRA for that?

@pitrou

Copy link
Copy Markdown
MemberAuthor

I don't understand the question. What do you want a JIRA for?

@kszucskszucs closed this in 64b80ddNov 3, 2021
@pitrou
pitrou deleted the oss-fuzz-gtest branch November 3, 2021 15:48
@ursabot

ursabot commented Nov 3, 2021

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 09b79a1 and contender = 64b80dd. 64b80dd is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️1.03% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.36% ⬆️0.04%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

@kszucs

Copy link
Copy Markdown
Member

I don't understand the question. What do you want a JIRA for?

Does this fix the windows verification issue as well?

@pitrou

pitrou commented Nov 3, 2021

Copy link
Copy Markdown
MemberAuthor

Does this fix the windows verification issue as well?

Hopefully, but I have no way to verify it (as explained above :-)).

@pitrou

Copy link
Copy Markdown
MemberAuthor

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@pitrou@kszucs@ursabot