Skip to content

[CI][C++] MinGW jobs fail every run after MSYS2 toolchain updates #49930

Description

@tadeja

Describe the bug, including details regarding any error messages, version, and platform.

The MINGW64 and CLANG64 MinGW jobs in .github/workflows/cpp.yml have failed every run since 2026 April 30 - first failure on main: https://github.com/apache/arrow/actions/runs/25213414021/job/73928546151#step:12:2644

The following tests FAILED:
41 - arrow-async-utility-test (Exit code 0xc0000374) arrow-tests unittest
62 - arrow-dataset-dataset-writer-test (SEGFAULT) arrow_dataset unittest
65 - arrow-dataset-file-test (Failed) arrow_dataset unittest
Error: Process completed with exit code 8.

Recent failure 2026 May 05 - https://github.com/apache/arrow/actions/runs/25379694834/job/74424280352#step:12:2468

The following tests FAILED:
41 - arrow-async-utility-test (Exit code 0xc0000374) arrow-tests unittest
44 - arrow-threading-utility-test (Timeout) arrow-tests unittest
62 - arrow-dataset-dataset-writer-test (Failed) arrow_dataset unittest
65 - arrow-dataset-file-test (Failed) arrow_dataset unittest
76 - arrow-s3fs-test (Failed) arrow-tests filesystem unittest
Error: Process completed with exit code 8.

My empirical isolation (PR #49931 commit fb4d7a0, all pins removed, MinIO bumped to RELEASE.2025-01-20T14-49-07Z) - CI run https://github.com/apache/arrow/actions/runs/25442601074/job/74637722454#step:13:1327

The following tests FAILED:
41 - arrow-async-utility-test (Exit code 0xc0000374) arrow-tests unittest
44 - arrow-threading-utility-test (Timeout) arrow-tests unittest
62 - arrow-dataset-dataset-writer-test (Failed) arrow_dataset unittest
65 - arrow-dataset-file-test (Failed) arrow_dataset unittest
76 - arrow-s3fs-test (Timeout) arrow-tests filesystem unittest

CLANG64 in the same run: 100% pass. Confirms gcc-16 (not the gcc-libs cascade or aws-sdk-cpp behavior) is what triggers the cluster.


Likely triggered by two MSYS2 updates:

  1. MINGW64: gcc 15.2 -> 16.1 triggers several test failures
    Latest CI status from PR 49462's MINGW64 CI - https://github.com/apache/arrow/actions/runs/25458933029/job/74695767495?pr=49462

Per-test status:
(always failing)

  • arrow-async-utility-test : Heap corruption (0xc0000374) - Unfixed; root cause unclear
  • arrow-threading-utility-test : SegFault then 300s timeout - Unfixed; could be another __emutls site or a gcc-16 codegen issue
  • arrow-dataset-dataset-writer-test : std::bad_weak_ptr in BatchWriteConcurrent - Unfixed; possibly an Arrowshared_from_this() lifetime bug exposed by gcc-16
  • arrow-dataset-file-test : SegFault - Unfixed; possibly the same race as dataset-writer-test
  • arrow-s3fs-test : see point 2. below

(intermittent)

PR #49462 is necessary but not enough. The other failing tests need their own fixes: possibly in Arrow, possibly in MSYS2/gcc, possibly some mix.

  1. MINGW64 + CLANG64: arrow-s3fs-test fails - aws-sdk-cpp 1.11.479 -> 1.11.801 stopped sending Content-Md5 on DeleteObjects; bundled MinIO (RELEASE.2024-09-13) still requires it. Resolved in GH-49930: [CI][C++] Pin MinGW MSYS2 packages to unblock CI #49931 by bumping bundled MinIO.

Temporary workaround: pin MSYS2 packages to last known good versions on MINGW64 in #49931

This issue tracks reverting to current upstream MSYS2 packages once
all 5(or more) failing tests pass on current upstream MSYS2 (without temporary pins of #49931!). See point (1.) for per-TEST STATUS

Component(s)

Continuous Integration, C++

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions