Rerun flaky SSHRemoteJobOperator kill test on process-group races by shahar1 · Pull Request #69384 · apache/airflow

shahar1 · 2026-07-04T15:28:10Z

TestPosixKillBehaviour::test_kill_terminates_whole_job_tree (added in #68644) intermittently fails its pre-kill assertion on CI:

AssertionError: job tree should be running before kill
assert False = _group_alive(326)

The wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real process group is already empty by the time pgrep -g runs — failing the check before the kill is even exercised. Passes locally (15/15, and under xdist), fails only on certain CI runners, so it's environment-dependent and lands on unrelated PRs.

Marks the test with @pytest.mark.flaky(reruns=5) (the convention already used elsewhere in this provider) so a fresh re-launch clears the race.

Note: if setsid forks in a real SSH session too, $! would be the wrong PID there and on_kill could orphan the job — a possible latent issue in the wrapper itself, left for a separate change.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_kill_terminates_whole_job_tree intermittently fails its pre-kill check on CI: the wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real group is empty by the time pgrep -g runs. Rerun on a fresh draw so the environment-dependent race does not fail unrelated PRs.

test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - #69384 had them on the sibling test until #69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…pache#70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

shahar1 · 2026-07-04T15:28:10Z

TestPosixKillBehaviour::test_kill_terminates_whole_job_tree (added in #68644) intermittently fails its pre-kill assertion on CI:

AssertionError: job tree should be running before kill
assert False = _group_alive(326)

The wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real process group is already empty by the time pgrep -g runs — failing the check before the kill is even exercised. Passes locally (15/15, and under xdist), fails only on certain CI runners, so it's environment-dependent and lands on unrelated PRs.

Marks the test with @pytest.mark.flaky(reruns=5) (the convention already used elsewhere in this provider) so a fresh re-launch clears the race.

Note: if setsid forks in a real SSH session too, $! would be the wrong PID there and on_kill could orphan the job — a possible latent issue in the wrapper itself, left for a separate change.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_kill_terminates_whole_job_tree intermittently fails its pre-kill check on CI: the wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real group is empty by the time pgrep -g runs. Rerun on a fresh draw so the environment-dependent race does not fail unrelated PRs.

test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - #69384 had them on the sibling test until #69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…pache#70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

shahar1 · 2026-07-04T15:28:10Z

TestPosixKillBehaviour::test_kill_terminates_whole_job_tree (added in #68644) intermittently fails its pre-kill assertion on CI:

AssertionError: job tree should be running before kill
assert False = _group_alive(326)

The wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real process group is already empty by the time pgrep -g runs — failing the check before the kill is even exercised. Passes locally (15/15, and under xdist), fails only on certain CI runners, so it's environment-dependent and lands on unrelated PRs.

Marks the test with @pytest.mark.flaky(reruns=5) (the convention already used elsewhere in this provider) so a fresh re-launch clears the race.

Note: if setsid forks in a real SSH session too, $! would be the wrong PID there and on_kill could orphan the job — a possible latent issue in the wrapper itself, left for a separate change.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_kill_terminates_whole_job_tree intermittently fails its pre-kill check on CI: the wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real group is empty by the time pgrep -g runs. Rerun on a fresh draw so the environment-dependent race does not fail unrelated PRs.

test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - #69384 had them on the sibling test until #69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…pache#70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

shahar1 · 2026-07-04T15:28:10Z

TestPosixKillBehaviour::test_kill_terminates_whole_job_tree (added in #68644) intermittently fails its pre-kill assertion on CI:

AssertionError: job tree should be running before kill
assert False = _group_alive(326)

The wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real process group is already empty by the time pgrep -g runs — failing the check before the kill is even exercised. Passes locally (15/15, and under xdist), fails only on certain CI runners, so it's environment-dependent and lands on unrelated PRs.

Marks the test with @pytest.mark.flaky(reruns=5) (the convention already used elsewhere in this provider) so a fresh re-launch clears the race.

Note: if setsid forks in a real SSH session too, $! would be the wrong PID there and on_kill could orphan the job — a possible latent issue in the wrapper itself, left for a separate change.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_kill_terminates_whole_job_tree intermittently fails its pre-kill check on CI: the wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real group is empty by the time pgrep -g runs. Rerun on a fresh draw so the environment-dependent race does not fail unrelated PRs.

test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - #69384 had them on the sibling test until #69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…pache#70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

shahar1 · 2026-07-04T15:28:10Z

TestPosixKillBehaviour::test_kill_terminates_whole_job_tree (added in #68644) intermittently fails its pre-kill assertion on CI:

AssertionError: job tree should be running before kill
assert False = _group_alive(326)

The wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real process group is already empty by the time pgrep -g runs — failing the check before the kill is even exercised. Passes locally (15/15, and under xdist), fails only on certain CI runners, so it's environment-dependent and lands on unrelated PRs.

Marks the test with @pytest.mark.flaky(reruns=5) (the convention already used elsewhere in this provider) so a fresh re-launch clears the race.

Note: if setsid forks in a real SSH session too, $! would be the wrong PID there and on_kill could orphan the job — a possible latent issue in the wrapper itself, left for a separate change.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_kill_terminates_whole_job_tree intermittently fails its pre-kill check on CI: the wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real group is empty by the time pgrep -g runs. Rerun on a fresh draw so the environment-dependent race does not fail unrelated PRs.

test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - #69384 had them on the sibling test until #69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…pache#70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

shahar1 · 2026-07-04T15:28:10Z

TestPosixKillBehaviour::test_kill_terminates_whole_job_tree (added in #68644) intermittently fails its pre-kill assertion on CI:

AssertionError: job tree should be running before kill
assert False = _group_alive(326)

The wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real process group is already empty by the time pgrep -g runs — failing the check before the kill is even exercised. Passes locally (15/15, and under xdist), fails only on certain CI runners, so it's environment-dependent and lands on unrelated PRs.

Marks the test with @pytest.mark.flaky(reruns=5) (the convention already used elsewhere in this provider) so a fresh re-launch clears the race.

Note: if setsid forks in a real SSH session too, $! would be the wrong PID there and on_kill could orphan the job — a possible latent issue in the wrapper itself, left for a separate change.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_kill_terminates_whole_job_tree intermittently fails its pre-kill check on CI: the wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real group is empty by the time pgrep -g runs. Rerun on a fresh draw so the environment-dependent race does not fail unrelated PRs.

test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - #69384 had them on the sibling test until #69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…pache#70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

shahar1 · 2026-07-04T15:28:10Z

TestPosixKillBehaviour::test_kill_terminates_whole_job_tree (added in #68644) intermittently fails its pre-kill assertion on CI:

AssertionError: job tree should be running before kill
assert False = _group_alive(326)

The wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real process group is already empty by the time pgrep -g runs — failing the check before the kill is even exercised. Passes locally (15/15, and under xdist), fails only on certain CI runners, so it's environment-dependent and lands on unrelated PRs.

Marks the test with @pytest.mark.flaky(reruns=5) (the convention already used elsewhere in this provider) so a fresh re-launch clears the race.

Note: if setsid forks in a real SSH session too, $! would be the wrong PID there and on_kill could orphan the job — a possible latent issue in the wrapper itself, left for a separate change.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_kill_terminates_whole_job_tree intermittently fails its pre-kill check on CI: the wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real group is empty by the time pgrep -g runs. Rerun on a fresh draw so the environment-dependent race does not fail unrelated PRs.

test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - #69384 had them on the sibling test until #69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…pache#70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

shahar1 · 2026-07-04T15:28:10Z

TestPosixKillBehaviour::test_kill_terminates_whole_job_tree (added in #68644) intermittently fails its pre-kill assertion on CI:

AssertionError: job tree should be running before kill
assert False = _group_alive(326)

The wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real process group is already empty by the time pgrep -g runs — failing the check before the kill is even exercised. Passes locally (15/15, and under xdist), fails only on certain CI runners, so it's environment-dependent and lands on unrelated PRs.

Marks the test with @pytest.mark.flaky(reruns=5) (the convention already used elsewhere in this provider) so a fresh re-launch clears the race.

Note: if setsid forks in a real SSH session too, $! would be the wrong PID there and on_kill could orphan the job — a possible latent issue in the wrapper itself, left for a separate change.

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Opus 4.8)

Generated-by: Claude Code (Opus 4.8) following the guidelines

test_kill_terminates_whole_job_tree intermittently fails its pre-kill check on CI: the wrapper records $! expecting it to equal the job PGID, but setsid only skips forking when the launching shell is not a process-group leader. On some runners it forks, so $! is the short-lived setsid parent and the job's real group is empty by the time pgrep -g runs. Rerun on a fresh draw so the environment-dependent race does not fail unrelated PRs.

test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - #69384 had them on the sibling test until #69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

…pache#70562) test_kill_terminates_whole_job_tree_under_job_control failed on main with "job never wrote its pid file". The run took 5.16s, so SUBMIT_DONE arrived promptly and the whole budget went on polling for a pid file that never appeared. That symptom has two very different causes and the test cannot tell them apart: _run_bash_mc_under_pty returns silently when the marker never arrives, so a launcher that died immediately (EOF, no marker, returns at once) and a job that died after being launched both surface later as the same empty pid file - and both produce the same ~5s runtime. Two changes separate them: - assert the marker was actually seen, quoting what the pty did produce - report the job directory contents when the pid file stays empty; the wrapper creates that directory and the log file before it backgrounds anything, so a missing directory means the launcher never got there and an empty one means the job was launched and died before its first statement The teardown also has a real ordering hazard, closed here: the marker only says the launcher returned, which it does the moment it backgrounds setsid - before that child has forked, called setsid(2) and exec'd into the job. Closing the pty master hangs up the terminal, and pty.fork() makes the launcher the session leader, so hanging up inside that window could take the job down with the session. The pty is now held open until the job proves it left the session by recording its own pid. I could not reproduce the failure. macOS has no setsid(1) so the class skips there; under Linux in Docker the real wrapper ran through this exact harness 65 times, 25 idle and 40 with the container throttled to 0.35 CPU against six busy loops, and recorded its pid every time with and without the hold. So the hold is hardening, not a demonstrated fix. Reruns come back for that reason - apache#69384 had them on the sibling test until apache#69490 dropped the marker while writing this one - and the first attempt's assertion text still reaches the CI log, so the next occurrence should identify which half broke. Generated-by: Claude Opus 5 (1M context)

shahar1 requested a review from kaxil July 4, 2026 15:28

boring-cyborgBot added area:providers provider:ssh labels Jul 4, 2026

potiuk approved these changes Jul 5, 2026
View reviewed changes

potiuk merged commit ea10f4f into apache:mainJul 5, 2026
81 checks passed

shahar1 deleted the fix-flaky-ssh-remote-job-kill-test branch July 5, 2026 20:35

kaxil mentioned this pull request Jul 6, 2026
Fix SSHRemoteJobOperator still orphaning the remote job on cancellation #69490
Merged

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

potiuk mentioned this pull request Jul 27, 2026
Make SSH remote-job kill test diagnosable and restore its reruns #70562
Merged
1 task

shahar1 requested a review from kaxil July 4, 2026 15:28

boring-cyborgBot added area:providers provider:ssh labels Jul 4, 2026

potiuk approved these changes Jul 5, 2026
View reviewed changes

potiuk merged commit ea10f4f into apache:mainJul 5, 2026
81 checks passed

shahar1 deleted the fix-flaky-ssh-remote-job-kill-test branch July 5, 2026 20:35

kaxil mentioned this pull request Jul 6, 2026
Fix SSHRemoteJobOperator still orphaning the remote job on cancellation #69490
Merged

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

potiuk mentioned this pull request Jul 27, 2026
Make SSH remote-job kill test diagnosable and restore its reruns #70562
Merged
1 task

shahar1 requested a review from kaxil July 4, 2026 15:28

boring-cyborgBot added area:providers provider:ssh labels Jul 4, 2026

potiuk approved these changes Jul 5, 2026
View reviewed changes

potiuk merged commit ea10f4f into apache:mainJul 5, 2026
81 checks passed

shahar1 deleted the fix-flaky-ssh-remote-job-kill-test branch July 5, 2026 20:35

kaxil mentioned this pull request Jul 6, 2026
Fix SSHRemoteJobOperator still orphaning the remote job on cancellation #69490
Merged

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

potiuk mentioned this pull request Jul 27, 2026
Make SSH remote-job kill test diagnosable and restore its reruns #70562
Merged
1 task

shahar1 requested a review from kaxil July 4, 2026 15:28

boring-cyborgBot added area:providers provider:ssh labels Jul 4, 2026

potiuk approved these changes Jul 5, 2026
View reviewed changes

potiuk merged commit ea10f4f into apache:mainJul 5, 2026
81 checks passed

shahar1 deleted the fix-flaky-ssh-remote-job-kill-test branch July 5, 2026 20:35

kaxil mentioned this pull request Jul 6, 2026
Fix SSHRemoteJobOperator still orphaning the remote job on cancellation #69490
Merged

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

potiuk mentioned this pull request Jul 27, 2026
Make SSH remote-job kill test diagnosable and restore its reruns #70562
Merged
1 task

shahar1 requested a review from kaxil July 4, 2026 15:28

boring-cyborgBot added area:providers provider:ssh labels Jul 4, 2026

potiuk approved these changes Jul 5, 2026
View reviewed changes

potiuk merged commit ea10f4f into apache:mainJul 5, 2026
81 checks passed

shahar1 deleted the fix-flaky-ssh-remote-job-kill-test branch July 5, 2026 20:35

kaxil mentioned this pull request Jul 6, 2026
Fix SSHRemoteJobOperator still orphaning the remote job on cancellation #69490
Merged

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

potiuk mentioned this pull request Jul 27, 2026
Make SSH remote-job kill test diagnosable and restore its reruns #70562
Merged
1 task

shahar1 requested a review from kaxil July 4, 2026 15:28

boring-cyborgBot added area:providers provider:ssh labels Jul 4, 2026

potiuk approved these changes Jul 5, 2026
View reviewed changes

potiuk merged commit ea10f4f into apache:mainJul 5, 2026
81 checks passed

shahar1 deleted the fix-flaky-ssh-remote-job-kill-test branch July 5, 2026 20:35

kaxil mentioned this pull request Jul 6, 2026
Fix SSHRemoteJobOperator still orphaning the remote job on cancellation #69490
Merged

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

potiuk mentioned this pull request Jul 27, 2026
Make SSH remote-job kill test diagnosable and restore its reruns #70562
Merged
1 task

shahar1 requested a review from kaxil July 4, 2026 15:28

boring-cyborgBot added area:providers provider:ssh labels Jul 4, 2026

potiuk approved these changes Jul 5, 2026
View reviewed changes

potiuk merged commit ea10f4f into apache:mainJul 5, 2026
81 checks passed

shahar1 deleted the fix-flaky-ssh-remote-job-kill-test branch July 5, 2026 20:35

kaxil mentioned this pull request Jul 6, 2026
Fix SSHRemoteJobOperator still orphaning the remote job on cancellation #69490
Merged

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

potiuk mentioned this pull request Jul 27, 2026
Make SSH remote-job kill test diagnosable and restore its reruns #70562
Merged
1 task

shahar1 requested a review from kaxil July 4, 2026 15:28

boring-cyborgBot added area:providers provider:ssh labels Jul 4, 2026

potiuk approved these changes Jul 5, 2026
View reviewed changes

potiuk merged commit ea10f4f into apache:mainJul 5, 2026
81 checks passed

shahar1 deleted the fix-flaky-ssh-remote-job-kill-test branch July 5, 2026 20:35

kaxil mentioned this pull request Jul 6, 2026
Fix SSHRemoteJobOperator still orphaning the remote job on cancellation #69490
Merged

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

potiuk mentioned this pull request Jul 27, 2026
Make SSH remote-job kill test diagnosable and restore its reruns #70562
Merged
1 task

Uh oh!

Conversation

shahar1 commented Jul 4, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shahar1 commented Jul 4, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shahar1 commented Jul 4, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shahar1 commented Jul 4, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shahar1 commented Jul 4, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

shahar1 commented Jul 4, 2026

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects