[release/2.12] Fix hardcoded rendezvous ports in distributed tests (#195994)

[albmalamd]

Summary

Three paths in the distributed test infrastructure hardcode a rendezvous port, so any two distributed test processes on the same host contend for it. PythonProcessGroupExtensionTest in test_c10d_common.py and _dynamo_dist_per_rank_init in common_distributed.py both pin MASTER_PORT=6789, and DynamoDistributedSingleProcTestCase pins 12355. The _dynamo_dist_per_rank_init path is the widest: it backs roughly a hundred tests across test_dynamo_distributed.py, test_inductor_collectives.py, test_compute_comm_reordering.py, and test_aten_comm_compute_reordering.py. A sequential run never sees the collision. The ROCm full-suite executor runs 16 tasks in parallel, and the loser either dies with DistNetworkError ... port: 6789 ... EADDRINUSE or attaches to the other run's TCPStore and hangs until the 300s harness timeout.

Every path now rendezvouses over a file rather than a TCP port. PythonProcessGroupExtensionTest uses the per-test temp file that MultiProcessTestCase already creates and propagates to every rank, wrapped in dist.FileStore(self.file_name, self.world_size) and passed as store=; ProcessGroupWithDispatchedCollectivesTests in the same file and DistributedTestBase.create_pg already do exactly this, and the store= form sidesteps the win32 file:// URL quirk that _init_methods works around. _dynamo_dist_per_rank_init gains an rdvz_file keyword and builds its FileStore from that; its context manager has no access to the harness state, so each call site passes self.file_name explicitly. DynamoDistributedSingleProcTestCase allocates its own temp file in setUpClass and drops the MASTER_ADDR/MASTER_PORT patch entirely.

find_free_port was considered and rejected for both paths. It binds and closes, so another of the parallel tasks can still claim the port in the window before the store binds it, and keeping the reservation alive does not work either: TCPStore would have to bind that same port itself, and its master_listen_fd cannot be inherited through spawn under an env:// rendezvous. The file rendezvous has no such gap.

The env:// fallback in _dynamo_dist_per_rank_init is kept for callers that pass no rdvz_file, so out-of-tree users of the helper keep their current behavior. No in-tree caller takes it.

The nine PythonProcessGroupExtensionTest tests that rendezvous no longer exercise TCPStore. That is acceptable: they test the Python process-group extension API, and TCPStore has dedicated coverage elsewhere in the same file.

Test Plan

Reproduction requires concurrency; every affected test passes when run on its own. Which instance fails is nondeterministic, and a losing instance may report the 300s timeout rather than EADDRINUSE.

Concurrent instances of the test_c10d_common.py tests. Before the change one instance fails with EADDRINUSE on port 6789 while the other completes in ~336s because of the 300s hang; after it both pass in ~39s, and a four-way stress passes in every instance:

cd test/distributed
for i in 1 2 3 4; do python -m pytest test_c10d_common.py -q -k PythonProcessGroupExtensionTest & done; wait

Same for the dynamo path, on disjoint GPUs so the port is the only shared resource. Otherwise NCCL resource exhaustion from oversubscribing the GPUs masks the port conflict with a different failure. One instance fails with EADDRINUSE before the change; both pass after it:

cd test/distributed
HIP_VISIBLE_DEVICES=0,1 python -m pytest test_dynamo_distributed.py -q -k test_ddp_baseline_aot_eager_multiprocess &
HIP_VISIBLE_DEVICES=2,3 python -m pytest test_dynamo_distributed.py -q -k test_ddp_baseline_aot_eager_multiprocess &
wait

No regressions. On 8x MI300-class GPUs with 2.13.0+rocm10.0.0rc4, giving 32 passed, 11 passed, and 21 passed / 7 skipped respectively:

cd test/distributed
PYTORCH_TEST_WITH_ROCM=1 python -m pytest test_c10d_common.py -q
PYTORCH_TEST_WITH_ROCM=1 python -m pytest test_c10d_object_collectives.py test_c10d_logger.py -q
PYTORCH_TEST_WITH_ROCM=1 python -m pytest test_dynamo_distributed.py -q -k TestMultiProc

On 4x gfx90a with HIP 7.14, covering the rewritten single-process harness, giving 32 passed:

cd test/distributed
python -m pytest test_dynamo_distributed.py -q -k TestSingleProc

Authored with the assistance of an AI coding assistant.

Pull Request resolved: pytorch#195994
Approved by: https://github.com/jeffdaily

(cherry picked from commit 8b8a766)

Cherry-pick conflicts were limited to test bodies that exist on main but not on release/2.12 (test_overlap_scheduling_flex_attention_backward and the three test_manual_bucketing_*_with_intermediate_deps / test_manual_bucketing_ag_forward_and_backward tests in test_aten_comm_compute_reordering.py, and test_new_group_delegates_to_pg in test_c10d_common.py). Those hunks were dropped; every _dynamo_dist_per_rank_init call site that does exist on this branch takes rdvz_file, and the torch/testing/_internal/common_distributed.py change applies identically to the release/2.13 backport.