Skip to content

[FSDP] Stabilize fully_shard overlap timing test - #3592

Draft
albmalamd wants to merge 1 commit into
release/2.13from
fix_fsdp_overlap_timing_stability_release_2_13
Draft

albmalamd wants to merge 1 commit into
release/2.13from
fix_fsdp_overlap_timing_stability_release_2_13

Conversation

@albmalamd

Copy link
Copy Markdown

The overlap test emulates compute and communication with fixed-cycle GPU sleeps. DVFS and device contention can change their wall-clock duration between separate single-shot measurements, causing false failures under load.

Interleave reference and test measurements and compare their least-perturbed samples. This only improves timing stability; it does not relax assertions or tolerances, change emulated sleep durations, or add conditional retries or skips.

Test Plan:

python -m pytest test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap

Disappeared timing assertion flakiness

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

The overlap test emulates compute and communication with fixed-cycle GPU sleeps. DVFS and device contention can change their wall-clock duration between separate single-shot measurements, causing false failures under load.

Interleave reference and test measurements and compare their least-perturbed samples. This only improves timing stability; it does not relax assertions or tolerances, change emulated sleep durations, or add conditional retries or skips.

Test Plan:

```
python -m pytest test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap
```

Disappeared timing assertion flakiness
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant