Skip to content

Debug CI tests on Ada - #397

Merged
timmoon10 merged 27 commits into
NVIDIA:mainfrom
timmoon10:ada-ci-debug
Oct 12, 2023
Merged

Debug CI tests on Ada#397
timmoon10 merged 27 commits into
NVIDIA:mainfrom
timmoon10:ada-ci-debug

Conversation

@timmoon10

@timmoon10timmoon10 commented Aug 23, 2023

Copy link
Copy Markdown
Member

This applies the changes in #393 to the PyTorch and Paddle tests. In particular, it only runs tests involving cuDNN fused attention on compute capabilities 8.0 and 9.0.

@timmoon10
Debug PyTorch and Paddle tests on Ada
53991e9
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10timmoon10 added the bug Something isn't working label Aug 23, 2023
@timmoon10

timmoon10 commented Aug 23, 2023

Copy link
Copy Markdown
MemberAuthor

Pipeline 9489089

@timmoon10
Only run Paddle layer tests with cuDNN fMHA on supported archs
8b8c107
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10timmoon10 mentioned this pull request Aug 24, 2023
Merged
@timmoon10
Debug PyTorch fMHA tests
d171496
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10
Reduce JAX FP8 GEMM sizes
02eb19e
Avoid split-k kernels on Ada.
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10

timmoon10 commented Aug 24, 2023

Copy link
Copy Markdown
MemberAuthor

Running on an L40, I found that the JAX FP8 GEMM tests on integer matrices were failing. It seems cuBLAS chooses a split-k kernel that prevents us from getting bit-wise correct results, although it is still within the expected FP8 error. I've changed the matrix dims to help cuBLAS pick a nicer kernel.

Pipeline 9504876.

@timmoon10

timmoon10 commented Sep 6, 2023

Copy link
Copy Markdown
MemberAuthor

Pipeline 9617488 is green.

cyanguwa
cyanguwa requested changes Sep 12, 2023
View reviewed changes
Comment threadtests/pytorch/test_fused_attn.py Outdated
Comment threadtests/jax/test_custom_call_compute.py
Comment threadtests/jax/test_fused_attn.py Outdated
@timmoon10
Merge branch 'main' into ada-ci-debug
434bc2a
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10
Merge branch 'main' into ada-ci-debug
f021843
@timmoon10
Run supported fused attention tests on Ada
9320700
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10
Run supported fused attention JAX tests on Ada
b79b163
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10

timmoon10 commented Sep 23, 2023

Copy link
Copy Markdown
MemberAuthor

I've tweaked the PyTorch and JAX fused attention tests so we check if there's a supported backed (namely F16_arbitrary_seqlen on Ada). These pass when I run manually on an L40 and I've launched pipeline 9938409.

#403 adds some PyTorch attention tests and #411 adds backend detection logic to Paddle. We should hold off on merging until those are in.

@timmoon10

timmoon10 commented Oct 4, 2023•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

This PR is now good to go, pending pipeline 10094388 pipeline 70748350.

@timmoon10
Enable Paddle fused attention on Ada
0a864fc
Signed-off-by: Tim Moon <tmoon@nvidia.com>
cyanguwa
cyanguwa approved these changes Oct 5, 2023
View reviewed changes

@cyanguwacyanguwa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that one comment, otherwise looks good!

Comment threadtests/pytorch/test_fused_attn.py
@timmoon10timmoon10 mentioned this pull request Oct 7, 2023
Merged
@timmoon10
Restore backend support to reference FP8 attention impl in PyT test
24c8814
Review suggestion from @cyanguwa
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10
Merge branch 'main' into ada-ci-debug
2d9cd6b
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10
Fix merge conflicts
44ac1f8
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10
Debug Paddle tests on Ada
ae8af09
Signed-off-by: Tim Moon <tmoon@nvidia.com>
@timmoon10
Merge branch 'main' into ada-ci-debug
6726c30
@timmoon10
Loosen tolerances for Paddle attention tests
777e45b
Signed-off-by: Tim Moon <tmoon@nvidia.com>
timmoon10
timmoon10 commented Oct 11, 2023
View reviewed changes
Comment threadtests/paddle/test_operators.py Outdated
@timmoon10timmoon10 changed the title Debug PyTorch and Paddle tests on AdaDebug CI tests on AdaOct 11, 2023
@timmoon10

timmoon10 commented Oct 12, 2023

Copy link
Copy Markdown
MemberAuthor

Tests passed in pipeline 10211932.

ksivaman
ksivaman approved these changes Oct 12, 2023
View reviewed changes

@ksivamanksivaman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@timmoon10
timmoon10 merged commit 4ae3476 into NVIDIA:mainOct 12, 2023
@timmoon10
timmoon10 deleted the ada-ci-debug branch October 12, 2023 19:30
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cyanguwacyanguwacyanguwa approved these changes
@ksivamanksivamanksivaman approved these changes
+1 more reviewer
@zlsh80826zlsh80826zlsh80826 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bugSomething isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@timmoon10@cyanguwa@zlsh80826@ksivaman