Add export_llama performance regression test using expected ops by jackzhxng · Pull Request #9158 · pytorch/executorch

jackzhxng · 2025-03-11T20:29:42Z

Summary

Add a proxy for an export_llama performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base ModelArgs values for llama_transformer will be used.

Test plan

N/A

pytorch-bot · 2025-03-11T20:29:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 54cd638 with merge base 306b649 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tarun292 · 2025-03-18T22:56:06Z

+# Ops expected to be found in the default exported llama_transformer. Obtained through
+# print_delegation_info from the backend_debug module, which is displayed with
+# export_llama under --verbose.
+BASE_EXPECTED_OPS = {


I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.

tarun292 · 2025-03-19T00:01:41Z

+ # we cannot test quantization args in this way
+ # since quantization requires promoting meta tensors
+ # to the cpu device, which requires real weights.
+ export_args_str = """


Also why do this? Why not just generate the args directly and use them.

Oh I feel like it's more clear to read?

Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.

facebook-github-bot · 2025-03-22T20:53:27Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-23T01:58:40Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:21:22Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:30:21Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A

jackzhxng · 2025-03-11T20:29:42Z

Summary

Add a proxy for an export_llama performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base ModelArgs values for llama_transformer will be used.

Test plan

N/A

pytorch-bot · 2025-03-11T20:29:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 54cd638 with merge base 306b649 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tarun292 · 2025-03-18T22:56:06Z

+# Ops expected to be found in the default exported llama_transformer. Obtained through
+# print_delegation_info from the backend_debug module, which is displayed with
+# export_llama under --verbose.
+BASE_EXPECTED_OPS = {


I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.

tarun292 · 2025-03-19T00:01:41Z

+ # we cannot test quantization args in this way
+ # since quantization requires promoting meta tensors
+ # to the cpu device, which requires real weights.
+ export_args_str = """


Also why do this? Why not just generate the args directly and use them.

Oh I feel like it's more clear to read?

Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.

facebook-github-bot · 2025-03-22T20:53:27Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-23T01:58:40Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:21:22Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:30:21Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A

jackzhxng · 2025-03-11T20:29:42Z

Summary

Add a proxy for an export_llama performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base ModelArgs values for llama_transformer will be used.

Test plan

N/A

pytorch-bot · 2025-03-11T20:29:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 54cd638 with merge base 306b649 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tarun292 · 2025-03-18T22:56:06Z

+# Ops expected to be found in the default exported llama_transformer. Obtained through
+# print_delegation_info from the backend_debug module, which is displayed with
+# export_llama under --verbose.
+BASE_EXPECTED_OPS = {


I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.

tarun292 · 2025-03-19T00:01:41Z

+ # we cannot test quantization args in this way
+ # since quantization requires promoting meta tensors
+ # to the cpu device, which requires real weights.
+ export_args_str = """


Also why do this? Why not just generate the args directly and use them.

Oh I feel like it's more clear to read?

Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.

facebook-github-bot · 2025-03-22T20:53:27Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-23T01:58:40Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:21:22Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:30:21Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A

jackzhxng · 2025-03-11T20:29:42Z

Summary

Add a proxy for an export_llama performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base ModelArgs values for llama_transformer will be used.

Test plan

N/A

pytorch-bot · 2025-03-11T20:29:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 54cd638 with merge base 306b649 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tarun292 · 2025-03-18T22:56:06Z

+# Ops expected to be found in the default exported llama_transformer. Obtained through
+# print_delegation_info from the backend_debug module, which is displayed with
+# export_llama under --verbose.
+BASE_EXPECTED_OPS = {


I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.

tarun292 · 2025-03-19T00:01:41Z

+ # we cannot test quantization args in this way
+ # since quantization requires promoting meta tensors
+ # to the cpu device, which requires real weights.
+ export_args_str = """


Also why do this? Why not just generate the args directly and use them.

Oh I feel like it's more clear to read?

Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.

facebook-github-bot · 2025-03-22T20:53:27Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-23T01:58:40Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:21:22Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:30:21Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A

jackzhxng · 2025-03-11T20:29:42Z

Summary

Add a proxy for an export_llama performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base ModelArgs values for llama_transformer will be used.

Test plan

N/A

pytorch-bot · 2025-03-11T20:29:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 54cd638 with merge base 306b649 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tarun292 · 2025-03-18T22:56:06Z

+# Ops expected to be found in the default exported llama_transformer. Obtained through
+# print_delegation_info from the backend_debug module, which is displayed with
+# export_llama under --verbose.
+BASE_EXPECTED_OPS = {


I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.

tarun292 · 2025-03-19T00:01:41Z

+ # we cannot test quantization args in this way
+ # since quantization requires promoting meta tensors
+ # to the cpu device, which requires real weights.
+ export_args_str = """


Also why do this? Why not just generate the args directly and use them.

Oh I feel like it's more clear to read?

Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.

facebook-github-bot · 2025-03-22T20:53:27Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-23T01:58:40Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:21:22Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:30:21Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A

jackzhxng · 2025-03-11T20:29:42Z

Summary

Add a proxy for an export_llama performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base ModelArgs values for llama_transformer will be used.

Test plan

N/A

pytorch-bot · 2025-03-11T20:29:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 54cd638 with merge base 306b649 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tarun292 · 2025-03-18T22:56:06Z

+# Ops expected to be found in the default exported llama_transformer. Obtained through
+# print_delegation_info from the backend_debug module, which is displayed with
+# export_llama under --verbose.
+BASE_EXPECTED_OPS = {


I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.

tarun292 · 2025-03-19T00:01:41Z

+ # we cannot test quantization args in this way
+ # since quantization requires promoting meta tensors
+ # to the cpu device, which requires real weights.
+ export_args_str = """


Also why do this? Why not just generate the args directly and use them.

Oh I feel like it's more clear to read?

Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.

facebook-github-bot · 2025-03-22T20:53:27Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-23T01:58:40Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:21:22Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:30:21Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A

jackzhxng · 2025-03-11T20:29:42Z

Summary

Add a proxy for an export_llama performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base ModelArgs values for llama_transformer will be used.

Test plan

N/A

pytorch-bot · 2025-03-11T20:29:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 54cd638 with merge base 306b649 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tarun292 · 2025-03-18T22:56:06Z

+# Ops expected to be found in the default exported llama_transformer. Obtained through
+# print_delegation_info from the backend_debug module, which is displayed with
+# export_llama under --verbose.
+BASE_EXPECTED_OPS = {


I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.

tarun292 · 2025-03-19T00:01:41Z

+ # we cannot test quantization args in this way
+ # since quantization requires promoting meta tensors
+ # to the cpu device, which requires real weights.
+ export_args_str = """


Also why do this? Why not just generate the args directly and use them.

Oh I feel like it's more clear to read?

Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.

facebook-github-bot · 2025-03-22T20:53:27Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-23T01:58:40Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:21:22Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:30:21Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A

jackzhxng · 2025-03-11T20:29:42Z

Summary

Add a proxy for an export_llama performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base ModelArgs values for llama_transformer will be used.

Test plan

N/A

pytorch-bot · 2025-03-11T20:29:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9158

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 54cd638 with merge base 306b649 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tarun292 · 2025-03-18T22:56:06Z

+# Ops expected to be found in the default exported llama_transformer. Obtained through
+# print_delegation_info from the backend_debug module, which is displayed with
+# export_llama under --verbose.
+BASE_EXPECTED_OPS = {


I think this might be a little risky to test. If the IR changes which is a common possibility this count will change and we'll have to keep fixing this.

tarun292 · 2025-03-19T00:01:41Z

+ # we cannot test quantization args in this way
+ # since quantization requires promoting meta tensors
+ # to the cpu device, which requires real weights.
+ export_args_str = """


Also why do this? Why not just generate the args directly and use them.

Oh I feel like it's more clear to read?

Hmm generally it's not good practice to hard code strings like this as they'll tend to change. The underlying args they refer to tend to be more stable so i'd say switch over to the args directly.

facebook-github-bot · 2025-03-22T20:53:27Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-23T01:58:40Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:21:22Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2025-03-26T09:30:21Z

@jackzhxng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

### Summary Add a proxy for an `export_llama` performance regression test by comparing the ops in the graph before and after the PR. The export happens without loading a checkpoint or params file, which means that all of the base `ModelArgs` values for `llama_transformer` will be used. ### Test plan N/A

jackzhxng added 3 commits March 10, 2025 18:17

Fix pre-autograd transforms not getting persisted during xnnpack export
affe92d

Graph module as SOT
9978148

Add perf proxy regression test using expected ops
537458b

jackzhxng requested a review from kimishpatel March 11, 2025 20:29

jackzhxng requested a review from lucylq as a code owner March 11, 2025 20:29

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 11, 2025

Base automatically changed from jz/fix-regression to mainMarch 11, 2025 21:06

jackzhxng requested review from iseeyuan, larryliu0820 and swolchok as code owners March 11, 2025 21:06

swolchok removed their request for review March 11, 2025 21:22

Merge branch 'main' into jz/regression-test-llama-perf
c5ed931

jackzhxng added the release notes: examples Changes to any of our example LLMs integrations, such as Llama3 and Llava label Mar 13, 2025

tarun292 reviewed Mar 18, 2025
View reviewed changes

Only check unwanted ops
f125d74

tarun292 reviewed Mar 19, 2025
View reviewed changes

tarun292 approved these changes Mar 19, 2025
View reviewed changes

Merge branch 'main' into jz/regression-test-llama-perf
945a203

jackzhxng force-pushed the jz/regression-test-llama-perf branch from 738cdf0 to d3d8e7dCompare March 22, 2025 22:50

Tarun pr rev / fix test
785904b

jackzhxng force-pushed the jz/regression-test-llama-perf branch from d3d8e7d to 785904bCompare March 23, 2025 01:54

jackzhxng added 6 commits March 24, 2025 02:13

Fix test
2f18046

Lint
e11ddec

Revert model args default changes
792c295

Merge branch 'main' into jz/regression-test-llama-perf
134fd29

Merge branch 'main' into jz/regression-test-llama-perf
4752fa6

Test
cdda3cc

jackzhxng added 3 commits March 24, 2025 17:04

Merge branch 'main' into jz/regression-test-llama-perf
2b642a1

Try fix test
3887bbe

Merge branch 'main' into jz/regression-test-llama-perf
8ca7e05

jackzhxng force-pushed the jz/regression-test-llama-perf branch from c963335 to 8ca7e05Compare March 25, 2025 15:10

jackzhxng added 2 commits March 26, 2025 00:44

Merge branch 'main' into jz/regression-test-llama-perf
68e5722

Fix Llava
ebfa760

Small change to retrigger meta internal sync CI
54cd638

jackzhxng merged commit 2f65c3a into mainMar 26, 2025

jackzhxng deleted the jz/regression-test-llama-perf branch March 26, 2025 13:07