Reset Dynamo at Setup for all tests

[mcr229]

https://github.com/pytorch/executorch/actions/runs/14047575373/job/39331644423

There seems to be some CI issues with:

torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing

To help resolve this we reset dynamo at setup for all unittests. Let's see if this helps

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9561

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 5417019 with merge base 5c5b84e ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

any side effects from dynamo? Also I guess stuff like inheritance or decorator is just confusing?

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mergennachin]

why is this recompiling in the first place?

[mcr229]

why is this recompiling in the first place?

We have a for loop exporting the same model for different configurations multiple times. I think the issue lies there where we are "re-exporting" the same model. So it seems like we need to reset dynamo after each export. It is strange though because in each loop we are technically initializing a new module, I'm guessing there is some internal cache in dynamo that recognizes its the same model structure, and tries to recompile instead.

[GregoryComer]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

[mergennachin]

@tugsbayasgalan do you know if this is the right change?

[tugsbayasgalan]

@tugsbayasgalan do you know if this is the right change?

I am not really sure, but i know torchdynamo test cases do the same thing due to some caching stuff. Maybe cc: @anijain2305 knows better

[mcr229]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

yea i was thinking that, but i didn't want to add another abstraction we maintain just for _dynamo.reset(). I guess in the future could be useful.

[mcr229]

trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (push)

seems to be flaky and have intermittent failures. This PR likely doesn't have anything to do with that failure. Will follow up on that flaky test after landing this PR.

[mcr229]

https://github.com/pytorch/executorch/actions/runs/14047575373/job/39331644423

There seems to be some CI issues with:

torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing

To help resolve this we reset dynamo at setup for all unittests. Let's see if this helps

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9561

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 5417019 with merge base 5c5b84e ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

any side effects from dynamo? Also I guess stuff like inheritance or decorator is just confusing?

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mergennachin]

why is this recompiling in the first place?

[mcr229]

why is this recompiling in the first place?

We have a for loop exporting the same model for different configurations multiple times. I think the issue lies there where we are "re-exporting" the same model. So it seems like we need to reset dynamo after each export. It is strange though because in each loop we are technically initializing a new module, I'm guessing there is some internal cache in dynamo that recognizes its the same model structure, and tries to recompile instead.

[GregoryComer]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

[mergennachin]

@tugsbayasgalan do you know if this is the right change?

[tugsbayasgalan]

@tugsbayasgalan do you know if this is the right change?

I am not really sure, but i know torchdynamo test cases do the same thing due to some caching stuff. Maybe cc: @anijain2305 knows better

[mcr229]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

yea i was thinking that, but i didn't want to add another abstraction we maintain just for _dynamo.reset(). I guess in the future could be useful.

[mcr229]

trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (push)

seems to be flaky and have intermittent failures. This PR likely doesn't have anything to do with that failure. Will follow up on that flaky test after landing this PR.

[mcr229]

https://github.com/pytorch/executorch/actions/runs/14047575373/job/39331644423

There seems to be some CI issues with:

torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing

To help resolve this we reset dynamo at setup for all unittests. Let's see if this helps

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9561

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 5417019 with merge base 5c5b84e ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

any side effects from dynamo? Also I guess stuff like inheritance or decorator is just confusing?

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mergennachin]

why is this recompiling in the first place?

[mcr229]

why is this recompiling in the first place?

We have a for loop exporting the same model for different configurations multiple times. I think the issue lies there where we are "re-exporting" the same model. So it seems like we need to reset dynamo after each export. It is strange though because in each loop we are technically initializing a new module, I'm guessing there is some internal cache in dynamo that recognizes its the same model structure, and tries to recompile instead.

[GregoryComer]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

[mergennachin]

@tugsbayasgalan do you know if this is the right change?

[tugsbayasgalan]

@tugsbayasgalan do you know if this is the right change?

I am not really sure, but i know torchdynamo test cases do the same thing due to some caching stuff. Maybe cc: @anijain2305 knows better

[mcr229]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

yea i was thinking that, but i didn't want to add another abstraction we maintain just for _dynamo.reset(). I guess in the future could be useful.

[mcr229]

trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (push)

seems to be flaky and have intermittent failures. This PR likely doesn't have anything to do with that failure. Will follow up on that flaky test after landing this PR.

[mcr229]

https://github.com/pytorch/executorch/actions/runs/14047575373/job/39331644423

There seems to be some CI issues with:

torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing

To help resolve this we reset dynamo at setup for all unittests. Let's see if this helps

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9561

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 5417019 with merge base 5c5b84e ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

any side effects from dynamo? Also I guess stuff like inheritance or decorator is just confusing?

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mergennachin]

why is this recompiling in the first place?

[mcr229]

why is this recompiling in the first place?

We have a for loop exporting the same model for different configurations multiple times. I think the issue lies there where we are "re-exporting" the same model. So it seems like we need to reset dynamo after each export. It is strange though because in each loop we are technically initializing a new module, I'm guessing there is some internal cache in dynamo that recognizes its the same model structure, and tries to recompile instead.

[GregoryComer]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

[mergennachin]

@tugsbayasgalan do you know if this is the right change?

[tugsbayasgalan]

@tugsbayasgalan do you know if this is the right change?

I am not really sure, but i know torchdynamo test cases do the same thing due to some caching stuff. Maybe cc: @anijain2305 knows better

[mcr229]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

yea i was thinking that, but i didn't want to add another abstraction we maintain just for _dynamo.reset(). I guess in the future could be useful.

[mcr229]

trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (push)

seems to be flaky and have intermittent failures. This PR likely doesn't have anything to do with that failure. Will follow up on that flaky test after landing this PR.

[mcr229]

https://github.com/pytorch/executorch/actions/runs/14047575373/job/39331644423

There seems to be some CI issues with:

torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing

To help resolve this we reset dynamo at setup for all unittests. Let's see if this helps

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9561

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 5417019 with merge base 5c5b84e ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

any side effects from dynamo? Also I guess stuff like inheritance or decorator is just confusing?

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mergennachin]

why is this recompiling in the first place?

[mcr229]

why is this recompiling in the first place?

We have a for loop exporting the same model for different configurations multiple times. I think the issue lies there where we are "re-exporting" the same model. So it seems like we need to reset dynamo after each export. It is strange though because in each loop we are technically initializing a new module, I'm guessing there is some internal cache in dynamo that recognizes its the same model structure, and tries to recompile instead.

[GregoryComer]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

[mergennachin]

@tugsbayasgalan do you know if this is the right change?

[tugsbayasgalan]

@tugsbayasgalan do you know if this is the right change?

I am not really sure, but i know torchdynamo test cases do the same thing due to some caching stuff. Maybe cc: @anijain2305 knows better

[mcr229]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

yea i was thinking that, but i didn't want to add another abstraction we maintain just for _dynamo.reset(). I guess in the future could be useful.

[mcr229]

trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (push)

seems to be flaky and have intermittent failures. This PR likely doesn't have anything to do with that failure. Will follow up on that flaky test after landing this PR.

[mcr229]

https://github.com/pytorch/executorch/actions/runs/14047575373/job/39331644423

There seems to be some CI issues with:

torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing

To help resolve this we reset dynamo at setup for all unittests. Let's see if this helps

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9561

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 5417019 with merge base 5c5b84e ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

any side effects from dynamo? Also I guess stuff like inheritance or decorator is just confusing?

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mergennachin]

why is this recompiling in the first place?

[mcr229]

why is this recompiling in the first place?

We have a for loop exporting the same model for different configurations multiple times. I think the issue lies there where we are "re-exporting" the same model. So it seems like we need to reset dynamo after each export. It is strange though because in each loop we are technically initializing a new module, I'm guessing there is some internal cache in dynamo that recognizes its the same model structure, and tries to recompile instead.

[GregoryComer]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

[mergennachin]

@tugsbayasgalan do you know if this is the right change?

[tugsbayasgalan]

@tugsbayasgalan do you know if this is the right change?

I am not really sure, but i know torchdynamo test cases do the same thing due to some caching stuff. Maybe cc: @anijain2305 knows better

[mcr229]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

yea i was thinking that, but i didn't want to add another abstraction we maintain just for _dynamo.reset(). I guess in the future could be useful.

[mcr229]

trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (push)

seems to be flaky and have intermittent failures. This PR likely doesn't have anything to do with that failure. Will follow up on that flaky test after landing this PR.

[mcr229]

https://github.com/pytorch/executorch/actions/runs/14047575373/job/39331644423

There seems to be some CI issues with:

torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing

To help resolve this we reset dynamo at setup for all unittests. Let's see if this helps

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9561

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 5417019 with merge base 5c5b84e ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

any side effects from dynamo? Also I guess stuff like inheritance or decorator is just confusing?

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mergennachin]

why is this recompiling in the first place?

[mcr229]

why is this recompiling in the first place?

We have a for loop exporting the same model for different configurations multiple times. I think the issue lies there where we are "re-exporting" the same model. So it seems like we need to reset dynamo after each export. It is strange though because in each loop we are technically initializing a new module, I'm guessing there is some internal cache in dynamo that recognizes its the same model structure, and tries to recompile instead.

[GregoryComer]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

[mergennachin]

@tugsbayasgalan do you know if this is the right change?

[tugsbayasgalan]

@tugsbayasgalan do you know if this is the right change?

I am not really sure, but i know torchdynamo test cases do the same thing due to some caching stuff. Maybe cc: @anijain2305 knows better

[mcr229]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

yea i was thinking that, but i didn't want to add another abstraction we maintain just for _dynamo.reset(). I guess in the future could be useful.

[mcr229]

trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (push)

seems to be flaky and have intermittent failures. This PR likely doesn't have anything to do with that failure. Will follow up on that flaky test after landing this PR.

[mcr229]

https://github.com/pytorch/executorch/actions/runs/14047575373/job/39331644423

There seems to be some CI issues with:

torch._dynamo.exc.FailOnRecompileLimitHit: recompile_limit reached with one_graph=True. Excessive recompilations can degrade performance due to the compilation overhead of each recompilation. To monitor recompilations, enable TORCH_LOGS=recompiles. If recompilations are expected, consider increasing

To help resolve this we reset dynamo at setup for all unittests. Let's see if this helps

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9561

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 5417019 with merge base 5c5b84e ():

NEW FAILURE - The following job has failed:

• trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

any side effects from dynamo? Also I guess stuff like inheritance or decorator is just confusing?

[facebook-github-bot]

@mcr229 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[mergennachin]

why is this recompiling in the first place?

[mcr229]

why is this recompiling in the first place?

We have a for loop exporting the same model for different configurations multiple times. I think the issue lies there where we are "re-exporting" the same model. So it seems like we need to reset dynamo after each export. It is strange though because in each loop we are technically initializing a new module, I'm guessing there is some internal cache in dynamo that recognizes its the same model structure, and tries to recompile instead.

[GregoryComer]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

[mergennachin]

@tugsbayasgalan do you know if this is the right change?

[tugsbayasgalan]

@tugsbayasgalan do you know if this is the right change?

I am not really sure, but i know torchdynamo test cases do the same thing due to some caching stuff. Maybe cc: @anijain2305 knows better

[mcr229]

Nit: It might be nice to define a common test fixture for models + ops. If this fixes it, maybe we can create a "good first issue"-tagged issue to get help doing the refactoring.

yea i was thinking that, but i didn't want to add another abstraction we maintain just for _dynamo.reset(). I guess in the future could be useful.

[mcr229]

trunk / test-models-macos (llama2, xnnpack-quantization-delegation) / macos-job (push)

seems to be flaky and have intermittent failures. This PR likely doesn't have anything to do with that failure. Will follow up on that flaky test after landing this PR.