Aot compiler fix

[mcr229]

Summary

Changes:

1. When initializing Llama2 for aot_compiler, since checkpoints can only e downloaded from hugging face, we initialize llama2 with uninitialized weights. The problem with this is that when running quantization, we can run into errors with the histogram if the unitialized values are nan. We fix this by initializing the weights with zeros if no check point is provided. This enforces that quantization step can still work.
2. Quant Type in AoT compiler. When looking at the model options available to XNNPACK, everything is quantized with per-tensor static quantization. This isn't the best option for all the models available. For example transformer based models like Llama and MobileBert would likely prefer dynamically quantized per channel weights, where has CNN like MobileNet would prefer statically quantized per channel weights. We add this type of Quant Type to the existing models options. This also helps with Test Timeouts. per-tensor static quantization on a model like llama can take a long time due to the introduction of MANY q/dq nodes, and the complex partitions it creates. As a result, proposing partitions can take a long time due to the constant BFS to find the largest possible partition. By specifying the more apt quantization scheme like dynamic per-channel quantization, we can avoid this complexity.

Overall this should help with flakey [nan, nan] errors in the quantization histogram, and it should also help with CI timing out.

Test plan

OSS XNNPACK CI for all model delegation

cc @digantdesai@cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9634

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit ccf664e with merge base 4b8ac94 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-linux-aarch64 (phi_4_mini, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t fe2f1f663ff952269e03a3d674e379f9abda3381abdf9d58218ccb94f5758da3 /exec failed with exit code 137

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Great!

[mcr229]

phi_4_mini ci tests are failing with:

.ci/scripts/test_model.sh: line 75: 30893 Killed "${PYTHON_EXECUTABLE}" -m examples.models.llama.export_llama --model "${MODEL_NAME}" -c examples/models/llama/params/demo_rand_params.pth -p examples/models/phi_4_mini/config.json

there is no error message with the run, so I don't think anything is failing but just that this tests is getting killed. Running it locally on my laptop seems to pass

[jackzhxng]

Seems fine, doesn't seem like it should have affected the phi test. If it's consistently getting killed it might be OOMing

[mcr229]

Summary

Changes:

1. When initializing Llama2 for aot_compiler, since checkpoints can only e downloaded from hugging face, we initialize llama2 with uninitialized weights. The problem with this is that when running quantization, we can run into errors with the histogram if the unitialized values are nan. We fix this by initializing the weights with zeros if no check point is provided. This enforces that quantization step can still work.
2. Quant Type in AoT compiler. When looking at the model options available to XNNPACK, everything is quantized with per-tensor static quantization. This isn't the best option for all the models available. For example transformer based models like Llama and MobileBert would likely prefer dynamically quantized per channel weights, where has CNN like MobileNet would prefer statically quantized per channel weights. We add this type of Quant Type to the existing models options. This also helps with Test Timeouts. per-tensor static quantization on a model like llama can take a long time due to the introduction of MANY q/dq nodes, and the complex partitions it creates. As a result, proposing partitions can take a long time due to the constant BFS to find the largest possible partition. By specifying the more apt quantization scheme like dynamic per-channel quantization, we can avoid this complexity.

Overall this should help with flakey [nan, nan] errors in the quantization histogram, and it should also help with CI timing out.

Test plan

OSS XNNPACK CI for all model delegation

cc @digantdesai@cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9634

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit ccf664e with merge base 4b8ac94 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-linux-aarch64 (phi_4_mini, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t fe2f1f663ff952269e03a3d674e379f9abda3381abdf9d58218ccb94f5758da3 /exec failed with exit code 137

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Great!

[mcr229]

phi_4_mini ci tests are failing with:

.ci/scripts/test_model.sh: line 75: 30893 Killed "${PYTHON_EXECUTABLE}" -m examples.models.llama.export_llama --model "${MODEL_NAME}" -c examples/models/llama/params/demo_rand_params.pth -p examples/models/phi_4_mini/config.json

there is no error message with the run, so I don't think anything is failing but just that this tests is getting killed. Running it locally on my laptop seems to pass

[jackzhxng]

Seems fine, doesn't seem like it should have affected the phi test. If it's consistently getting killed it might be OOMing

[mcr229]

Summary

Changes:

1. When initializing Llama2 for aot_compiler, since checkpoints can only e downloaded from hugging face, we initialize llama2 with uninitialized weights. The problem with this is that when running quantization, we can run into errors with the histogram if the unitialized values are nan. We fix this by initializing the weights with zeros if no check point is provided. This enforces that quantization step can still work.
2. Quant Type in AoT compiler. When looking at the model options available to XNNPACK, everything is quantized with per-tensor static quantization. This isn't the best option for all the models available. For example transformer based models like Llama and MobileBert would likely prefer dynamically quantized per channel weights, where has CNN like MobileNet would prefer statically quantized per channel weights. We add this type of Quant Type to the existing models options. This also helps with Test Timeouts. per-tensor static quantization on a model like llama can take a long time due to the introduction of MANY q/dq nodes, and the complex partitions it creates. As a result, proposing partitions can take a long time due to the constant BFS to find the largest possible partition. By specifying the more apt quantization scheme like dynamic per-channel quantization, we can avoid this complexity.

Overall this should help with flakey [nan, nan] errors in the quantization histogram, and it should also help with CI timing out.

Test plan

OSS XNNPACK CI for all model delegation

cc @digantdesai@cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9634

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit ccf664e with merge base 4b8ac94 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-linux-aarch64 (phi_4_mini, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t fe2f1f663ff952269e03a3d674e379f9abda3381abdf9d58218ccb94f5758da3 /exec failed with exit code 137

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Great!

[mcr229]

phi_4_mini ci tests are failing with:

.ci/scripts/test_model.sh: line 75: 30893 Killed "${PYTHON_EXECUTABLE}" -m examples.models.llama.export_llama --model "${MODEL_NAME}" -c examples/models/llama/params/demo_rand_params.pth -p examples/models/phi_4_mini/config.json

there is no error message with the run, so I don't think anything is failing but just that this tests is getting killed. Running it locally on my laptop seems to pass

[jackzhxng]

Seems fine, doesn't seem like it should have affected the phi test. If it's consistently getting killed it might be OOMing

[mcr229]

Summary

Changes:

1. When initializing Llama2 for aot_compiler, since checkpoints can only e downloaded from hugging face, we initialize llama2 with uninitialized weights. The problem with this is that when running quantization, we can run into errors with the histogram if the unitialized values are nan. We fix this by initializing the weights with zeros if no check point is provided. This enforces that quantization step can still work.
2. Quant Type in AoT compiler. When looking at the model options available to XNNPACK, everything is quantized with per-tensor static quantization. This isn't the best option for all the models available. For example transformer based models like Llama and MobileBert would likely prefer dynamically quantized per channel weights, where has CNN like MobileNet would prefer statically quantized per channel weights. We add this type of Quant Type to the existing models options. This also helps with Test Timeouts. per-tensor static quantization on a model like llama can take a long time due to the introduction of MANY q/dq nodes, and the complex partitions it creates. As a result, proposing partitions can take a long time due to the constant BFS to find the largest possible partition. By specifying the more apt quantization scheme like dynamic per-channel quantization, we can avoid this complexity.

Overall this should help with flakey [nan, nan] errors in the quantization histogram, and it should also help with CI timing out.

Test plan

OSS XNNPACK CI for all model delegation

cc @digantdesai@cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9634

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit ccf664e with merge base 4b8ac94 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-linux-aarch64 (phi_4_mini, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t fe2f1f663ff952269e03a3d674e379f9abda3381abdf9d58218ccb94f5758da3 /exec failed with exit code 137

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Great!

[mcr229]

phi_4_mini ci tests are failing with:

.ci/scripts/test_model.sh: line 75: 30893 Killed "${PYTHON_EXECUTABLE}" -m examples.models.llama.export_llama --model "${MODEL_NAME}" -c examples/models/llama/params/demo_rand_params.pth -p examples/models/phi_4_mini/config.json

there is no error message with the run, so I don't think anything is failing but just that this tests is getting killed. Running it locally on my laptop seems to pass

[jackzhxng]

Seems fine, doesn't seem like it should have affected the phi test. If it's consistently getting killed it might be OOMing

[mcr229]

Summary

Changes:

1. When initializing Llama2 for aot_compiler, since checkpoints can only e downloaded from hugging face, we initialize llama2 with uninitialized weights. The problem with this is that when running quantization, we can run into errors with the histogram if the unitialized values are nan. We fix this by initializing the weights with zeros if no check point is provided. This enforces that quantization step can still work.
2. Quant Type in AoT compiler. When looking at the model options available to XNNPACK, everything is quantized with per-tensor static quantization. This isn't the best option for all the models available. For example transformer based models like Llama and MobileBert would likely prefer dynamically quantized per channel weights, where has CNN like MobileNet would prefer statically quantized per channel weights. We add this type of Quant Type to the existing models options. This also helps with Test Timeouts. per-tensor static quantization on a model like llama can take a long time due to the introduction of MANY q/dq nodes, and the complex partitions it creates. As a result, proposing partitions can take a long time due to the constant BFS to find the largest possible partition. By specifying the more apt quantization scheme like dynamic per-channel quantization, we can avoid this complexity.

Overall this should help with flakey [nan, nan] errors in the quantization histogram, and it should also help with CI timing out.

Test plan

OSS XNNPACK CI for all model delegation

cc @digantdesai@cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9634

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit ccf664e with merge base 4b8ac94 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-linux-aarch64 (phi_4_mini, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t fe2f1f663ff952269e03a3d674e379f9abda3381abdf9d58218ccb94f5758da3 /exec failed with exit code 137

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Great!

[mcr229]

phi_4_mini ci tests are failing with:

.ci/scripts/test_model.sh: line 75: 30893 Killed "${PYTHON_EXECUTABLE}" -m examples.models.llama.export_llama --model "${MODEL_NAME}" -c examples/models/llama/params/demo_rand_params.pth -p examples/models/phi_4_mini/config.json

there is no error message with the run, so I don't think anything is failing but just that this tests is getting killed. Running it locally on my laptop seems to pass

[jackzhxng]

Seems fine, doesn't seem like it should have affected the phi test. If it's consistently getting killed it might be OOMing

[mcr229]

Summary

Changes:

1. When initializing Llama2 for aot_compiler, since checkpoints can only e downloaded from hugging face, we initialize llama2 with uninitialized weights. The problem with this is that when running quantization, we can run into errors with the histogram if the unitialized values are nan. We fix this by initializing the weights with zeros if no check point is provided. This enforces that quantization step can still work.
2. Quant Type in AoT compiler. When looking at the model options available to XNNPACK, everything is quantized with per-tensor static quantization. This isn't the best option for all the models available. For example transformer based models like Llama and MobileBert would likely prefer dynamically quantized per channel weights, where has CNN like MobileNet would prefer statically quantized per channel weights. We add this type of Quant Type to the existing models options. This also helps with Test Timeouts. per-tensor static quantization on a model like llama can take a long time due to the introduction of MANY q/dq nodes, and the complex partitions it creates. As a result, proposing partitions can take a long time due to the constant BFS to find the largest possible partition. By specifying the more apt quantization scheme like dynamic per-channel quantization, we can avoid this complexity.

Overall this should help with flakey [nan, nan] errors in the quantization histogram, and it should also help with CI timing out.

Test plan

OSS XNNPACK CI for all model delegation

cc @digantdesai@cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9634

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit ccf664e with merge base 4b8ac94 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-linux-aarch64 (phi_4_mini, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t fe2f1f663ff952269e03a3d674e379f9abda3381abdf9d58218ccb94f5758da3 /exec failed with exit code 137

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Great!

[mcr229]

phi_4_mini ci tests are failing with:

.ci/scripts/test_model.sh: line 75: 30893 Killed "${PYTHON_EXECUTABLE}" -m examples.models.llama.export_llama --model "${MODEL_NAME}" -c examples/models/llama/params/demo_rand_params.pth -p examples/models/phi_4_mini/config.json

there is no error message with the run, so I don't think anything is failing but just that this tests is getting killed. Running it locally on my laptop seems to pass

[jackzhxng]

Seems fine, doesn't seem like it should have affected the phi test. If it's consistently getting killed it might be OOMing

[mcr229]

Summary

Changes:

1. When initializing Llama2 for aot_compiler, since checkpoints can only e downloaded from hugging face, we initialize llama2 with uninitialized weights. The problem with this is that when running quantization, we can run into errors with the histogram if the unitialized values are nan. We fix this by initializing the weights with zeros if no check point is provided. This enforces that quantization step can still work.
2. Quant Type in AoT compiler. When looking at the model options available to XNNPACK, everything is quantized with per-tensor static quantization. This isn't the best option for all the models available. For example transformer based models like Llama and MobileBert would likely prefer dynamically quantized per channel weights, where has CNN like MobileNet would prefer statically quantized per channel weights. We add this type of Quant Type to the existing models options. This also helps with Test Timeouts. per-tensor static quantization on a model like llama can take a long time due to the introduction of MANY q/dq nodes, and the complex partitions it creates. As a result, proposing partitions can take a long time due to the constant BFS to find the largest possible partition. By specifying the more apt quantization scheme like dynamic per-channel quantization, we can avoid this complexity.

Overall this should help with flakey [nan, nan] errors in the quantization histogram, and it should also help with CI timing out.

Test plan

OSS XNNPACK CI for all model delegation

cc @digantdesai@cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9634

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit ccf664e with merge base 4b8ac94 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-linux-aarch64 (phi_4_mini, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t fe2f1f663ff952269e03a3d674e379f9abda3381abdf9d58218ccb94f5758da3 /exec failed with exit code 137

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Great!

[mcr229]

phi_4_mini ci tests are failing with:

.ci/scripts/test_model.sh: line 75: 30893 Killed "${PYTHON_EXECUTABLE}" -m examples.models.llama.export_llama --model "${MODEL_NAME}" -c examples/models/llama/params/demo_rand_params.pth -p examples/models/phi_4_mini/config.json

there is no error message with the run, so I don't think anything is failing but just that this tests is getting killed. Running it locally on my laptop seems to pass

[jackzhxng]

Seems fine, doesn't seem like it should have affected the phi test. If it's consistently getting killed it might be OOMing

[mcr229]

Summary

Changes:

1. When initializing Llama2 for aot_compiler, since checkpoints can only e downloaded from hugging face, we initialize llama2 with uninitialized weights. The problem with this is that when running quantization, we can run into errors with the histogram if the unitialized values are nan. We fix this by initializing the weights with zeros if no check point is provided. This enforces that quantization step can still work.
2. Quant Type in AoT compiler. When looking at the model options available to XNNPACK, everything is quantized with per-tensor static quantization. This isn't the best option for all the models available. For example transformer based models like Llama and MobileBert would likely prefer dynamically quantized per channel weights, where has CNN like MobileNet would prefer statically quantized per channel weights. We add this type of Quant Type to the existing models options. This also helps with Test Timeouts. per-tensor static quantization on a model like llama can take a long time due to the introduction of MANY q/dq nodes, and the complex partitions it creates. As a result, proposing partitions can take a long time due to the constant BFS to find the largest possible partition. By specifying the more apt quantization scheme like dynamic per-channel quantization, we can avoid this complexity.

Overall this should help with flakey [nan, nan] errors in the quantization histogram, and it should also help with CI timing out.

Test plan

OSS XNNPACK CI for all model delegation

cc @digantdesai@cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9634

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit ccf664e with merge base 4b8ac94 ():

NEW FAILURE - The following job has failed:

• trunk / test-models-linux-aarch64 (phi_4_mini, portable) / linux-job (gh)
  RuntimeError: Command docker exec -t fe2f1f663ff952269e03a3d674e379f9abda3381abdf9d58218ccb94f5758da3 /exec failed with exit code 137

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mergennachin]

Great!

[mcr229]

phi_4_mini ci tests are failing with:

.ci/scripts/test_model.sh: line 75: 30893 Killed "${PYTHON_EXECUTABLE}" -m examples.models.llama.export_llama --model "${MODEL_NAME}" -c examples/models/llama/params/demo_rand_params.pth -p examples/models/phi_4_mini/config.json

there is no error message with the run, so I don't think anything is failing but just that this tests is getting killed. Running it locally on my laptop seems to pass

[jackzhxng]

Seems fine, doesn't seem like it should have affected the phi test. If it's consistently getting killed it might be OOMing