Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs - #8573

Merged
kimishpatel merged 254 commits into
pytorch:mainfrom
ynimmaga:openvino_backend
Mar 28, 2025
Merged

Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs#8573
kimishpatel merged 254 commits into
pytorch:mainfrom
ynimmaga:openvino_backend

Conversation

@ynimmaga

@ynimmagaynimmaga commented Feb 19, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

This PR introduces support for the OpenVINO backend in Executorch, enabling accelerated inference on Intel hardware, including CPU, GPU, and NPU devices. OpenVINO optimizes deep learning model performance by leveraging hardware-specific enhancements. The PR also introduces the OpenVINO quantizer with NNCF (Neural Network Compression Framework) for model optimization. The functionality has been tested on several torchvision and timm models, with plans to test and enable support for additional model types in the future.

Below is a description of the features:

  • OpenVINO Backend Integration: The backends/openvino directory includes build scripts, AOT components (partitioner, preprocesser), OpenVINO Quantizer, and runtime backend files that register the OpenVINO backend, manage OpenVINO’s inference engine interactions, including model execution, device-specific optimizations, and backend initialization. It also contains tests for layers and models. See backends/openvino/README.md for usage.

  • OpenVINO Examples: The examples/openvino directory provides scripts for AOT optimization, quantization, and C++ executor examples. It includes instructions for optimizing the models, quantizing them, and exporting Executorch programs with OpenVINO optimizations. Refer to examples/openvino/README.md for details.

  • E2E Tutorial: Added an end-to-end tutorial in docs/source/build-run-openvino.md.

Test plan

This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors for CPU, GPU, and NPU devices. To run the layer tests and model tests, please refer to backends/openvino/tests/README.md

cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev

ynimmagaand others added 30 commits November 13, 2024 14:52
@ynimmaga
Fixed library path errors in cmake
86f685e
@ynimmaga
Configure device with compile spec
bb12f60
@ynimmaga
Added aot compiler script for openvino
c9c8967
@ynimmaga
Temporary build instructions
f5bd832
@ynimmaga
Added standalone build script for openvino backend
f3dc62c
@ynimmaga
Merge branch 'openvino_backend' ofhttps://github.com/ynimmaga/execut…
eec7831
…orch into openvino_backend
@cavusmustafa
Handling multiple inputs/outputs with zero-copy
ca852de
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
bdddb16
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
9803bd8
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
b1f38f8
@ynimmaga
Merge pull request#5from cavusmustafa/zero_copy_output
7c43014
Handling multiple inputs/outputs with zero-copy
@ynimmaga
Added fallback with portable kernels
0703814
@ynimmaga
Added openvino_functions.yaml containing only unsupported ops
0a769be
@ynimmaga
Updated the unsupported ops for fallback
0e6707d
@ynimmaga
Removed redundant example for openvino backend
68a1cd4
@ynimmaga
Reconfigured openvino backend functions
4d1b4eb
@ynimmaga
Updated openvino backend cmake file
5c55a56
@ynimmaga
Merge pull request#6from ynimmaga/portable_kernels
3c6d123
Added fallback with portable kernels
@ynimmaga
Added arguments for model path and num iters to openvino example
29d8400
@cavusmustafa
Merge pull request#8from ynimmaga/portable_kernels
379937e
Enhancements to openvino example
@cavusmustafa
Initial unit tests for OpenVINO backend
5806788
@cavusmustafa
Unit test update and cleanup
916ba64
@cavusmustafa
Input/Output processing for example and unit tests
e0b1bb7
@cavusmustafa
Added executorch parameter to openvino_compile call
9108770
@cavusmustafa
New op unit tests added
ecbe5e2
@ynimmaga
Merge pull request#7from cavusmustafa/openvino_backend_unit_tests
d563e4c
Initial unit tests for OpenVINO backend
@ynimmaga
Added license headers to the openvino files
a4d7458
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
197cdc2
@ynimmaga
Merge pull request#9from ynimmaga/doc_changes
268255b
Added license headers to the openvino files
@ynimmaga
Get openvino backend device from compile specs
8302911
@kimishpatel

kimishpatel commented Mar 27, 2025

Copy link
Copy Markdown
Contributor

@ynimmaga
In openvino install process make -j($nproc) fails for me. I am on a linux machine. Here is the paste https://gist.github.com/kimishpatel/5ec3448568c047a3524d49bc1c878ad3

Hi @kimishpatel, just want to confirm if you followed the below steps from the install instructions:
git submodule update --init --recursive sudo ./install_build_dependencies.sh

yes all of those. i was compiling some feedback on them as well, but yes i did all that. I can try to blow build directory and try rebuilding it, but cloning worked fine and build deps also installed ok. it seems there are some compilation errors for some targets. Not all. Target that built fine, via cmake, seems to ahve been installed. look at my gh gist

I was following instructions here https://github.com/ynimmaga/executorch/blob/openvino_backend/backends/openvino/README.md

Could you please try a clean build and also let us know more about the configuration (Linux version, system config etc.)? Here are more details regarding system requirements for OpenVINO: https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html

This is what I have

NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"

It is not ubuntu or other versions suppoted.

daniil-lyakhov
daniil-lyakhov reviewed Mar 27, 2025
View reviewed changes
Comment threadexamples/openvino/README.md Outdated
@mergennachin

mergennachin commented Mar 27, 2025

Copy link
Copy Markdown
Contributor

@ynimmaga thanks for the lint patches. a few still remaining looks like

https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573

@suryasidd

suryasidd commented Mar 27, 2025•
edited
Loading

Copy link
Copy Markdown
Collaborator

@ynimmaga thanks for the lint patches. a few still remaining looks like

https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573

Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?

@kimishpatel

kimishpatel commented Mar 28, 2025

Copy link
Copy Markdown
Contributor

https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573

there is also sklearn related lint error

@kimishpatel

kimishpatel commented Mar 28, 2025

Copy link
Copy Markdown
Contributor

@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573

Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?

has this been fixed?

@ynimmaga

ynimmaga commented Mar 28, 2025

Copy link
Copy Markdown
ContributorAuthor

@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573

Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?

has this been fixed?

Hi @kimishpatel, thanks for approving the PR. Yes, this has been fixed with ynimmaga#45

@kimishpatel
kimishpatel merged commit ce74f8e into pytorch:mainMar 28, 2025
@kimishpatel

kimishpatel commented Mar 28, 2025

Copy link
Copy Markdown
Contributor

ok landed

@cbilgin

cbilgin commented Mar 31, 2025

Copy link
Copy Markdown

As fyi, we have a discord channel and have highlighted this PR there too: https://discordapp.com/channels/1334270993966825602/1334274132182827069/1355286663701725356

cc: @ilya-lavrenov, @alexsu52, @AlexKoff88, @suryasidd, @cavusmustafa, @daniil-lyakhov

Thank you for this great contribution!

kirklandsign pushed a commit that referenced this pull request Apr 11, 2025
@ynimmaga@cavusmustafa
@alexsu52@daniil-lyakhov@kimishpatel@suryasidd@kirklandsign
Openvino backend for Executorch to enable inference on Intel CPUs, GP…
6cf95d4
…Us, NPUs (#8573)
### Summary
This PR introduces support for the OpenVINO backend in Executorch,
enabling accelerated inference on Intel hardware, including CPU, GPU,
and NPU devices. OpenVINO optimizes deep learning model performance by
leveraging hardware-specific enhancements. The PR also introduces the
OpenVINO quantizer with NNCF (Neural Network Compression Framework) for
model optimization. The functionality has been tested on several
torchvision and timm models, with plans to test and enable support for
additional model types in the future.
Below is a description of the features:
- OpenVINO Backend Integration: The backends/openvino directory includes
build scripts, AOT components (partitioner, preprocesser), OpenVINO
Quantizer, and runtime backend files that register the OpenVINO backend,
manage OpenVINO’s inference engine interactions, including model
execution, device-specific optimizations, and backend initialization. It
also contains tests for layers and models. See
backends/openvino/README.md for usage.
- OpenVINO Examples: The examples/openvino directory provides scripts
for AOT optimization, quantization, and C++ executor examples. It
includes instructions for optimizing the models, quantizing them, and
exporting Executorch programs with OpenVINO optimizations. Refer to
examples/openvino/README.md for details.
- E2E Tutorial: Added an end-to-end tutorial in
docs/source/build-run-openvino.md.
### Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors
for CPU, GPU, and NPU devices. To run the layer tests and model tests,
please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
---------
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
Co-authored-by: Aleksandr Suslov <alexander.suslov@intel.com>
Co-authored-by: dlyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Kimish Patel <kimishpatel@fb.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@ynimmaga
ynimmaga deleted the openvino_backend branch April 14, 2025 22:28
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin left review comments
@jathujathujathu left review comments
@suryasiddsuryasiddsuryasidd left review comments
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@dbortdbortAwaiting requested review from dbort
+7 more reviewers
@ilya-lavrenovilya-lavrenovilya-lavrenov left review comments
@alexsu52alexsu52alexsu52 left review comments
@cccclaicccclaicccclai left review comments
@AlexKoff88AlexKoff88AlexKoff88 left review comments
@cavusmustafacavusmustafacavusmustafa left review comments
@daniil-lyakhovdaniil-lyakhovdaniil-lyakhov left review comments
@kimishpatelkimishpatelkimishpatel approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.partner: intelFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Intelrelease notes: openvinoOpenVino backend related issues, features, bugs etc.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

13 participants

@ynimmaga@facebook-github-bot@kimishpatel@cbilgin@mergennachin@suryasidd@ilya-lavrenov@alexsu52@jathu@cccclai@AlexKoff88@cavusmustafa@daniil-lyakhov
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs - #8573
Merged
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Mar 28, 2025
Merged
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs#8573
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Conversation
@ynimmaga
@ynimmagaynimmaga
commented
Feb 19, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
This PR introduces support for the OpenVINO backend in Executorch, enabling accelerated inference on Intel hardware, including CPU, GPU, and NPU devices. OpenVINO optimizes deep learning model performance by leveraging hardware-specific enhancements. The PR also introduces the OpenVINO quantizer with NNCF (Neural Network Compression Framework) for model optimization. The functionality has been tested on several torchvision and timm models, with plans to test and enable support for additional model types in the future.
Below is a description of the features:
  • OpenVINO Backend Integration: The backends/openvino directory includes build scripts, AOT components (partitioner, preprocesser), OpenVINO Quantizer, and runtime backend files that register the OpenVINO backend, manage OpenVINO’s inference engine interactions, including model execution, device-specific optimizations, and backend initialization. It also contains tests for layers and models. See backends/openvino/README.md for usage.
  • OpenVINO Examples: The examples/openvino directory provides scripts for AOT optimization, quantization, and C++ executor examples. It includes instructions for optimizing the models, quantizing them, and exporting Executorch programs with OpenVINO optimizations. Refer to examples/openvino/README.md for details.
  • E2E Tutorial: Added an end-to-end tutorial in docs/source/build-run-openvino.md.
Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors for CPU, GPU, and NPU devices. To run the layer tests and model tests, please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
ynimmagaand others
added 30 commits
November 13, 2024 14:52
@ynimmaga
Fixed library path errors in cmake
86f685e
@ynimmaga
Configure device with compile spec
bb12f60
@ynimmaga
Added aot compiler script for openvino
c9c8967
@ynimmaga
Temporary build instructions
f5bd832
@ynimmaga
Added standalone build script for openvino backend
f3dc62c
@ynimmaga
Merge branch 'openvino_backend' ofhttps://github.com/ynimmaga/execut…
eec7831
…orch into openvino_backend
@cavusmustafa
Handling multiple inputs/outputs with zero-copy
ca852de
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
bdddb16
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
9803bd8
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
b1f38f8
@ynimmaga
Merge pull request#5from cavusmustafa/zero_copy_output
7c43014
Handling multiple inputs/outputs with zero-copy
@ynimmaga
Added fallback with portable kernels
0703814
@ynimmaga
Added openvino_functions.yaml containing only unsupported ops
0a769be
@ynimmaga
Updated the unsupported ops for fallback
0e6707d
@ynimmaga
Removed redundant example for openvino backend
68a1cd4
@ynimmaga
Reconfigured openvino backend functions
4d1b4eb
@ynimmaga
Updated openvino backend cmake file
5c55a56
@ynimmaga
Merge pull request#6from ynimmaga/portable_kernels
3c6d123
Added fallback with portable kernels
@ynimmaga
Added arguments for model path and num iters to openvino example
29d8400
@cavusmustafa
Merge pull request#8from ynimmaga/portable_kernels
379937e
Enhancements to openvino example
@cavusmustafa
Initial unit tests for OpenVINO backend
5806788
@cavusmustafa
Unit test update and cleanup
916ba64
@cavusmustafa
Input/Output processing for example and unit tests
e0b1bb7
@cavusmustafa
Added executorch parameter to openvino_compile call
9108770
@cavusmustafa
New op unit tests added
ecbe5e2
@ynimmaga
Merge pull request#7from cavusmustafa/openvino_backend_unit_tests
d563e4c
Initial unit tests for OpenVINO backend
@ynimmaga
Added license headers to the openvino files
a4d7458
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
197cdc2
@ynimmaga
Merge pull request#9from ynimmaga/doc_changes
268255b
Added license headers to the openvino files
@ynimmaga
Get openvino backend device from compile specs
8302911
@kimishpatel
kimishpatel
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga

In openvino install process make -j($nproc) fails for me. I am on a linux machine. Here is the paste https://gist.github.com/kimishpatel/5ec3448568c047a3524d49bc1c878ad3
Hi @kimishpatel, just want to confirm if you followed the below steps from the install instructions:

git submodule update --init --recursive sudo ./install_build_dependencies.sh
yes all of those. i was compiling some feedback on them as well, but yes i did all that. I can try to blow build directory and try rebuilding it, but cloning worked fine and build deps also installed ok. it seems there are some compilation errors for some targets. Not all. Target that built fine, via cmake, seems to ahve been installed. look at my gh gist
I was following instructions here https://github.com/ynimmaga/executorch/blob/openvino_backend/backends/openvino/README.md
Could you please try a clean build and also let us know more about the configuration (Linux version, system config etc.)? Here are more details regarding system requirements for OpenVINO: https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html
This is what I have
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"

It is not ubuntu or other versions suppoted.
daniil-lyakhov
daniil-lyakhov
reviewed
Mar 27, 2025
View reviewed changes
Comment threadexamples/openvino/README.md
Outdated

@mergennachin
mergennachin
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
@suryasidd
suryasidd
commented
Mar 27, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
there is also sklearn related lint error
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
@ynimmaga
ynimmaga
commented
Mar 28, 2025

Copy link


Copy Markdown

ContributorAuthor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
Hi @kimishpatel, thanks for approving the PR. Yes, this has been fixed with ynimmaga#45
@kimishpatel
kimishpatel
merged commit ce74f8e
into
pytorch:mainMar 28, 2025
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
ok landed
@cbilgin
cbilgin
commented
Mar 31, 2025

Copy link


Copy Markdown

As fyi, we have a discord channel and have highlighted this PR there too: https://discordapp.com/channels/1334270993966825602/1334274132182827069/1355286663701725356
cc: @ilya-lavrenov, @alexsu52, @AlexKoff88, @suryasidd, @cavusmustafa, @daniil-lyakhov
Thank you for this great contribution!
kirklandsign pushed a commit
that referenced
this pull request
Apr 11, 2025
@ynimmaga@cavusmustafa
@alexsu52@daniil-lyakhov@kimishpatel@suryasidd@kirklandsign
Openvino backend for Executorch to enable inference on Intel CPUs, GP…
6cf95d4
…Us, NPUs (#8573)
### Summary
This PR introduces support for the OpenVINO backend in Executorch,
enabling accelerated inference on Intel hardware, including CPU, GPU,
and NPU devices. OpenVINO optimizes deep learning model performance by
leveraging hardware-specific enhancements. The PR also introduces the
OpenVINO quantizer with NNCF (Neural Network Compression Framework) for
model optimization. The functionality has been tested on several
torchvision and timm models, with plans to test and enable support for
additional model types in the future.
Below is a description of the features:
- OpenVINO Backend Integration: The backends/openvino directory includes
build scripts, AOT components (partitioner, preprocesser), OpenVINO
Quantizer, and runtime backend files that register the OpenVINO backend,
manage OpenVINO’s inference engine interactions, including model
execution, device-specific optimizations, and backend initialization. It
also contains tests for layers and models. See
backends/openvino/README.md for usage.
- OpenVINO Examples: The examples/openvino directory provides scripts
for AOT optimization, quantization, and C++ executor examples. It
includes instructions for optimizing the models, quantizing them, and
exporting Executorch programs with OpenVINO optimizations. Refer to
examples/openvino/README.md for details.
- E2E Tutorial: Added an end-to-end tutorial in
docs/source/build-run-openvino.md.
### Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors
for CPU, GPU, and NPU devices. To run the layer tests and model tests,
please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
---------
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
Co-authored-by: Aleksandr Suslov <alexander.suslov@intel.com>
Co-authored-by: dlyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Kimish Patel <kimishpatel@fb.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@ynimmaga
ynimmaga
deleted the

openvino_backend

branch
April 14, 2025 22:28
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin left review comments
@jathujathujathu left review comments
@suryasiddsuryasiddsuryasidd left review comments
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@dbortdbortAwaiting requested review from dbort

+7 more reviewers

@ilya-lavrenovilya-lavrenovilya-lavrenov left review comments
@alexsu52alexsu52alexsu52 left review comments
@cccclaicccclaicccclai left review comments
@AlexKoff88AlexKoff88AlexKoff88 left review comments
@cavusmustafacavusmustafacavusmustafa left review comments
@daniil-lyakhovdaniil-lyakhovdaniil-lyakhov left review comments
@kimishpatelkimishpatelkimishpatel approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.partner: intelFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Intelrelease notes: openvinoOpenVino backend related issues, features, bugs etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

13 participants

@ynimmaga@facebook-github-bot@kimishpatel@cbilgin@mergennachin@suryasidd@ilya-lavrenov@alexsu52@jathu@cccclai@AlexKoff88@cavusmustafa@daniil-lyakhov

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs - #8573
Merged
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Mar 28, 2025
Merged
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs#8573
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Conversation
@ynimmaga
@ynimmagaynimmaga
commented
Feb 19, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
This PR introduces support for the OpenVINO backend in Executorch, enabling accelerated inference on Intel hardware, including CPU, GPU, and NPU devices. OpenVINO optimizes deep learning model performance by leveraging hardware-specific enhancements. The PR also introduces the OpenVINO quantizer with NNCF (Neural Network Compression Framework) for model optimization. The functionality has been tested on several torchvision and timm models, with plans to test and enable support for additional model types in the future.
Below is a description of the features:
  • OpenVINO Backend Integration: The backends/openvino directory includes build scripts, AOT components (partitioner, preprocesser), OpenVINO Quantizer, and runtime backend files that register the OpenVINO backend, manage OpenVINO’s inference engine interactions, including model execution, device-specific optimizations, and backend initialization. It also contains tests for layers and models. See backends/openvino/README.md for usage.
  • OpenVINO Examples: The examples/openvino directory provides scripts for AOT optimization, quantization, and C++ executor examples. It includes instructions for optimizing the models, quantizing them, and exporting Executorch programs with OpenVINO optimizations. Refer to examples/openvino/README.md for details.
  • E2E Tutorial: Added an end-to-end tutorial in docs/source/build-run-openvino.md.
Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors for CPU, GPU, and NPU devices. To run the layer tests and model tests, please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
ynimmagaand others
added 30 commits
November 13, 2024 14:52
@ynimmaga
Fixed library path errors in cmake
86f685e
@ynimmaga
Configure device with compile spec
bb12f60
@ynimmaga
Added aot compiler script for openvino
c9c8967
@ynimmaga
Temporary build instructions
f5bd832
@ynimmaga
Added standalone build script for openvino backend
f3dc62c
@ynimmaga
Merge branch 'openvino_backend' ofhttps://github.com/ynimmaga/execut…
eec7831
…orch into openvino_backend
@cavusmustafa
Handling multiple inputs/outputs with zero-copy
ca852de
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
bdddb16
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
9803bd8
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
b1f38f8
@ynimmaga
Merge pull request#5from cavusmustafa/zero_copy_output
7c43014
Handling multiple inputs/outputs with zero-copy
@ynimmaga
Added fallback with portable kernels
0703814
@ynimmaga
Added openvino_functions.yaml containing only unsupported ops
0a769be
@ynimmaga
Updated the unsupported ops for fallback
0e6707d
@ynimmaga
Removed redundant example for openvino backend
68a1cd4
@ynimmaga
Reconfigured openvino backend functions
4d1b4eb
@ynimmaga
Updated openvino backend cmake file
5c55a56
@ynimmaga
Merge pull request#6from ynimmaga/portable_kernels
3c6d123
Added fallback with portable kernels
@ynimmaga
Added arguments for model path and num iters to openvino example
29d8400
@cavusmustafa
Merge pull request#8from ynimmaga/portable_kernels
379937e
Enhancements to openvino example
@cavusmustafa
Initial unit tests for OpenVINO backend
5806788
@cavusmustafa
Unit test update and cleanup
916ba64
@cavusmustafa
Input/Output processing for example and unit tests
e0b1bb7
@cavusmustafa
Added executorch parameter to openvino_compile call
9108770
@cavusmustafa
New op unit tests added
ecbe5e2
@ynimmaga
Merge pull request#7from cavusmustafa/openvino_backend_unit_tests
d563e4c
Initial unit tests for OpenVINO backend
@ynimmaga
Added license headers to the openvino files
a4d7458
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
197cdc2
@ynimmaga
Merge pull request#9from ynimmaga/doc_changes
268255b
Added license headers to the openvino files
@ynimmaga
Get openvino backend device from compile specs
8302911
@kimishpatel
kimishpatel
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga

In openvino install process make -j($nproc) fails for me. I am on a linux machine. Here is the paste https://gist.github.com/kimishpatel/5ec3448568c047a3524d49bc1c878ad3
Hi @kimishpatel, just want to confirm if you followed the below steps from the install instructions:

git submodule update --init --recursive sudo ./install_build_dependencies.sh
yes all of those. i was compiling some feedback on them as well, but yes i did all that. I can try to blow build directory and try rebuilding it, but cloning worked fine and build deps also installed ok. it seems there are some compilation errors for some targets. Not all. Target that built fine, via cmake, seems to ahve been installed. look at my gh gist
I was following instructions here https://github.com/ynimmaga/executorch/blob/openvino_backend/backends/openvino/README.md
Could you please try a clean build and also let us know more about the configuration (Linux version, system config etc.)? Here are more details regarding system requirements for OpenVINO: https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html
This is what I have
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"

It is not ubuntu or other versions suppoted.
daniil-lyakhov
daniil-lyakhov
reviewed
Mar 27, 2025
View reviewed changes
Comment threadexamples/openvino/README.md
Outdated

@mergennachin
mergennachin
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
@suryasidd
suryasidd
commented
Mar 27, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
there is also sklearn related lint error
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
@ynimmaga
ynimmaga
commented
Mar 28, 2025

Copy link


Copy Markdown

ContributorAuthor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
Hi @kimishpatel, thanks for approving the PR. Yes, this has been fixed with ynimmaga#45
@kimishpatel
kimishpatel
merged commit ce74f8e
into
pytorch:mainMar 28, 2025
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
ok landed
@cbilgin
cbilgin
commented
Mar 31, 2025

Copy link


Copy Markdown

As fyi, we have a discord channel and have highlighted this PR there too: https://discordapp.com/channels/1334270993966825602/1334274132182827069/1355286663701725356
cc: @ilya-lavrenov, @alexsu52, @AlexKoff88, @suryasidd, @cavusmustafa, @daniil-lyakhov
Thank you for this great contribution!
kirklandsign pushed a commit
that referenced
this pull request
Apr 11, 2025
@ynimmaga@cavusmustafa
@alexsu52@daniil-lyakhov@kimishpatel@suryasidd@kirklandsign
Openvino backend for Executorch to enable inference on Intel CPUs, GP…
6cf95d4
…Us, NPUs (#8573)
### Summary
This PR introduces support for the OpenVINO backend in Executorch,
enabling accelerated inference on Intel hardware, including CPU, GPU,
and NPU devices. OpenVINO optimizes deep learning model performance by
leveraging hardware-specific enhancements. The PR also introduces the
OpenVINO quantizer with NNCF (Neural Network Compression Framework) for
model optimization. The functionality has been tested on several
torchvision and timm models, with plans to test and enable support for
additional model types in the future.
Below is a description of the features:
- OpenVINO Backend Integration: The backends/openvino directory includes
build scripts, AOT components (partitioner, preprocesser), OpenVINO
Quantizer, and runtime backend files that register the OpenVINO backend,
manage OpenVINO’s inference engine interactions, including model
execution, device-specific optimizations, and backend initialization. It
also contains tests for layers and models. See
backends/openvino/README.md for usage.
- OpenVINO Examples: The examples/openvino directory provides scripts
for AOT optimization, quantization, and C++ executor examples. It
includes instructions for optimizing the models, quantizing them, and
exporting Executorch programs with OpenVINO optimizations. Refer to
examples/openvino/README.md for details.
- E2E Tutorial: Added an end-to-end tutorial in
docs/source/build-run-openvino.md.
### Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors
for CPU, GPU, and NPU devices. To run the layer tests and model tests,
please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
---------
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
Co-authored-by: Aleksandr Suslov <alexander.suslov@intel.com>
Co-authored-by: dlyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Kimish Patel <kimishpatel@fb.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@ynimmaga
ynimmaga
deleted the

openvino_backend

branch
April 14, 2025 22:28
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin left review comments
@jathujathujathu left review comments
@suryasiddsuryasiddsuryasidd left review comments
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@dbortdbortAwaiting requested review from dbort

+7 more reviewers

@ilya-lavrenovilya-lavrenovilya-lavrenov left review comments
@alexsu52alexsu52alexsu52 left review comments
@cccclaicccclaicccclai left review comments
@AlexKoff88AlexKoff88AlexKoff88 left review comments
@cavusmustafacavusmustafacavusmustafa left review comments
@daniil-lyakhovdaniil-lyakhovdaniil-lyakhov left review comments
@kimishpatelkimishpatelkimishpatel approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.partner: intelFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Intelrelease notes: openvinoOpenVino backend related issues, features, bugs etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

13 participants

@ynimmaga@facebook-github-bot@kimishpatel@cbilgin@mergennachin@suryasidd@ilya-lavrenov@alexsu52@jathu@cccclai@AlexKoff88@cavusmustafa@daniil-lyakhov

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs - #8573
Merged
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Mar 28, 2025
Merged
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs#8573
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Conversation
@ynimmaga
@ynimmagaynimmaga
commented
Feb 19, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
This PR introduces support for the OpenVINO backend in Executorch, enabling accelerated inference on Intel hardware, including CPU, GPU, and NPU devices. OpenVINO optimizes deep learning model performance by leveraging hardware-specific enhancements. The PR also introduces the OpenVINO quantizer with NNCF (Neural Network Compression Framework) for model optimization. The functionality has been tested on several torchvision and timm models, with plans to test and enable support for additional model types in the future.
Below is a description of the features:
  • OpenVINO Backend Integration: The backends/openvino directory includes build scripts, AOT components (partitioner, preprocesser), OpenVINO Quantizer, and runtime backend files that register the OpenVINO backend, manage OpenVINO’s inference engine interactions, including model execution, device-specific optimizations, and backend initialization. It also contains tests for layers and models. See backends/openvino/README.md for usage.
  • OpenVINO Examples: The examples/openvino directory provides scripts for AOT optimization, quantization, and C++ executor examples. It includes instructions for optimizing the models, quantizing them, and exporting Executorch programs with OpenVINO optimizations. Refer to examples/openvino/README.md for details.
  • E2E Tutorial: Added an end-to-end tutorial in docs/source/build-run-openvino.md.
Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors for CPU, GPU, and NPU devices. To run the layer tests and model tests, please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
ynimmagaand others
added 30 commits
November 13, 2024 14:52
@ynimmaga
Fixed library path errors in cmake
86f685e
@ynimmaga
Configure device with compile spec
bb12f60
@ynimmaga
Added aot compiler script for openvino
c9c8967
@ynimmaga
Temporary build instructions
f5bd832
@ynimmaga
Added standalone build script for openvino backend
f3dc62c
@ynimmaga
Merge branch 'openvino_backend' ofhttps://github.com/ynimmaga/execut…
eec7831
…orch into openvino_backend
@cavusmustafa
Handling multiple inputs/outputs with zero-copy
ca852de
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
bdddb16
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
9803bd8
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
b1f38f8
@ynimmaga
Merge pull request#5from cavusmustafa/zero_copy_output
7c43014
Handling multiple inputs/outputs with zero-copy
@ynimmaga
Added fallback with portable kernels
0703814
@ynimmaga
Added openvino_functions.yaml containing only unsupported ops
0a769be
@ynimmaga
Updated the unsupported ops for fallback
0e6707d
@ynimmaga
Removed redundant example for openvino backend
68a1cd4
@ynimmaga
Reconfigured openvino backend functions
4d1b4eb
@ynimmaga
Updated openvino backend cmake file
5c55a56
@ynimmaga
Merge pull request#6from ynimmaga/portable_kernels
3c6d123
Added fallback with portable kernels
@ynimmaga
Added arguments for model path and num iters to openvino example
29d8400
@cavusmustafa
Merge pull request#8from ynimmaga/portable_kernels
379937e
Enhancements to openvino example
@cavusmustafa
Initial unit tests for OpenVINO backend
5806788
@cavusmustafa
Unit test update and cleanup
916ba64
@cavusmustafa
Input/Output processing for example and unit tests
e0b1bb7
@cavusmustafa
Added executorch parameter to openvino_compile call
9108770
@cavusmustafa
New op unit tests added
ecbe5e2
@ynimmaga
Merge pull request#7from cavusmustafa/openvino_backend_unit_tests
d563e4c
Initial unit tests for OpenVINO backend
@ynimmaga
Added license headers to the openvino files
a4d7458
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
197cdc2
@ynimmaga
Merge pull request#9from ynimmaga/doc_changes
268255b
Added license headers to the openvino files
@ynimmaga
Get openvino backend device from compile specs
8302911
@kimishpatel
kimishpatel
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga

In openvino install process make -j($nproc) fails for me. I am on a linux machine. Here is the paste https://gist.github.com/kimishpatel/5ec3448568c047a3524d49bc1c878ad3
Hi @kimishpatel, just want to confirm if you followed the below steps from the install instructions:

git submodule update --init --recursive sudo ./install_build_dependencies.sh
yes all of those. i was compiling some feedback on them as well, but yes i did all that. I can try to blow build directory and try rebuilding it, but cloning worked fine and build deps also installed ok. it seems there are some compilation errors for some targets. Not all. Target that built fine, via cmake, seems to ahve been installed. look at my gh gist
I was following instructions here https://github.com/ynimmaga/executorch/blob/openvino_backend/backends/openvino/README.md
Could you please try a clean build and also let us know more about the configuration (Linux version, system config etc.)? Here are more details regarding system requirements for OpenVINO: https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html
This is what I have
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"

It is not ubuntu or other versions suppoted.
daniil-lyakhov
daniil-lyakhov
reviewed
Mar 27, 2025
View reviewed changes
Comment threadexamples/openvino/README.md
Outdated

@mergennachin
mergennachin
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
@suryasidd
suryasidd
commented
Mar 27, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
there is also sklearn related lint error
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
@ynimmaga
ynimmaga
commented
Mar 28, 2025

Copy link


Copy Markdown

ContributorAuthor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
Hi @kimishpatel, thanks for approving the PR. Yes, this has been fixed with ynimmaga#45
@kimishpatel
kimishpatel
merged commit ce74f8e
into
pytorch:mainMar 28, 2025
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
ok landed
@cbilgin
cbilgin
commented
Mar 31, 2025

Copy link


Copy Markdown

As fyi, we have a discord channel and have highlighted this PR there too: https://discordapp.com/channels/1334270993966825602/1334274132182827069/1355286663701725356
cc: @ilya-lavrenov, @alexsu52, @AlexKoff88, @suryasidd, @cavusmustafa, @daniil-lyakhov
Thank you for this great contribution!
kirklandsign pushed a commit
that referenced
this pull request
Apr 11, 2025
@ynimmaga@cavusmustafa
@alexsu52@daniil-lyakhov@kimishpatel@suryasidd@kirklandsign
Openvino backend for Executorch to enable inference on Intel CPUs, GP…
6cf95d4
…Us, NPUs (#8573)
### Summary
This PR introduces support for the OpenVINO backend in Executorch,
enabling accelerated inference on Intel hardware, including CPU, GPU,
and NPU devices. OpenVINO optimizes deep learning model performance by
leveraging hardware-specific enhancements. The PR also introduces the
OpenVINO quantizer with NNCF (Neural Network Compression Framework) for
model optimization. The functionality has been tested on several
torchvision and timm models, with plans to test and enable support for
additional model types in the future.
Below is a description of the features:
- OpenVINO Backend Integration: The backends/openvino directory includes
build scripts, AOT components (partitioner, preprocesser), OpenVINO
Quantizer, and runtime backend files that register the OpenVINO backend,
manage OpenVINO’s inference engine interactions, including model
execution, device-specific optimizations, and backend initialization. It
also contains tests for layers and models. See
backends/openvino/README.md for usage.
- OpenVINO Examples: The examples/openvino directory provides scripts
for AOT optimization, quantization, and C++ executor examples. It
includes instructions for optimizing the models, quantizing them, and
exporting Executorch programs with OpenVINO optimizations. Refer to
examples/openvino/README.md for details.
- E2E Tutorial: Added an end-to-end tutorial in
docs/source/build-run-openvino.md.
### Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors
for CPU, GPU, and NPU devices. To run the layer tests and model tests,
please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
---------
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
Co-authored-by: Aleksandr Suslov <alexander.suslov@intel.com>
Co-authored-by: dlyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Kimish Patel <kimishpatel@fb.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@ynimmaga
ynimmaga
deleted the

openvino_backend

branch
April 14, 2025 22:28
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin left review comments
@jathujathujathu left review comments
@suryasiddsuryasiddsuryasidd left review comments
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@dbortdbortAwaiting requested review from dbort

+7 more reviewers

@ilya-lavrenovilya-lavrenovilya-lavrenov left review comments
@alexsu52alexsu52alexsu52 left review comments
@cccclaicccclaicccclai left review comments
@AlexKoff88AlexKoff88AlexKoff88 left review comments
@cavusmustafacavusmustafacavusmustafa left review comments
@daniil-lyakhovdaniil-lyakhovdaniil-lyakhov left review comments
@kimishpatelkimishpatelkimishpatel approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.partner: intelFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Intelrelease notes: openvinoOpenVino backend related issues, features, bugs etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

13 participants

@ynimmaga@facebook-github-bot@kimishpatel@cbilgin@mergennachin@suryasidd@ilya-lavrenov@alexsu52@jathu@cccclai@AlexKoff88@cavusmustafa@daniil-lyakhov

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs - #8573
Merged
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Mar 28, 2025
Merged
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs#8573
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Conversation
@ynimmaga
@ynimmagaynimmaga
commented
Feb 19, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
This PR introduces support for the OpenVINO backend in Executorch, enabling accelerated inference on Intel hardware, including CPU, GPU, and NPU devices. OpenVINO optimizes deep learning model performance by leveraging hardware-specific enhancements. The PR also introduces the OpenVINO quantizer with NNCF (Neural Network Compression Framework) for model optimization. The functionality has been tested on several torchvision and timm models, with plans to test and enable support for additional model types in the future.
Below is a description of the features:
  • OpenVINO Backend Integration: The backends/openvino directory includes build scripts, AOT components (partitioner, preprocesser), OpenVINO Quantizer, and runtime backend files that register the OpenVINO backend, manage OpenVINO’s inference engine interactions, including model execution, device-specific optimizations, and backend initialization. It also contains tests for layers and models. See backends/openvino/README.md for usage.
  • OpenVINO Examples: The examples/openvino directory provides scripts for AOT optimization, quantization, and C++ executor examples. It includes instructions for optimizing the models, quantizing them, and exporting Executorch programs with OpenVINO optimizations. Refer to examples/openvino/README.md for details.
  • E2E Tutorial: Added an end-to-end tutorial in docs/source/build-run-openvino.md.
Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors for CPU, GPU, and NPU devices. To run the layer tests and model tests, please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
ynimmagaand others
added 30 commits
November 13, 2024 14:52
@ynimmaga
Fixed library path errors in cmake
86f685e
@ynimmaga
Configure device with compile spec
bb12f60
@ynimmaga
Added aot compiler script for openvino
c9c8967
@ynimmaga
Temporary build instructions
f5bd832
@ynimmaga
Added standalone build script for openvino backend
f3dc62c
@ynimmaga
Merge branch 'openvino_backend' ofhttps://github.com/ynimmaga/execut…
eec7831
…orch into openvino_backend
@cavusmustafa
Handling multiple inputs/outputs with zero-copy
ca852de
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
bdddb16
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
9803bd8
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
b1f38f8
@ynimmaga
Merge pull request#5from cavusmustafa/zero_copy_output
7c43014
Handling multiple inputs/outputs with zero-copy
@ynimmaga
Added fallback with portable kernels
0703814
@ynimmaga
Added openvino_functions.yaml containing only unsupported ops
0a769be
@ynimmaga
Updated the unsupported ops for fallback
0e6707d
@ynimmaga
Removed redundant example for openvino backend
68a1cd4
@ynimmaga
Reconfigured openvino backend functions
4d1b4eb
@ynimmaga
Updated openvino backend cmake file
5c55a56
@ynimmaga
Merge pull request#6from ynimmaga/portable_kernels
3c6d123
Added fallback with portable kernels
@ynimmaga
Added arguments for model path and num iters to openvino example
29d8400
@cavusmustafa
Merge pull request#8from ynimmaga/portable_kernels
379937e
Enhancements to openvino example
@cavusmustafa
Initial unit tests for OpenVINO backend
5806788
@cavusmustafa
Unit test update and cleanup
916ba64
@cavusmustafa
Input/Output processing for example and unit tests
e0b1bb7
@cavusmustafa
Added executorch parameter to openvino_compile call
9108770
@cavusmustafa
New op unit tests added
ecbe5e2
@ynimmaga
Merge pull request#7from cavusmustafa/openvino_backend_unit_tests
d563e4c
Initial unit tests for OpenVINO backend
@ynimmaga
Added license headers to the openvino files
a4d7458
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
197cdc2
@ynimmaga
Merge pull request#9from ynimmaga/doc_changes
268255b
Added license headers to the openvino files
@ynimmaga
Get openvino backend device from compile specs
8302911
@kimishpatel
kimishpatel
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga

In openvino install process make -j($nproc) fails for me. I am on a linux machine. Here is the paste https://gist.github.com/kimishpatel/5ec3448568c047a3524d49bc1c878ad3
Hi @kimishpatel, just want to confirm if you followed the below steps from the install instructions:

git submodule update --init --recursive sudo ./install_build_dependencies.sh
yes all of those. i was compiling some feedback on them as well, but yes i did all that. I can try to blow build directory and try rebuilding it, but cloning worked fine and build deps also installed ok. it seems there are some compilation errors for some targets. Not all. Target that built fine, via cmake, seems to ahve been installed. look at my gh gist
I was following instructions here https://github.com/ynimmaga/executorch/blob/openvino_backend/backends/openvino/README.md
Could you please try a clean build and also let us know more about the configuration (Linux version, system config etc.)? Here are more details regarding system requirements for OpenVINO: https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html
This is what I have
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"

It is not ubuntu or other versions suppoted.
daniil-lyakhov
daniil-lyakhov
reviewed
Mar 27, 2025
View reviewed changes
Comment threadexamples/openvino/README.md
Outdated

@mergennachin
mergennachin
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
@suryasidd
suryasidd
commented
Mar 27, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
there is also sklearn related lint error
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
@ynimmaga
ynimmaga
commented
Mar 28, 2025

Copy link


Copy Markdown

ContributorAuthor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
Hi @kimishpatel, thanks for approving the PR. Yes, this has been fixed with ynimmaga#45
@kimishpatel
kimishpatel
merged commit ce74f8e
into
pytorch:mainMar 28, 2025
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
ok landed
@cbilgin
cbilgin
commented
Mar 31, 2025

Copy link


Copy Markdown

As fyi, we have a discord channel and have highlighted this PR there too: https://discordapp.com/channels/1334270993966825602/1334274132182827069/1355286663701725356
cc: @ilya-lavrenov, @alexsu52, @AlexKoff88, @suryasidd, @cavusmustafa, @daniil-lyakhov
Thank you for this great contribution!
kirklandsign pushed a commit
that referenced
this pull request
Apr 11, 2025
@ynimmaga@cavusmustafa
@alexsu52@daniil-lyakhov@kimishpatel@suryasidd@kirklandsign
Openvino backend for Executorch to enable inference on Intel CPUs, GP…
6cf95d4
…Us, NPUs (#8573)
### Summary
This PR introduces support for the OpenVINO backend in Executorch,
enabling accelerated inference on Intel hardware, including CPU, GPU,
and NPU devices. OpenVINO optimizes deep learning model performance by
leveraging hardware-specific enhancements. The PR also introduces the
OpenVINO quantizer with NNCF (Neural Network Compression Framework) for
model optimization. The functionality has been tested on several
torchvision and timm models, with plans to test and enable support for
additional model types in the future.
Below is a description of the features:
- OpenVINO Backend Integration: The backends/openvino directory includes
build scripts, AOT components (partitioner, preprocesser), OpenVINO
Quantizer, and runtime backend files that register the OpenVINO backend,
manage OpenVINO’s inference engine interactions, including model
execution, device-specific optimizations, and backend initialization. It
also contains tests for layers and models. See
backends/openvino/README.md for usage.
- OpenVINO Examples: The examples/openvino directory provides scripts
for AOT optimization, quantization, and C++ executor examples. It
includes instructions for optimizing the models, quantizing them, and
exporting Executorch programs with OpenVINO optimizations. Refer to
examples/openvino/README.md for details.
- E2E Tutorial: Added an end-to-end tutorial in
docs/source/build-run-openvino.md.
### Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors
for CPU, GPU, and NPU devices. To run the layer tests and model tests,
please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
---------
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
Co-authored-by: Aleksandr Suslov <alexander.suslov@intel.com>
Co-authored-by: dlyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Kimish Patel <kimishpatel@fb.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@ynimmaga
ynimmaga
deleted the

openvino_backend

branch
April 14, 2025 22:28
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin left review comments
@jathujathujathu left review comments
@suryasiddsuryasiddsuryasidd left review comments
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@dbortdbortAwaiting requested review from dbort

+7 more reviewers

@ilya-lavrenovilya-lavrenovilya-lavrenov left review comments
@alexsu52alexsu52alexsu52 left review comments
@cccclaicccclaicccclai left review comments
@AlexKoff88AlexKoff88AlexKoff88 left review comments
@cavusmustafacavusmustafacavusmustafa left review comments
@daniil-lyakhovdaniil-lyakhovdaniil-lyakhov left review comments
@kimishpatelkimishpatelkimishpatel approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.partner: intelFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Intelrelease notes: openvinoOpenVino backend related issues, features, bugs etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

13 participants

@ynimmaga@facebook-github-bot@kimishpatel@cbilgin@mergennachin@suryasidd@ilya-lavrenov@alexsu52@jathu@cccclai@AlexKoff88@cavusmustafa@daniil-lyakhov

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs - #8573
Merged
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Mar 28, 2025
Merged
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs#8573
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Conversation
@ynimmaga
@ynimmagaynimmaga
commented
Feb 19, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
This PR introduces support for the OpenVINO backend in Executorch, enabling accelerated inference on Intel hardware, including CPU, GPU, and NPU devices. OpenVINO optimizes deep learning model performance by leveraging hardware-specific enhancements. The PR also introduces the OpenVINO quantizer with NNCF (Neural Network Compression Framework) for model optimization. The functionality has been tested on several torchvision and timm models, with plans to test and enable support for additional model types in the future.
Below is a description of the features:
  • OpenVINO Backend Integration: The backends/openvino directory includes build scripts, AOT components (partitioner, preprocesser), OpenVINO Quantizer, and runtime backend files that register the OpenVINO backend, manage OpenVINO’s inference engine interactions, including model execution, device-specific optimizations, and backend initialization. It also contains tests for layers and models. See backends/openvino/README.md for usage.
  • OpenVINO Examples: The examples/openvino directory provides scripts for AOT optimization, quantization, and C++ executor examples. It includes instructions for optimizing the models, quantizing them, and exporting Executorch programs with OpenVINO optimizations. Refer to examples/openvino/README.md for details.
  • E2E Tutorial: Added an end-to-end tutorial in docs/source/build-run-openvino.md.
Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors for CPU, GPU, and NPU devices. To run the layer tests and model tests, please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
ynimmagaand others
added 30 commits
November 13, 2024 14:52
@ynimmaga
Fixed library path errors in cmake
86f685e
@ynimmaga
Configure device with compile spec
bb12f60
@ynimmaga
Added aot compiler script for openvino
c9c8967
@ynimmaga
Temporary build instructions
f5bd832
@ynimmaga
Added standalone build script for openvino backend
f3dc62c
@ynimmaga
Merge branch 'openvino_backend' ofhttps://github.com/ynimmaga/execut…
eec7831
…orch into openvino_backend
@cavusmustafa
Handling multiple inputs/outputs with zero-copy
ca852de
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
bdddb16
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
9803bd8
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
b1f38f8
@ynimmaga
Merge pull request#5from cavusmustafa/zero_copy_output
7c43014
Handling multiple inputs/outputs with zero-copy
@ynimmaga
Added fallback with portable kernels
0703814
@ynimmaga
Added openvino_functions.yaml containing only unsupported ops
0a769be
@ynimmaga
Updated the unsupported ops for fallback
0e6707d
@ynimmaga
Removed redundant example for openvino backend
68a1cd4
@ynimmaga
Reconfigured openvino backend functions
4d1b4eb
@ynimmaga
Updated openvino backend cmake file
5c55a56
@ynimmaga
Merge pull request#6from ynimmaga/portable_kernels
3c6d123
Added fallback with portable kernels
@ynimmaga
Added arguments for model path and num iters to openvino example
29d8400
@cavusmustafa
Merge pull request#8from ynimmaga/portable_kernels
379937e
Enhancements to openvino example
@cavusmustafa
Initial unit tests for OpenVINO backend
5806788
@cavusmustafa
Unit test update and cleanup
916ba64
@cavusmustafa
Input/Output processing for example and unit tests
e0b1bb7
@cavusmustafa
Added executorch parameter to openvino_compile call
9108770
@cavusmustafa
New op unit tests added
ecbe5e2
@ynimmaga
Merge pull request#7from cavusmustafa/openvino_backend_unit_tests
d563e4c
Initial unit tests for OpenVINO backend
@ynimmaga
Added license headers to the openvino files
a4d7458
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
197cdc2
@ynimmaga
Merge pull request#9from ynimmaga/doc_changes
268255b
Added license headers to the openvino files
@ynimmaga
Get openvino backend device from compile specs
8302911
@kimishpatel
kimishpatel
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga

In openvino install process make -j($nproc) fails for me. I am on a linux machine. Here is the paste https://gist.github.com/kimishpatel/5ec3448568c047a3524d49bc1c878ad3
Hi @kimishpatel, just want to confirm if you followed the below steps from the install instructions:

git submodule update --init --recursive sudo ./install_build_dependencies.sh
yes all of those. i was compiling some feedback on them as well, but yes i did all that. I can try to blow build directory and try rebuilding it, but cloning worked fine and build deps also installed ok. it seems there are some compilation errors for some targets. Not all. Target that built fine, via cmake, seems to ahve been installed. look at my gh gist
I was following instructions here https://github.com/ynimmaga/executorch/blob/openvino_backend/backends/openvino/README.md
Could you please try a clean build and also let us know more about the configuration (Linux version, system config etc.)? Here are more details regarding system requirements for OpenVINO: https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html
This is what I have
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"

It is not ubuntu or other versions suppoted.
daniil-lyakhov
daniil-lyakhov
reviewed
Mar 27, 2025
View reviewed changes
Comment threadexamples/openvino/README.md
Outdated

@mergennachin
mergennachin
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
@suryasidd
suryasidd
commented
Mar 27, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
there is also sklearn related lint error
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
@ynimmaga
ynimmaga
commented
Mar 28, 2025

Copy link


Copy Markdown

ContributorAuthor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
Hi @kimishpatel, thanks for approving the PR. Yes, this has been fixed with ynimmaga#45
@kimishpatel
kimishpatel
merged commit ce74f8e
into
pytorch:mainMar 28, 2025
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
ok landed
@cbilgin
cbilgin
commented
Mar 31, 2025

Copy link


Copy Markdown

As fyi, we have a discord channel and have highlighted this PR there too: https://discordapp.com/channels/1334270993966825602/1334274132182827069/1355286663701725356
cc: @ilya-lavrenov, @alexsu52, @AlexKoff88, @suryasidd, @cavusmustafa, @daniil-lyakhov
Thank you for this great contribution!
kirklandsign pushed a commit
that referenced
this pull request
Apr 11, 2025
@ynimmaga@cavusmustafa
@alexsu52@daniil-lyakhov@kimishpatel@suryasidd@kirklandsign
Openvino backend for Executorch to enable inference on Intel CPUs, GP…
6cf95d4
…Us, NPUs (#8573)
### Summary
This PR introduces support for the OpenVINO backend in Executorch,
enabling accelerated inference on Intel hardware, including CPU, GPU,
and NPU devices. OpenVINO optimizes deep learning model performance by
leveraging hardware-specific enhancements. The PR also introduces the
OpenVINO quantizer with NNCF (Neural Network Compression Framework) for
model optimization. The functionality has been tested on several
torchvision and timm models, with plans to test and enable support for
additional model types in the future.
Below is a description of the features:
- OpenVINO Backend Integration: The backends/openvino directory includes
build scripts, AOT components (partitioner, preprocesser), OpenVINO
Quantizer, and runtime backend files that register the OpenVINO backend,
manage OpenVINO’s inference engine interactions, including model
execution, device-specific optimizations, and backend initialization. It
also contains tests for layers and models. See
backends/openvino/README.md for usage.
- OpenVINO Examples: The examples/openvino directory provides scripts
for AOT optimization, quantization, and C++ executor examples. It
includes instructions for optimizing the models, quantizing them, and
exporting Executorch programs with OpenVINO optimizations. Refer to
examples/openvino/README.md for details.
- E2E Tutorial: Added an end-to-end tutorial in
docs/source/build-run-openvino.md.
### Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors
for CPU, GPU, and NPU devices. To run the layer tests and model tests,
please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
---------
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
Co-authored-by: Aleksandr Suslov <alexander.suslov@intel.com>
Co-authored-by: dlyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Kimish Patel <kimishpatel@fb.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@ynimmaga
ynimmaga
deleted the

openvino_backend

branch
April 14, 2025 22:28
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin left review comments
@jathujathujathu left review comments
@suryasiddsuryasiddsuryasidd left review comments
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@dbortdbortAwaiting requested review from dbort

+7 more reviewers

@ilya-lavrenovilya-lavrenovilya-lavrenov left review comments
@alexsu52alexsu52alexsu52 left review comments
@cccclaicccclaicccclai left review comments
@AlexKoff88AlexKoff88AlexKoff88 left review comments
@cavusmustafacavusmustafacavusmustafa left review comments
@daniil-lyakhovdaniil-lyakhovdaniil-lyakhov left review comments
@kimishpatelkimishpatelkimishpatel approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.partner: intelFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Intelrelease notes: openvinoOpenVino backend related issues, features, bugs etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

13 participants

@ynimmaga@facebook-github-bot@kimishpatel@cbilgin@mergennachin@suryasidd@ilya-lavrenov@alexsu52@jathu@cccclai@AlexKoff88@cavusmustafa@daniil-lyakhov

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs - #8573
Merged
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Mar 28, 2025
Merged
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs#8573
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Conversation
@ynimmaga
@ynimmagaynimmaga
commented
Feb 19, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
This PR introduces support for the OpenVINO backend in Executorch, enabling accelerated inference on Intel hardware, including CPU, GPU, and NPU devices. OpenVINO optimizes deep learning model performance by leveraging hardware-specific enhancements. The PR also introduces the OpenVINO quantizer with NNCF (Neural Network Compression Framework) for model optimization. The functionality has been tested on several torchvision and timm models, with plans to test and enable support for additional model types in the future.
Below is a description of the features:
  • OpenVINO Backend Integration: The backends/openvino directory includes build scripts, AOT components (partitioner, preprocesser), OpenVINO Quantizer, and runtime backend files that register the OpenVINO backend, manage OpenVINO’s inference engine interactions, including model execution, device-specific optimizations, and backend initialization. It also contains tests for layers and models. See backends/openvino/README.md for usage.
  • OpenVINO Examples: The examples/openvino directory provides scripts for AOT optimization, quantization, and C++ executor examples. It includes instructions for optimizing the models, quantizing them, and exporting Executorch programs with OpenVINO optimizations. Refer to examples/openvino/README.md for details.
  • E2E Tutorial: Added an end-to-end tutorial in docs/source/build-run-openvino.md.
Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors for CPU, GPU, and NPU devices. To run the layer tests and model tests, please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
ynimmagaand others
added 30 commits
November 13, 2024 14:52
@ynimmaga
Fixed library path errors in cmake
86f685e
@ynimmaga
Configure device with compile spec
bb12f60
@ynimmaga
Added aot compiler script for openvino
c9c8967
@ynimmaga
Temporary build instructions
f5bd832
@ynimmaga
Added standalone build script for openvino backend
f3dc62c
@ynimmaga
Merge branch 'openvino_backend' ofhttps://github.com/ynimmaga/execut…
eec7831
…orch into openvino_backend
@cavusmustafa
Handling multiple inputs/outputs with zero-copy
ca852de
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
bdddb16
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
9803bd8
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
b1f38f8
@ynimmaga
Merge pull request#5from cavusmustafa/zero_copy_output
7c43014
Handling multiple inputs/outputs with zero-copy
@ynimmaga
Added fallback with portable kernels
0703814
@ynimmaga
Added openvino_functions.yaml containing only unsupported ops
0a769be
@ynimmaga
Updated the unsupported ops for fallback
0e6707d
@ynimmaga
Removed redundant example for openvino backend
68a1cd4
@ynimmaga
Reconfigured openvino backend functions
4d1b4eb
@ynimmaga
Updated openvino backend cmake file
5c55a56
@ynimmaga
Merge pull request#6from ynimmaga/portable_kernels
3c6d123
Added fallback with portable kernels
@ynimmaga
Added arguments for model path and num iters to openvino example
29d8400
@cavusmustafa
Merge pull request#8from ynimmaga/portable_kernels
379937e
Enhancements to openvino example
@cavusmustafa
Initial unit tests for OpenVINO backend
5806788
@cavusmustafa
Unit test update and cleanup
916ba64
@cavusmustafa
Input/Output processing for example and unit tests
e0b1bb7
@cavusmustafa
Added executorch parameter to openvino_compile call
9108770
@cavusmustafa
New op unit tests added
ecbe5e2
@ynimmaga
Merge pull request#7from cavusmustafa/openvino_backend_unit_tests
d563e4c
Initial unit tests for OpenVINO backend
@ynimmaga
Added license headers to the openvino files
a4d7458
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
197cdc2
@ynimmaga
Merge pull request#9from ynimmaga/doc_changes
268255b
Added license headers to the openvino files
@ynimmaga
Get openvino backend device from compile specs
8302911
@kimishpatel
kimishpatel
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga

In openvino install process make -j($nproc) fails for me. I am on a linux machine. Here is the paste https://gist.github.com/kimishpatel/5ec3448568c047a3524d49bc1c878ad3
Hi @kimishpatel, just want to confirm if you followed the below steps from the install instructions:

git submodule update --init --recursive sudo ./install_build_dependencies.sh
yes all of those. i was compiling some feedback on them as well, but yes i did all that. I can try to blow build directory and try rebuilding it, but cloning worked fine and build deps also installed ok. it seems there are some compilation errors for some targets. Not all. Target that built fine, via cmake, seems to ahve been installed. look at my gh gist
I was following instructions here https://github.com/ynimmaga/executorch/blob/openvino_backend/backends/openvino/README.md
Could you please try a clean build and also let us know more about the configuration (Linux version, system config etc.)? Here are more details regarding system requirements for OpenVINO: https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html
This is what I have
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"

It is not ubuntu or other versions suppoted.
daniil-lyakhov
daniil-lyakhov
reviewed
Mar 27, 2025
View reviewed changes
Comment threadexamples/openvino/README.md
Outdated

@mergennachin
mergennachin
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
@suryasidd
suryasidd
commented
Mar 27, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
there is also sklearn related lint error
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
@ynimmaga
ynimmaga
commented
Mar 28, 2025

Copy link


Copy Markdown

ContributorAuthor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
Hi @kimishpatel, thanks for approving the PR. Yes, this has been fixed with ynimmaga#45
@kimishpatel
kimishpatel
merged commit ce74f8e
into
pytorch:mainMar 28, 2025
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
ok landed
@cbilgin
cbilgin
commented
Mar 31, 2025

Copy link


Copy Markdown

As fyi, we have a discord channel and have highlighted this PR there too: https://discordapp.com/channels/1334270993966825602/1334274132182827069/1355286663701725356
cc: @ilya-lavrenov, @alexsu52, @AlexKoff88, @suryasidd, @cavusmustafa, @daniil-lyakhov
Thank you for this great contribution!
kirklandsign pushed a commit
that referenced
this pull request
Apr 11, 2025
@ynimmaga@cavusmustafa
@alexsu52@daniil-lyakhov@kimishpatel@suryasidd@kirklandsign
Openvino backend for Executorch to enable inference on Intel CPUs, GP…
6cf95d4
…Us, NPUs (#8573)
### Summary
This PR introduces support for the OpenVINO backend in Executorch,
enabling accelerated inference on Intel hardware, including CPU, GPU,
and NPU devices. OpenVINO optimizes deep learning model performance by
leveraging hardware-specific enhancements. The PR also introduces the
OpenVINO quantizer with NNCF (Neural Network Compression Framework) for
model optimization. The functionality has been tested on several
torchvision and timm models, with plans to test and enable support for
additional model types in the future.
Below is a description of the features:
- OpenVINO Backend Integration: The backends/openvino directory includes
build scripts, AOT components (partitioner, preprocesser), OpenVINO
Quantizer, and runtime backend files that register the OpenVINO backend,
manage OpenVINO’s inference engine interactions, including model
execution, device-specific optimizations, and backend initialization. It
also contains tests for layers and models. See
backends/openvino/README.md for usage.
- OpenVINO Examples: The examples/openvino directory provides scripts
for AOT optimization, quantization, and C++ executor examples. It
includes instructions for optimizing the models, quantizing them, and
exporting Executorch programs with OpenVINO optimizations. Refer to
examples/openvino/README.md for details.
- E2E Tutorial: Added an end-to-end tutorial in
docs/source/build-run-openvino.md.
### Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors
for CPU, GPU, and NPU devices. To run the layer tests and model tests,
please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
---------
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
Co-authored-by: Aleksandr Suslov <alexander.suslov@intel.com>
Co-authored-by: dlyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Kimish Patel <kimishpatel@fb.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@ynimmaga
ynimmaga
deleted the

openvino_backend

branch
April 14, 2025 22:28
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin left review comments
@jathujathujathu left review comments
@suryasiddsuryasiddsuryasidd left review comments
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@dbortdbortAwaiting requested review from dbort

+7 more reviewers

@ilya-lavrenovilya-lavrenovilya-lavrenov left review comments
@alexsu52alexsu52alexsu52 left review comments
@cccclaicccclaicccclai left review comments
@AlexKoff88AlexKoff88AlexKoff88 left review comments
@cavusmustafacavusmustafacavusmustafa left review comments
@daniil-lyakhovdaniil-lyakhovdaniil-lyakhov left review comments
@kimishpatelkimishpatelkimishpatel approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.partner: intelFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Intelrelease notes: openvinoOpenVino backend related issues, features, bugs etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

13 participants

@ynimmaga@facebook-github-bot@kimishpatel@cbilgin@mergennachin@suryasidd@ilya-lavrenov@alexsu52@jathu@cccclai@AlexKoff88@cavusmustafa@daniil-lyakhov

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs - #8573
Merged
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Mar 28, 2025
Merged
Openvino backend for Executorch to enable inference on Intel CPUs, GPUs, NPUs#8573
kimishpatel merged 254 commits into
pytorch:mainfrom 
ynimmaga:openvino_backend
Conversation
@ynimmaga
@ynimmagaynimmaga
commented
Feb 19, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
Summary
This PR introduces support for the OpenVINO backend in Executorch, enabling accelerated inference on Intel hardware, including CPU, GPU, and NPU devices. OpenVINO optimizes deep learning model performance by leveraging hardware-specific enhancements. The PR also introduces the OpenVINO quantizer with NNCF (Neural Network Compression Framework) for model optimization. The functionality has been tested on several torchvision and timm models, with plans to test and enable support for additional model types in the future.
Below is a description of the features:
  • OpenVINO Backend Integration: The backends/openvino directory includes build scripts, AOT components (partitioner, preprocesser), OpenVINO Quantizer, and runtime backend files that register the OpenVINO backend, manage OpenVINO’s inference engine interactions, including model execution, device-specific optimizations, and backend initialization. It also contains tests for layers and models. See backends/openvino/README.md for usage.
  • OpenVINO Examples: The examples/openvino directory provides scripts for AOT optimization, quantization, and C++ executor examples. It includes instructions for optimizing the models, quantizing them, and exporting Executorch programs with OpenVINO optimizations. Refer to examples/openvino/README.md for details.
  • E2E Tutorial: Added an end-to-end tutorial in docs/source/build-run-openvino.md.
Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors for CPU, GPU, and NPU devices. To run the layer tests and model tests, please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
ynimmagaand others
added 30 commits
November 13, 2024 14:52
@ynimmaga
Fixed library path errors in cmake
86f685e
@ynimmaga
Configure device with compile spec
bb12f60
@ynimmaga
Added aot compiler script for openvino
c9c8967
@ynimmaga
Temporary build instructions
f5bd832
@ynimmaga
Added standalone build script for openvino backend
f3dc62c
@ynimmaga
Merge branch 'openvino_backend' ofhttps://github.com/ynimmaga/execut…
eec7831
…orch into openvino_backend
@cavusmustafa
Handling multiple inputs/outputs with zero-copy
ca852de
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
bdddb16
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
9803bd8
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
b1f38f8
@ynimmaga
Merge pull request#5from cavusmustafa/zero_copy_output
7c43014
Handling multiple inputs/outputs with zero-copy
@ynimmaga
Added fallback with portable kernels
0703814
@ynimmaga
Added openvino_functions.yaml containing only unsupported ops
0a769be
@ynimmaga
Updated the unsupported ops for fallback
0e6707d
@ynimmaga
Removed redundant example for openvino backend
68a1cd4
@ynimmaga
Reconfigured openvino backend functions
4d1b4eb
@ynimmaga
Updated openvino backend cmake file
5c55a56
@ynimmaga
Merge pull request#6from ynimmaga/portable_kernels
3c6d123
Added fallback with portable kernels
@ynimmaga
Added arguments for model path and num iters to openvino example
29d8400
@cavusmustafa
Merge pull request#8from ynimmaga/portable_kernels
379937e
Enhancements to openvino example
@cavusmustafa
Initial unit tests for OpenVINO backend
5806788
@cavusmustafa
Unit test update and cleanup
916ba64
@cavusmustafa
Input/Output processing for example and unit tests
e0b1bb7
@cavusmustafa
Added executorch parameter to openvino_compile call
9108770
@cavusmustafa
New op unit tests added
ecbe5e2
@ynimmaga
Merge pull request#7from cavusmustafa/openvino_backend_unit_tests
d563e4c
Initial unit tests for OpenVINO backend
@ynimmaga
Added license headers to the openvino files
a4d7458
@ynimmaga
Merge branch 'pytorch:main' into openvino_backend
197cdc2
@ynimmaga
Merge pull request#9from ynimmaga/doc_changes
268255b
Added license headers to the openvino files
@ynimmaga
Get openvino backend device from compile specs
8302911
@kimishpatel
kimishpatel
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga

In openvino install process make -j($nproc) fails for me. I am on a linux machine. Here is the paste https://gist.github.com/kimishpatel/5ec3448568c047a3524d49bc1c878ad3
Hi @kimishpatel, just want to confirm if you followed the below steps from the install instructions:

git submodule update --init --recursive sudo ./install_build_dependencies.sh
yes all of those. i was compiling some feedback on them as well, but yes i did all that. I can try to blow build directory and try rebuilding it, but cloning worked fine and build deps also installed ok. it seems there are some compilation errors for some targets. Not all. Target that built fine, via cmake, seems to ahve been installed. look at my gh gist
I was following instructions here https://github.com/ynimmaga/executorch/blob/openvino_backend/backends/openvino/README.md
Could you please try a clean build and also let us know more about the configuration (Linux version, system config etc.)? Here are more details regarding system requirements for OpenVINO: https://docs.openvino.ai/2025/about-openvino/release-notes-openvino/system-requirements.html
This is what I have
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"

It is not ubuntu or other versions suppoted.
daniil-lyakhov
daniil-lyakhov
reviewed
Mar 27, 2025
View reviewed changes
Comment threadexamples/openvino/README.md
Outdated

@mergennachin
mergennachin
commented
Mar 27, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
@suryasidd
suryasidd
commented
Mar 27, 2025•

edited

Loading

Copy link


Copy Markdown

Collaborator
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
there is also sklearn related lint error
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
@ynimmaga
ynimmaga
commented
Mar 28, 2025

Copy link


Copy Markdown

ContributorAuthor
@ynimmaga thanks for the lint patches. a few still remaining looks like
https://github.com/pytorch/executorch/actions/runs/14099141987/job/39492002002?pr=8573
Hey @mergennachin, the remaining issues are because nncf and openvino packages are not installed while running the linter. When the packages are not installed mypy is not able to find them. Should we add an ignore for import-not-found errors ?
has this been fixed?
Hi @kimishpatel, thanks for approving the PR. Yes, this has been fixed with ynimmaga#45
@kimishpatel
kimishpatel
merged commit ce74f8e
into
pytorch:mainMar 28, 2025
@kimishpatel
kimishpatel
commented
Mar 28, 2025

Copy link


Copy Markdown

Contributor
ok landed
@cbilgin
cbilgin
commented
Mar 31, 2025

Copy link


Copy Markdown

As fyi, we have a discord channel and have highlighted this PR there too: https://discordapp.com/channels/1334270993966825602/1334274132182827069/1355286663701725356
cc: @ilya-lavrenov, @alexsu52, @AlexKoff88, @suryasidd, @cavusmustafa, @daniil-lyakhov
Thank you for this great contribution!
kirklandsign pushed a commit
that referenced
this pull request
Apr 11, 2025
@ynimmaga@cavusmustafa
@alexsu52@daniil-lyakhov@kimishpatel@suryasidd@kirklandsign
Openvino backend for Executorch to enable inference on Intel CPUs, GP…
6cf95d4
…Us, NPUs (#8573)
### Summary
This PR introduces support for the OpenVINO backend in Executorch,
enabling accelerated inference on Intel hardware, including CPU, GPU,
and NPU devices. OpenVINO optimizes deep learning model performance by
leveraging hardware-specific enhancements. The PR also introduces the
OpenVINO quantizer with NNCF (Neural Network Compression Framework) for
model optimization. The functionality has been tested on several
torchvision and timm models, with plans to test and enable support for
additional model types in the future.
Below is a description of the features:
- OpenVINO Backend Integration: The backends/openvino directory includes
build scripts, AOT components (partitioner, preprocesser), OpenVINO
Quantizer, and runtime backend files that register the OpenVINO backend,
manage OpenVINO’s inference engine interactions, including model
execution, device-specific optimizations, and backend initialization. It
also contains tests for layers and models. See
backends/openvino/README.md for usage.
- OpenVINO Examples: The examples/openvino directory provides scripts
for AOT optimization, quantization, and C++ executor examples. It
includes instructions for optimizing the models, quantizing them, and
exporting Executorch programs with OpenVINO optimizations. Refer to
examples/openvino/README.md for details.
- E2E Tutorial: Added an end-to-end tutorial in
docs/source/build-run-openvino.md.
### Test plan
This PR is tested with OpenVINO backend on Intel Core Ultra 7 processors
for CPU, GPU, and NPU devices. To run the layer tests and model tests,
please refer to backends/openvino/tests/README.md
cc: @yury-gorbachev@alexsu52@cavusmustafa@daniil-lyakhov@suryasidd@AlexKoff88@MaximProshin@AlexanderDokuchaev
---------
Co-authored-by: Cavus Mustafa <mustafa.cavus@intel.com>
Co-authored-by: Aleksandr Suslov <alexander.suslov@intel.com>
Co-authored-by: dlyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Kimish Patel <kimishpatel@fb.com>
Co-authored-by: suryasidd <surya.siddharth.pemmaraju@intel.com>
@ynimmaga
ynimmaga
deleted the

openvino_backend

branch
April 14, 2025 22:28
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin left review comments
@jathujathujathu left review comments
@suryasiddsuryasiddsuryasidd left review comments
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@dbortdbortAwaiting requested review from dbort

+7 more reviewers

@ilya-lavrenovilya-lavrenovilya-lavrenov left review comments
@alexsu52alexsu52alexsu52 left review comments
@cccclaicccclaicccclai left review comments
@AlexKoff88AlexKoff88AlexKoff88 left review comments
@cavusmustafacavusmustafacavusmustafa left review comments
@daniil-lyakhovdaniil-lyakhovdaniil-lyakhov left review comments
@kimishpatelkimishpatelkimishpatel approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.partner: intelFor backend delegation, kernels, demo, etc. from the 3rd-party partner, Intelrelease notes: openvinoOpenVino backend related issues, features, bugs etc.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

13 participants

@ynimmaga@facebook-github-bot@kimishpatel@cbilgin@mergennachin@suryasidd@ilya-lavrenov@alexsu52@jathu@cccclai@AlexKoff88@cavusmustafa@daniil-lyakhov