diff --git a/docs/source/android-prebuilt-library.md b/docs/source/android-prebuilt-library.md deleted file mode 100644 index 324c63376c5..00000000000 --- a/docs/source/android-prebuilt-library.md +++ /dev/null @@ -1,37 +0,0 @@ -# Using Android prebuilt libraries (AAR) - -We provide two prebuilt Android libraries (AAR), `executorch.aar` for generic use case (image/audio processing) and `executorch_llama.aar` for LLAMA use case. - -## Contents of libraries -- `executorch.aar` - - [Java library](https://github.com/pytorch/executorch/tree/main/extension/android/src/main/java/org/pytorch/executorch) - - JNI contains the JNI binding for [NativePeer.java](https://github.com/pytorch/executorch/blob/main/extension/android/src/main/java/org/pytorch/executorch/NativePeer.java) and ExecuTorch native library, including core ExecuTorch runtime libraries, XNNPACK backend, Portable kernels, Optimized kernels, and Quantized kernels. - - Comes with two ABI variants, arm64-v8a and x86_64. -- `executorch_llama.aar` - - [Java library](https://github.com/pytorch/executorch/tree/main/extension/android/src/main/java/org/pytorch/executorch) (Note: it contains the same Java classes as the previous Java, but it does not contain the JNI binding for generic Module/NativePeer Java code). - - JNI contains the JNI binding for [LlamaModule.java](https://github.com/pytorch/executorch/blob/main/extension/android/src/main/java/org/pytorch/executorch/LlamaModule.java) and ExecuTorch native library, including core ExecuTorch runtime libraries, XNNPACK backend, Portable kernels, Optimized kernels, Quantized kernels, and LLAMA-specific Custom ops library. - - Comes with two ABI variants, arm64-v8a and x86_64. - -## Downloading AAR -[executorch.aar](https://ossci-android.s3.amazonaws.com/executorch/release/executorch-241002/executorch.aar) -[executorch.aar.sha256sums](https://ossci-android.s3.amazonaws.com/executorch/release/executorch-241002/executorch.aar.sha256sums) - -## Using prebuilt libraries - -To add the Java library to your app, simply download the AAR, and add it to your gradle build rule. - -In your app working directory, such as example executorch/examples/demo-apps/android/LlamaDemo, -``` -mkdir -p app/libs -curl https://ossci-android.s3.amazonaws.com/executorch/release/executorch-241002/executorch.aar -o app/libs/executorch.aar -``` - -And include it in gradle: -``` -# app/build.grardle.kts -dependencies { - implementation(files("libs/executorch.aar")) -} -``` - -Now you can compile your app with the ExecuTorch Android library. diff --git a/docs/source/backends-arm-ethos-u.md b/docs/source/backends-arm-ethos-u.md index d3c50b5bc78..a9f89bdabc8 100644 --- a/docs/source/backends-arm-ethos-u.md +++ b/docs/source/backends-arm-ethos-u.md @@ -7,8 +7,8 @@ :::{grid-item-card} Tutorials we recommend you complete before this: :class-card: card-prerequisites * [Introduction to ExecuTorch](./intro-how-it-works.md) -* [Setting up ExecuTorch](./getting-started-setup.md) -* [Building ExecuTorch with CMake](./runtime-build-and-cross-compilation.md) +* [Getting Started](./getting-started.md) +* [Building ExecuTorch with CMake](./using-executorch-building-from-source.md) ::: :::{grid-item-card} What you will learn in this tutorial: @@ -280,7 +280,7 @@ The `generate_pte_file` function in `run.sh` script produces the `.pte` files ba ExecuTorch's CMake build system produces a set of build pieces which are critical for us to include and run the ExecuTorch runtime with-in the bare-metal environment we have for Corstone FVPs from Ethos-U SDK. -[This](./runtime-build-and-cross-compilation.md) document provides a detailed overview of each individual build piece. For running either variant of the `.pte` file, we will need a core set of libraries. Here is a list, +[This](./using-executorch-building-from-source.md) document provides a detailed overview of each individual build piece. For running either variant of the `.pte` file, we will need a core set of libraries. Here is a list, - `libexecutorch.a` - `libportable_kernels.a` diff --git a/docs/source/backends-cadence.md b/docs/source/backends-cadence.md index a07e4b61598..278a2f9ec2b 100644 --- a/docs/source/backends-cadence.md +++ b/docs/source/backends-cadence.md @@ -17,9 +17,9 @@ On top of being able to run on the Xtensa HiFi4 DSP, another goal of this tutori ::: :::{grid-item-card} Tutorials we recommend you complete before this: :class-card: card-prerequisites -* [Introduction to ExecuTorch](intro-how-it-works.md) -* [Setting up ExecuTorch](getting-started-setup.md) -* [Building ExecuTorch with CMake](runtime-build-and-cross-compilation.md) +* [Introduction to ExecuTorch](./intro-how-it-works.md) +* [Getting Started](./getting-started.md) +* [Building ExecuTorch with CMake](./using-executorch-building-from-source.md) ::: :::: diff --git a/docs/source/backends-coreml.md b/docs/source/backends-coreml.md index e46531fa680..804f3fe3a93 100644 --- a/docs/source/backends-coreml.md +++ b/docs/source/backends-coreml.md @@ -11,9 +11,9 @@ Core ML delegate uses Core ML APIs to enable running neural networks via Apple's ::: :::{grid-item-card} Tutorials we recommend you complete before this: :class-card: card-prerequisites -* [Introduction to ExecuTorch](intro-how-it-works.md) -* [Setting up ExecuTorch](getting-started-setup.md) -* [Building ExecuTorch with CMake](runtime-build-and-cross-compilation.md) +* [Introduction to ExecuTorch](./intro-how-it-works.md) +* [Getting Started](./getting-started.md) +* [Building ExecuTorch with CMake](./using-executorch-building-from-source.md) * [ExecuTorch iOS Demo App](demo-apps-ios.md) ::: :::: diff --git a/docs/source/backends-mediatek.md b/docs/source/backends-mediatek.md index 0ce1d89e785..5bf99553b7d 100644 --- a/docs/source/backends-mediatek.md +++ b/docs/source/backends-mediatek.md @@ -11,9 +11,9 @@ MediaTek backend empowers ExecuTorch to speed up PyTorch models on edge devices ::: :::{grid-item-card} Tutorials we recommend you complete before this: :class-card: card-prerequisites -* [Introduction to ExecuTorch](intro-how-it-works.md) -* [Setting up ExecuTorch](getting-started-setup.md) -* [Building ExecuTorch with CMake](runtime-build-and-cross-compilation.md) +* [Introduction to ExecuTorch](./intro-how-it-works.md) +* [Getting Started](./getting-started.md) +* [Building ExecuTorch with CMake](./using-executorch-building-from-source.md) ::: :::: @@ -91,4 +91,4 @@ cd executorch ```bash export LD_LIBRARY_PATH=::$LD_LIBRARY_PATH - ``` \ No newline at end of file + ``` diff --git a/docs/source/backends-mps.md b/docs/source/backends-mps.md index 0da22ad579f..4947e3e0aea 100644 --- a/docs/source/backends-mps.md +++ b/docs/source/backends-mps.md @@ -12,9 +12,9 @@ The MPS backend device maps machine learning computational graphs and primitives ::: :::{grid-item-card} Tutorials we recommend you complete before this: :class-card: card-prerequisites -* [Introduction to ExecuTorch](intro-how-it-works.md) -* [Setting up ExecuTorch](getting-started-setup.md) -* [Building ExecuTorch with CMake](runtime-build-and-cross-compilation.md) +* [Introduction to ExecuTorch](./intro-how-it-works.md) +* [Getting Started](./getting-started.md) +* [Building ExecuTorch with CMake](./using-executorch-building-from-source.md) * [ExecuTorch iOS Demo App](demo-apps-ios.md) * [ExecuTorch iOS LLaMA Demo App](llm/llama-demo-ios.md) ::: diff --git a/docs/source/backends-overview.md b/docs/source/backends-overview.md new file mode 100644 index 00000000000..dd3aa0354bc --- /dev/null +++ b/docs/source/backends-overview.md @@ -0,0 +1,20 @@ +# Backend Overview + +ExecuTorch backends provide hardware acceleration for a specific hardware target. In order to achieve maximum performance on target hardware, ExecuTorch optimizes the model for a specific backend during the export and lowering process. This means that the resulting .pte file is specialized for the specific hardware. In order to deploy to multiple backends, such as Core ML on iOS and Arm CPU on Android, it is common to generate a dedicated .pte file for each. + +The choice of hardware backend is informed by the hardware that the model is intended to be deployed on. Each backend has specific hardware requires and level of model support. See the documentation for each hardware backend for more details. + +As part of the .pte file creation process, ExecuTorch identifies portions of the model (partitions) that are supported for the given backend. These sections are processed by the backend ahead of time to support efficient execution. Portions of the model that are not supported on the delegate, if any, are executed using the portable fallback implementation on CPU. This allows for partial model acceleration when not all model operators are supported on the backend, but may have negative performance implications. In addition, multiple partitioners can be specified in order of priority. This allows for operators not supported on GPU to run on CPU via XNNPACK, for example. + +### Available Backends + +Commonly used hardware backends are listed below. For mobile, consider using XNNPACK for Android and XNNPACK or Core ML for iOS. To create a .pte file for a specific backend, pass the appropriate partitioner class to `to_edge_transform_and_lower`. See the appropriate backend documentation for more information. + +- [XNNPACK (Mobile CPU)](backends-xnnpack.md) +- [Core ML (iOS)](backends-coreml.md) +- [Metal Performance Shaders (iOS GPU)](backends-mps.md) +- [Vulkan (Android GPU)](backends-vulkan.md) +- [Qualcomm NPU](backends-qualcomm.md) +- [MediaTek NPU](backends-mediatek.md) +- [Arm Ethos-U NPU](backends-arm-ethos-u.md) +- [Cadence DSP](backends-cadence.md) diff --git a/docs/source/backends-qualcomm.md b/docs/source/backends-qualcomm.md index 9001ae716c6..041c3d7fdc3 100644 --- a/docs/source/backends-qualcomm.md +++ b/docs/source/backends-qualcomm.md @@ -14,9 +14,9 @@ Qualcomm AI Engine Direct is also referred to as QNN in the source and documenta ::: :::{grid-item-card} Tutorials we recommend you complete before this: :class-card: card-prerequisites -* [Introduction to ExecuTorch](intro-how-it-works.md) -* [Setting up ExecuTorch](getting-started-setup.md) -* [Building ExecuTorch with CMake](runtime-build-and-cross-compilation.md) +* [Introduction to ExecuTorch](./intro-how-it-works.md) +* [Getting Started](./getting-started.md) +* [Building ExecuTorch with CMake](./using-executorch-building-from-source.md) ::: :::: diff --git a/docs/source/getting-started.md b/docs/source/getting-started.md index e099eb98f63..47b8921b832 100644 --- a/docs/source/getting-started.md +++ b/docs/source/getting-started.md @@ -14,7 +14,7 @@ Pip is the recommended way to install the ExecuTorch python package. This packag pip install executorch ``` -To build the framework from source, see [Building From Source](TODO). +To build the framework from source, see [Building From Source](using-executorch-building-from-source.md). Backend delegates may require additional dependencies. See the appropriate backend documentation for more information. @@ -29,7 +29,9 @@ The following are required to install the ExecuTorch host libraries, needed to e
## Preparing the Model -Exporting is the process of taking a PyTorch model and converting it to the .pte file format used by the ExecuTorch runtime. This is done using Python APIs. PTE files for common models can be found on HuggingFace (TODO add link). +Exporting is the process of taking a PyTorch model and converting it to the .pte file format used by the ExecuTorch runtime. This is done using Python APIs. PTE files for common models, such as Llama 3.2, can be found on HuggingFace under [ExecuTorch Community](https://huggingface.co/executorch-community). These models have been exported and lowered for ExecuTorch, and can be directly deployed without needing to go through the lowering process. + +A complete example of exporting, lowering, and verifying MobileNet V2 is available as a [Colab notebook](https://colab.research.google.com/drive/1qpxrXC3YdJQzly3mRg-4ayYiOjC6rue3?usp=sharing). ### Requirements - A PyTorch model. @@ -39,7 +41,7 @@ Exporting is the process of taking a PyTorch model and converting it to the .pte ### Selecting a Backend ExecuTorch provides hardware acceleration for a wide variety of hardware. The most commonly used backends are XNNPACK, for Arm and x86 CPU, Core ML (for iOS), Vulkan (for Android GPUs), and Qualcomm (for Qualcomm-powered Android phones). -For mobile use cases, consider using XNNPACK for Android and Core ML or XNNPACK for iOS as a first step. See [Delegates](/TODO.md) for a description of available backends. +For mobile use cases, consider using XNNPACK for Android and Core ML or XNNPACK for iOS as a first step. See [Hardware Backends](backends-overview.md) for more information. ### Exporting Exporting is done using Python APIs. ExecuTorch provides a high degree of customization during the export process, but the typical flow is as follows: @@ -50,13 +52,13 @@ model = MyModel() # The PyTorch model to export example_inputs = (torch.randn(1,3,64,64),) # A tuple of inputs et_program = - executorch.exir.to_edge_transform_and_lower( - torch.export.export(model, example_inputs) + executorch.exir.to_edge_transform_and_lower( + torch.export.export(model, example_inputs) partitioner=[XnnpackPartitioner()] ).to_executorch() with open(“model.pte”, “wb”) as f: - f.write(et_program.buffer) + f.write(et_program.buffer) ``` If the model requires varying input sizes, you will need to specify the varying dimensions and bounds as part of the `export` call. See [Model Export and Lowering](using-executorch-export.md) for more information. @@ -96,7 +98,7 @@ Quick Links: #### Installation ExecuTorch provides Java bindings for Android usage, which can be consumed from both Java and Kotlin. -To add the library to your app, download the AAR, and add it to the gradle build rule. TODO Replace with Maven/Gradle package management when available. +To add the library to your app, download the AAR, and add it to the gradle build rule. ``` mkdir -p app/libs @@ -113,39 +115,39 @@ dependencies { #### Runtime APIs Models can be loaded and run using the `Module` class: ```java -import org.pytorch.executorch.EValue -import org.pytorch.executorch.Module -import org.pytorch.executorch.Tensor +import org.pytorch.executorch.EValue; +import org.pytorch.executorch.Module; +import org.pytorch.executorch.Tensor; // … -Module model = Module.load(“/path/to/model.pte”) -// TODO Add input setup -EValue output = model.forward(input_evalue); +Module model = Module.load(“/path/to/model.pte”); + +Tensor input_tensor = Tensor.fromBlob(float_data, new long[] { 1, 3, height, width }); +EValue input_evalue = EValue.from(input_tensor); +EValue[] output = model.forward(input_evalue); +float[] scores = output[0].toTensor().getDataAsFloatArray(); ``` -For more information on Android development, including building from source, a full description of the Java APIs, and information on using ExecuTorch from Android native code, see [Using ExecuTorch on Android](/TODO.md). +For a full example of running a model on Android, see the [ExecuTorch Android Demo App](https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/ExecuTorchDemo/app/src/main/java/com/example/executorchdemo/ClassificationActivity.java). For more information on Android development, including building from source, a full description of the Java APIs, and information on using ExecuTorch from Android native code, see [Using ExecuTorch on Android](using-executorch-android.md). ### iOS #### Installation -ExecuTorch supports both iOS and MacOS via C++ and Objective-C bindings, as well as hardware backends for CoreML, MPS, and CPU. The iOS runtime library is provided as a collection of .xcframework targets and are made available as a Swift PM package. +ExecuTorch supports both iOS and MacOS via C++, as well as hardware backends for CoreML, MPS, and CPU. The iOS runtime library is provided as a collection of .xcframework targets and are made available as a Swift PM package. -To get started with Xcode, go to File > Add Package Dependencies. Paste the URL of the ExecuTorch repo into the search bar and select it. Make sure to change the branch name to the desired ExecuTorch version in format “swiftpm-”, (e.g. “swiftpm-0.5.0”). The ExecuTorch dependency can also be added to the package file manually. See [Using ExecuTorch on iOS](/TODO.md) for more information. +To get started with Xcode, go to File > Add Package Dependencies. Paste the URL of the ExecuTorch repo into the search bar and select it. Make sure to change the branch name to the desired ExecuTorch version in format “swiftpm-”, (e.g. “swiftpm-0.5.0”). The ExecuTorch dependency can also be added to the package file manually. See [Using ExecuTorch on iOS](using-executorch-ios.md) for more information. #### Runtime APIs -Models can be loaded and run from Swift as follows: -```swift -// TODO Code sample -``` +Models can be loaded and run from Objective-C using the C++ APIs. -For more information on iOS integration, including an API reference, logging setup, and building from source, see [Using ExecuTorch on iOS](/TODO.md). +For more information on iOS integration, including an API reference, logging setup, and building from source, see [Using ExecuTorch on iOS](using-executorch-ios.md). ### C++ ExecuTorch provides C++ APIs, which can be used to target embedded or mobile devices. The C++ APIs provide a greater level of control compared to other language bindings, allowing for advanced memory management, data loading, and platform integration. #### Installation -CMake is the preferred build system for the ExecuTorch C++ runtime. To use with CMake, clone the ExecuTorch repository as a subdirectory of your project, and use CMake's `add_subdirectory("executorch")` to include the dependency. The `executorch` target, as well as kernel and backend targets will be made available to link against. The runtime can also be built standalone to support diverse toolchains. See [Using ExecuTorch with C++](/TODO.md) for a detailed description of build integration, targets, and cross compilation. +CMake is the preferred build system for the ExecuTorch C++ runtime. To use with CMake, clone the ExecuTorch repository as a subdirectory of your project, and use CMake's `add_subdirectory("executorch")` to include the dependency. The `executorch` target, as well as kernel and backend targets will be made available to link against. The runtime can also be built standalone to support diverse toolchains. See [Using ExecuTorch with C++](using-executorch-cpp.md) for a detailed description of build integration, targets, and cross compilation. ``` git clone -b release/0.5 https://github.com/pytorch/executorch.git @@ -199,9 +201,9 @@ For more information on the C++ APIs, see [Running an ExecuTorch Model Using the ExecuTorch provides a high-degree of customizability to support diverse hardware targets. Depending on your use cases, consider exploring one or more of the following pages: - [Export and Lowering](using-executorch-export.md) for advanced model conversion options. -- [Delegates](/TODO.md) for available backends and configuration options. -- [Using ExecuTorch on Android](/TODO.md) and [Using ExecuTorch on iOS](TODO.md) for mobile runtime integration. -- [Using ExecuTorch with C++](/TODO.md) for embedded and mobile native development. -- [Troubleshooting, Profiling, and Optimization](/TODO.md) for developer tooling and debugging. -- [API Reference](/TODO.md) for a full description of available APIs. -- [Examples](https://github.com/pytorch/executorch/tree/main/examples) for demo apps and example code. \ No newline at end of file +- [Backend Overview](backends-overview.md) for available backends and configuration options. +- [Using ExecuTorch on Android](using-executorch-android.md) and [Using ExecuTorch on iOS](using-executorch-ios.md) for mobile runtime integration. +- [Using ExecuTorch with C++](using-executorch-cpp.md) for embedded and mobile native development. +- [Profiling and Debugging](using-executorch-troubleshooting.md) for developer tooling and debugging. +- [API Reference](export-to-executorch-api-reference.md) for a full description of available APIs. +- [Examples](https://github.com/pytorch/executorch/tree/main/examples) for demo apps and example code. diff --git a/docs/source/index.rst b/docs/source/index.rst index 5061d8b056d..1af4e41f2f1 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -91,8 +91,8 @@ Topics in this section will help you get started with ExecuTorch. using-executorch-cpp using-executorch-runtime-integration using-executorch-troubleshooting - using-executorch-faqs using-executorch-building-from-source + using-executorch-faqs .. toctree:: :glob: @@ -100,6 +100,7 @@ Topics in this section will help you get started with ExecuTorch. :caption: Backends :hidden: + backends-overview backends-xnnpack backends-coreml backends-mps @@ -139,6 +140,9 @@ Topics in this section will help you get started with ExecuTorch. :hidden: runtime-overview + extension-module + extension-tensor + running-a-model-cpp-tutorial runtime-backend-delegate-implementation-and-linking runtime-platform-abstraction-layer portable-cpp-programming diff --git a/docs/source/llm/build-run-llama3-qualcomm-ai-engine-direct-backend.md b/docs/source/llm/build-run-llama3-qualcomm-ai-engine-direct-backend.md index 7ed768baf23..c02701a839c 100644 --- a/docs/source/llm/build-run-llama3-qualcomm-ai-engine-direct-backend.md +++ b/docs/source/llm/build-run-llama3-qualcomm-ai-engine-direct-backend.md @@ -5,7 +5,7 @@ This tutorial demonstrates how to export Llama 3 8B Instruct for Qualcomm AI Eng ## Prerequisites - Set up your ExecuTorch repo and environment if you haven’t done so by following [the Setting up ExecuTorch](../getting-started-setup.md) to set up the repo and dev environment. -- Read [the Building and Running ExecuTorch with Qualcomm AI Engine Direct Backend page](../build-run-qualcomm-ai-engine-direct-backend.md) to understand how to export and run a model with Qualcomm AI Engine Direct Backend on Qualcomm device. +- Read [the Building and Running ExecuTorch with Qualcomm AI Engine Direct Backend page](../backends-qualcomm.md) to understand how to export and run a model with Qualcomm AI Engine Direct Backend on Qualcomm device. - Follow [the README for executorch llama](https://github.com/pytorch/executorch/tree/main/examples/models/llama) to know how to run a llama model on mobile via ExecuTorch. - A Qualcomm device with 16GB RAM - We are continuing to optimize our memory usage to ensure compatibility with lower memory devices. diff --git a/docs/source/llm/getting-started.md b/docs/source/llm/getting-started.md index fd5dd3ba3a2..3bcba08ddfb 100644 --- a/docs/source/llm/getting-started.md +++ b/docs/source/llm/getting-started.md @@ -592,8 +592,8 @@ I'm not sure if you've heard of the "Curse of the Dragon" or not, but it's a ver The delegated model should be noticeably faster compared to the non-delegated model. For more information regarding backend delegateion, see the ExecuTorch guides -for the [XNNPACK Backend](../tutorial-xnnpack-delegate-lowering.md), [Core ML -Backend](../build-run-coreml.md) and [Qualcomm AI Engine Direct Backend](build-run-llama3-qualcomm-ai-engine-direct-backend.md). +for the [XNNPACK Backend](../backends-xnnpack.md), [Core ML +Backend](../backends-coreml.md) and [Qualcomm AI Engine Direct Backend](build-run-llama3-qualcomm-ai-engine-direct-backend.md). ## Quantization diff --git a/docs/source/runtime-overview.md b/docs/source/runtime-overview.md index 1a421fdcc0a..911d0c142e8 100644 --- a/docs/source/runtime-overview.md +++ b/docs/source/runtime-overview.md @@ -157,7 +157,7 @@ For more details about the ExecuTorch runtime, please see: * [Detailed Runtime APIs Tutorial](running-a-model-cpp-tutorial.md) * [Simplified Runtime APIs Tutorial](extension-module.md) -* [Runtime Build and Cross Compilation](runtime-build-and-cross-compilation.md) +* [Building from Source](using-executorch-building-from-source.md) * [Runtime Platform Abstraction Layer](runtime-platform-abstraction-layer.md) * [Runtime Profiling](runtime-profiling.md) * [Backends and Delegates](compiler-delegate-and-partitioner.md) diff --git a/docs/source/tutorial-xnnpack-delegate-lowering.md b/docs/source/tutorial-xnnpack-delegate-lowering.md index d1148511c5f..a469edebd54 100644 --- a/docs/source/tutorial-xnnpack-delegate-lowering.md +++ b/docs/source/tutorial-xnnpack-delegate-lowering.md @@ -12,7 +12,7 @@ In this tutorial, you will learn how to export an XNNPACK lowered Model and run :class-card: card-prerequisites * [Setting up ExecuTorch](./getting-started-setup.md) * [Model Lowering Tutorial](./tutorials/export-to-executorch-tutorial) -* [ExecuTorch XNNPACK Delegate](./native-delegates-executorch-xnnpack-delegate.md) +* [ExecuTorch XNNPACK Delegate](./backends-xnnpack.md) ::: :::: diff --git a/docs/source/using-executorch-android.md b/docs/source/using-executorch-android.md index cedb355810b..063e83170d0 100644 --- a/docs/source/using-executorch-android.md +++ b/docs/source/using-executorch-android.md @@ -4,7 +4,41 @@ To use from Android, ExecuTorch provides Java API bindings and Android platform ## Installation -TODO Instructions on downloading the pre-built AAR. Replace with Maven/Gradle package management when available. +We provide two prebuilt Android libraries (AAR), `executorch.aar` for generic use case (image/audio processing) and `executorch_llama.aar` for LLAMA use case. + +## Contents of libraries +- `executorch.aar` + - [Java library](https://github.com/pytorch/executorch/tree/main/extension/android/src/main/java/org/pytorch/executorch) + - JNI contains the JNI binding for [NativePeer.java](https://github.com/pytorch/executorch/blob/main/extension/android/src/main/java/org/pytorch/executorch/NativePeer.java) and ExecuTorch native library, including core ExecuTorch runtime libraries, XNNPACK backend, Portable kernels, Optimized kernels, and Quantized kernels. + - Comes with two ABI variants, arm64-v8a and x86_64. +- `executorch_llama.aar` + - [Java library](https://github.com/pytorch/executorch/tree/main/extension/android/src/main/java/org/pytorch/executorch) (Note: it contains the same Java classes as the previous Java, but it does not contain the JNI binding for generic Module/NativePeer Java code). + - JNI contains the JNI binding for [LlamaModule.java](https://github.com/pytorch/executorch/blob/main/extension/android/src/main/java/org/pytorch/executorch/LlamaModule.java) and ExecuTorch native library, including core ExecuTorch runtime libraries, XNNPACK backend, Portable kernels, Optimized kernels, Quantized kernels, and LLAMA-specific Custom ops library. + - Comes with two ABI variants, arm64-v8a and x86_64. + +## Downloading AAR +[executorch.aar](https://ossci-android.s3.amazonaws.com/executorch/release/executorch-241002/executorch.aar) +[executorch.aar.sha256sums](https://ossci-android.s3.amazonaws.com/executorch/release/executorch-241002/executorch.aar.sha256sums) + +## Using prebuilt libraries + +To add the Java library to your app, simply download the AAR, and add it to your gradle build rule. + +In your app working directory, such as example executorch/examples/demo-apps/android/LlamaDemo, +``` +mkdir -p app/libs +curl https://ossci-android.s3.amazonaws.com/executorch/release/executorch-241002/executorch.aar -o app/libs/executorch.aar +``` + +And include it in gradle: +``` +# app/build.grardle.kts +dependencies { + implementation(files("libs/executorch.aar")) +} +``` + +Now you can compile your app with the ExecuTorch Android library. ### Building from Source diff --git a/docs/source/using-executorch-building-from-source.md b/docs/source/using-executorch-building-from-source.md index 01ec05b1156..145b3ef7285 100644 --- a/docs/source/using-executorch-building-from-source.md +++ b/docs/source/using-executorch-building-from-source.md @@ -4,6 +4,98 @@ ExecuTorch uses [CMake](https://cmake.org/) as the primary build system. Even if you don't use CMake directly, CMake can emit scripts for other format like Make, Ninja or Xcode. For information, see [cmake-generators(7)](https://cmake.org/cmake/help/latest/manual/cmake-generators.7.html). +## System Requirements +### Operating System + +We've tested these instructions on the following systems, although they should +also work in similar environments. + + +Linux (x86_64) +- CentOS 8+ +- Ubuntu 20.04.6 LTS+ +- RHEL 8+ + +macOS (x86_64/M1/M2) +- Big Sur (11.0)+ + +Windows (x86_64) +- Windows Subsystem for Linux (WSL) with any of the Linux options + +### Software +* `conda` or another virtual environment manager + - We recommend `conda` as it provides cross-language + support and integrates smoothly with `pip` (Python's built-in package manager) + - Otherwise, Python's built-in virtual environment manager `python venv` is a good alternative. +* `g++` version 7 or higher, `clang++` version 5 or higher, or another + C++17-compatible toolchain. + +Note that the cross-compilable core runtime code supports a wider range of +toolchains, down to C++17. See the [Runtime Overview](./runtime-overview.md) for +portability details. + +## Environment Setup + +### Create a Virtual Environment + +[Install conda on your machine](https://conda.io/projects/conda/en/latest/user-guide/install/index.html). Then, create a virtual environment to manage our dependencies. + ```bash + # Create and activate a conda environment named "executorch" + conda create -yn executorch python=3.10.0 + conda activate executorch + ``` + +### Clone and install ExecuTorch requirements + + ```bash + # Clone the ExecuTorch repo from GitHub + # 'main' branch is the primary development branch where you see the latest changes. + # 'viable/strict' contains all of the commits on main that pass all of the necessary CI checks. + git clone --branch viable/strict https://github.com/pytorch/executorch.git + cd executorch + + # Update and pull submodules + git submodule sync + git submodule update --init + + # Install ExecuTorch pip package and its dependencies, as well as + # development tools like CMake. + # If developing on a Mac, make sure to install the Xcode Command Line Tools first. + ./install_executorch.sh + ``` + + Use the [`--pybind` flag](https://github.com/pytorch/executorch/blob/main/install_executorch.sh#L26-L29) to install with pybindings and dependencies for other backends. + ```bash + ./install_executorch.sh --pybind + + # Example: pybindings with CoreML *only* + ./install_executorch.sh --pybind coreml + + # Example: pybinds with CoreML *and* XNNPACK + ./install_executorch.sh --pybind coreml xnnpack + ``` + + By default, `./install_executorch.sh` command installs pybindings for XNNPACK. To disable any pybindings altogether: + ```bash + ./install_executorch.sh --pybind off + ``` + +> **_NOTE:_** Cleaning the build system +> +> When fetching a new version of the upstream repo (via `git fetch` or `git +> pull`) it is a good idea to clean the old build artifacts. The build system +> does not currently adapt well to changes in build dependencies. +> +> You should also update and pull the submodules again, in case their versions +> have changed. +> +> ```bash +> # From the root of the executorch repo: +> ./install_executorch.sh --clean +> git submodule sync +> git submodule update --init +> ``` + ## Targets Built by the CMake Build System ExecuTorch's CMake build system covers the pieces of the runtime that are @@ -159,7 +251,7 @@ xcode-select --install ``` Run the above command with `--help` flag to learn more on how to build additional backends -(like [Core ML](build-run-coreml.md), [MPS](build-run-mps.md) or XNNPACK), etc. +(like [Core ML](backends-coreml.md), [MPS](backends-mps.md) or XNNPACK), etc. Note, some backends may require additional dependencies and certain versions of Xcode and iOS. 3. Copy over the generated `.xcframework` bundles to your Xcode project, link them against @@ -172,6 +264,6 @@ Check out the [iOS Demo App](demo-apps-ios.md) tutorial for more info. You have successfully cross-compiled `executor_runner` binary to iOS and Android platforms. You can start exploring advanced features and capabilities. Here is a list of sections you might want to read next: -* [Selective build](./kernel-library-selective_build) to build the runtime that links to only kernels used by the program, which can provide significant binary size savings. +* [Selective build](kernel-library-selective-build.md) to build the runtime that links to only kernels used by the program, which can provide significant binary size savings. * Tutorials on building [Android](./demo-apps-android.md) and [iOS](./demo-apps-ios.md) demo apps. -* Tutorials on deploying applications to embedded devices such as [ARM Cortex-M/Ethos-U](./executorch-arm-delegate-tutorial.md) and [XTensa HiFi DSP](./build-run-xtensa.md). +* Tutorials on deploying applications to embedded devices such as [ARM Cortex-M/Ethos-U](backends-arm-ethos-u.md) and [XTensa HiFi DSP](./backends-cadence.md). diff --git a/docs/source/using-executorch-cpp.md b/docs/source/using-executorch-cpp.md index 12728a6f575..d0be0e3bd33 100644 --- a/docs/source/using-executorch-cpp.md +++ b/docs/source/using-executorch-cpp.md @@ -1,3 +1,64 @@ # Using ExecuTorch with C++ -Placeholder for top-level C++ documentation \ No newline at end of file +In order to support a wide variety of devices, from high-end mobile phones down to tiny embedded systems, ExecuTorch provides an API surface with a high degree of customizability. The C++ APIs expose advanced configuration options, such as controlling memory allocation, placement, and data loading. To meet the needs of both application and embedded programming, ExecuTorch provides a low-level, highly-customizable core set of APIs, and set of high-level extensions, which abstract away many of the low-level details that are not relevant for mobile application programming. + +## High-Level APIs + +The C++ `Module` class provides the high-level interface to load and execute a model from C++. It is responsible for loading the .pte file, configuring memory allocation and placement, and running the model. The Module constructor takes a file path and provides a simplified `forward()` method to run the model. + +In addition the Module class, the tensor extension provides an encapsulated interface to define and manage tensor memory. It provides the `TensorPtr` class, which is a "fat" smart pointer. It provides ownership over the tensor data and metadata, such as size and strides. The `make_tensor_ptr` and `from_blob` methods, defined in `tensor.h`, provide owning and non-owning tensor creation APIs, respectively. + +```cpp +#include +#include + +using namespace ::executorch::extension; + +// Load the model. +Module module("/path/to/model.pte"); + +// Create an input tensor. +float input[1 * 3 * 256 * 256]; +auto tensor = from_blob(input, {1, 3, 256, 256}); + +// Perform an inference. +const auto result = module.forward(tensor); + +if (result.ok()) { + // Retrieve the output data. + const auto output = result->at(0).toTensor().const_data_ptr(); +} +``` + +For more information on the Module class, see [Running an ExecuTorch Model Using the Module Extension in C++](extension-module.md). For information on high-level tensor APIs, see [Managing Tensor Memory in C++](extension-tensor.md). + +## Low-Level APIs + +Running a model using the low-level runtime APIs allows for a high-degree of control over memory allocation, placement, and loading. This allows for advanced use cases, such as placing allocations in specific memory banks or loading a model without a file system. For an end to end example using the low-level runtime APIs, see [Running an ExecuTorch Model in C++ Tutorial](running-a-model-cpp-tutorial.md). + +## Building with C++ + +ExecuTorch uses CMake as the primary build system. Inclusion of the module and tensor APIs are controlled by the `EXECUTORCH_BUILD_EXTENSION_MODULE` and `EXECUTORCH_BUILD_EXTENSION_TENSOR` CMake options. As these APIs may not be supported on embedded systems, they are disabled by default when building from source. The low-level API surface is always included. To link, add the `executorch` target as a CMake dependency, along with `executorch_module_static` and `executorch_tensor`, if desired. + +``` +# CMakeLists.txt +add_subdirectory("executorch") +... +target_link_libraries( + my_target + PRIVATE executorch + executorch_module_static + executorch_tensor + optimized_native_cpu_ops_lib + xnnpack_backend) +``` + +See [Building from Source](using-executorch-building-from-source.md) for more information on the CMake build process. + +## Next Steps + +- [Runtime API Reference](executorch-runtime-api-reference.md) for documentation on the available C++ runtime APIs. +- [Running an ExecuTorch Model Using the Module Extension in C++](extension-module.md) for information on the high-level Module API. +- [Managing Tensor Memory in C++](extension-tensor.md) for information on high-level tensor APIs. +- [Running an ExecuTorch Model in C++ Tutorial](running-a-model-cpp-tutorial.md) for information on the low-level runtime APIs. +- [Building from Source](using-executorch-building-from-source.md) for information on CMake build integration. diff --git a/docs/source/using-executorch-export.md b/docs/source/using-executorch-export.md index 7357ca4554d..62a52edf839 100644 --- a/docs/source/using-executorch-export.md +++ b/docs/source/using-executorch-export.md @@ -32,14 +32,14 @@ As part of the .pte file creation process, ExecuTorch identifies portions of the Commonly used hardware backends are listed below. For mobile, consider using XNNPACK for Android and XNNPACK or Core ML for iOS. To create a .pte file for a specific backend, pass the appropriate partitioner class to `to_edge_transform_and_lower`. See the appropriate backend documentation and the [Export and Lowering](#export-and-lowering) section below for more information. -- [XNNPACK (Mobile CPU)](native-delegates-executorch-xnnpack-delegate.md) -- [Core ML (iOS)](native-delegates-executorch-coreml-delegate.md) -- [Metal Performance Shaders (iOS GPU)](native-delegates-executorch-mps-delegate.md) -- [Vulkan (Android GPU)](native-delegates-executorch-vulkan-delegate.md) -- [Qualcomm NPU](native-delegates-executorch-qualcomm-delegate.md) -- [MediaTek NPU](native-delegates-executorch-mediatek-delegate.md) -- [Arm Ethos-U NPU](native-delegates-executorch-arm-ethos-u-delegate.md) -- [Cadence DSP](native-delegates-executorch-cadence-delegate.md) +- [XNNPACK (Mobile CPU)](backends-xnnpack.md) +- [Core ML (iOS)](backends-coreml.md) +- [Metal Performance Shaders (iOS GPU)](backends-mps.md) +- [Vulkan (Android GPU)](backends-vulkan.md) +- [Qualcomm NPU](backends-qualcomm.md) +- [MediaTek NPU](backends-mediatek.md) +- [Arm Ethos-U NPU](backends-arm-ethos-u.md) +- [Cadence DSP](backends-cadence.md) ## Model Preparation diff --git a/docs/source/using-executorch-ios.md b/docs/source/using-executorch-ios.md index 68599065f6a..029914eb498 100644 --- a/docs/source/using-executorch-ios.md +++ b/docs/source/using-executorch-ios.md @@ -106,7 +106,7 @@ git clone https://github.com/pytorch/executorch.git --depth 1 --recurse-submodul python3 -m venv .venv && source .venv/bin/activate && pip install --upgrade pip ``` -4. Install the required dependencies, including those needed for the backends like [Core ML](build-run-coreml.md) or [MPS](build-run-mps.md), if you plan to build them as well: +4. Install the required dependencies, including those needed for the backends like [Core ML](backends-coreml.md) or [MPS](backends-mps.md), if you plan to build them as well: ```bash ./install_executorch.sh --pybind coreml mps xnnpack diff --git a/docs/source/using-executorch-runtime-integration.md b/docs/source/using-executorch-runtime-integration.md index 6534e5855cf..08e071e59ab 100644 --- a/docs/source/using-executorch-runtime-integration.md +++ b/docs/source/using-executorch-runtime-integration.md @@ -10,7 +10,7 @@ Logging is sent to STDOUT and STDERR by default on host platforms, and is redire To configure log level when building from source, specify `EXECUTORCH_ENABLE_LOGGING` as on or off and `EXECUTORCH_LOG_LEVEL` as one of debug, info, error, or fatal. Logging is enabled by default in debug builds and disabled in release. Log level defaults to info. -See [Building from Source](TODO) for more information. +See [Building from Source](using-executorch-building-from-source.md) for more information. ``` cmake -b cmake-out -DEXECUTORCH_ENABLE_LOGGING=ON -DEXECUTORCH_LOG_LEVEL=DEBUG ... @@ -50,4 +50,4 @@ The choice of kernel library is transparent to the user when using mobile pre-bu By default, ExecuTorch ships with all supported operator kernels, allowing it to run any supported model at any precision. This comes with a binary size of several megabytes, which may be undesirable for production use cases or resource constrained systems. To minimize binary size, ExecuTorch provides selective build functionality, in order to include only the operators needed to run specific models. -Note the selective build only applies to the portable and optimized kernel libraries. Delegates do not participate in selective build and can be included or excluded by linking indivually. See [Kernel Library Selective Build](kernel-library-selective-build.md) for more information. \ No newline at end of file +Note the selective build only applies to the portable and optimized kernel libraries. Delegates do not participate in selective build and can be included or excluded by linking indivually. See [Kernel Library Selective Build](kernel-library-selective-build.md) for more information. diff --git a/docs/source/using-executorch-troubleshooting.md b/docs/source/using-executorch-troubleshooting.md index 406c1ed8347..16006802611 100644 --- a/docs/source/using-executorch-troubleshooting.md +++ b/docs/source/using-executorch-troubleshooting.md @@ -1,3 +1,20 @@ # Profiling and Debugging -Placeholder for top-level troubleshooting, profiling, and devtool docs \ No newline at end of file +To faciliate model and runtime integration, ExecuTorch provides tools to profile model resource utilization, numerics, and more. This section describes the available troubleshooting tools and steps to resolve issues when integrating ExecuTorch. + +## General Troubleshooting Steps + +- To troubleshoot failure of runtime API calls, such as loading or running a model, ensure that ExecuTorch framework logging is enabled. See [Logging](using-executorch-runtime-integration.md#logging) for more information. +- As a prelimatinary step to troubleshoot slow run times, ensure that performance testing is being done in a release build, and that the model is delegated. See [Inference is Slow](using-executorch-faqs.md#inference-is-slow--performance-troubleshooting) for more information. +- Check [Frequently Asked Questions](using-executorch-faqs.md) for common issues and questions encountered during install, model export, and runtime integration. + +## Developer Tools + +The ExecuTorch developer tools, or devtools, are a collection of tooling for troubleshooting model performance, numerics, and resource utilization. See [Introduction to the ExecuTorch Developer Tools](devtools-overview.md) for an overview of the available developer tools and usage. + +## Next Steps + +- [Frequently Asked Questions](using-executorch-faqs.md) for solutions to commonly encountered questions and issues. +- [Introduction to the ExecuTorch Developer Tools](runtime-profiling.md) for a high-level introduction to available developer tooling. +- [Using the ExecuTorch Developer Tools to Profile a Model](tutorials/devtools-integration-tutorial.md) for information on runtime performance profiling. +- [Inspector APIs](runtime-profiling.md) for reference material on trace inspector APIs.