[TVM EP] Integrate tests for TVM EP into public onnxruntime CI by KJlaccHoeUM9l · Pull Request #10505 · microsoft/onnxruntime

Alexey Gladyshev (KJlaccHoeUM9l) · 2022-02-09T16:58:26Z

This PR is waiting for PR#10260 to be merged.

TVM EP is not currently being tested in public CI. Because of this, any changes made by third party contributors could potentially result in the TVM EP not working. To avoid this we have created this PR which is to add testing of TVM EP in public CI.

Notes:

Added support for bool types. A similar input tensor is used in this test;
Cleared post build actions for TVM EP;
Moved _ld_preload.py conditional extension from setup.py to CMake level;
Added support for TVM EP to run onnxruntime_test_python.py with build.py;
Added Dockerfile and workflow config to integrate TVM EP tests into azure pipeline;
Replaced TVM logging with ONNX RT logging. Additionally, implemented custom logger for TVM.

If there are things that were not taken into account for integrating TVM EP into public CI, please let us know.

…level

Alexey Gladyshev (KJlaccHoeUM9l) · 2022-02-16T10:12:40Z

Hello Edward Chen (@edgchen1)!
PR#10260 has been merged. I rebase to the current master so this PR can be reviewed.
Could you recheck our updates?

Thierry Moreau (tmoreau89) · 2022-02-16T17:42:33Z

Thank you Alexey Gladyshev (@KJlaccHoeUM9l) ! CC Xavier Dupré (@xadupre) - FYI; this PR adds CI for TVM EP; it's ready for review.

Edward Chen (edgchen1) · 2022-02-16T18:08:00Z

  endif()

+  add_definitions(-DTVM_LOG_CUSTOMIZE=1)
+  add_library(custom_logger STATIC ${ONNXRUNTIME_ROOT}/core/providers/tvm/custom_logging.cc)


can it be called tvm_custom_logger instead?

Edward Chen (edgchen1) · 2022-02-16T18:09:42Z

    set(USE_CUDA ON CACHE BOOL "Only defined for TVM" FORCE)
  endif()

+  add_definitions(-DTVM_LOG_CUSTOMIZE=1)


Suggested change

add_definitions(-DTVM_LOG_CUSTOMIZE=1)

add_compile_definitions(TVM_LOG_CUSTOMIZE=1)

Edward Chen (edgchen1) · 2022-02-16T18:24:57Z

+        displayName: 'Install python deps'
+        inputs:
+          script: |
+            python3 -m pip install decorator scipy


what are these used by? I only see a docker run later that does anything substantial. these dependencies won't be in the actual docker image right?

see comment below

Edward Chen (edgchen1) · 2022-02-16T18:26:52Z

@@ -0,0 +1,62 @@
+jobs:


has this pipeline been tested out yet?

see comment below

Edward Chen (edgchen1) · 2022-02-17T18:53:30Z

Created a build definition on our end to test this out.
https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=585915&view=results
Will need to use an existing agent pool.

FYI, I created a branch that mirrors this one that we can use to test the builds. I don't plan to update it outside of additional merges from this branch.
https://github.com/microsoft/onnxruntime/tree/edgchen1/tvm_ep_ci

Alexey Gladyshev (KJlaccHoeUM9l) · 2022-02-18T11:48:04Z

Edward Chen (@edgchen1), about using Dockerfile for TVM and azure pipeline.
Yes, you are absolutely right. The Python dependencies will be installed before the docker image is launched. Therefore, the dependencies will not be installed there.
Also you are right that this azure pipeline has not been tested yet. This testing involves running this pipeline in onnxruntime CI. Therefore, this is a "black box" for us, we cannot check the behavior locally.
In view of this, I reset the commit in which I deleted the Dockerfile.ubuntu_tvm. It seems that other EPs also install Python dependencies inside their Dockerfiles. Additionally, now the initial integration of TVM EP into CI. We use already existing tests. And to run them, we need to additionally deliver only two dependencies. However, in the future, when expanding the test suite, the list of these dependencies may grow, so the specified Dockerfile will grow and the difference will not be just one line.

Alexey Gladyshev (@KJlaccHoeUM9l) Regarding the separate Dockerfile, there is some infrastructure that caches CI build images in a container registry when we use get-docker-image-steps.yml. If the image is detected to be different (based on things like content of Dockerfile, Docker context directory), a new image is stored. This takes up additional space in the container registry. So if possible (if installing the dependencies at runtime is feasible), I would prefer to reuse the existing Dockerfile. If this changes in the future, we can switch to a separate Dockerfile then.

Something like this might work, if installation can be done without admin privileges:
docker run ... \
  onnxruntimecpubuild \
    /bin/bash -c \
      "install_tvm_test_dependencies.sh && \
      /opt/python/cp38-cp38/bin/python3 /onnxruntime_src/tools/ci_build/build.py ..."

Edward Chen (@edgchen1), thank you for the explanation! Resolved.

Alexey Gladyshev (KJlaccHoeUM9l) · 2022-02-18T11:48:53Z

/azp run Linux TVM CI Pipeline

azure-pipelines · 2022-02-18T11:48:57Z

Commenter does not have sufficient privileges for PR 10505 in repo microsoft/onnxruntime

Alexey Gladyshev (KJlaccHoeUM9l) · 2022-02-18T11:53:55Z

Edward Chen (@edgchen1), could you restart the pipeline?

Xavier Dupré (xadupre) · 2022-02-18T12:22:38Z

/azp run Linux TVM CI Pipeline

azure-pipelines · 2022-02-18T12:22:43Z

No pipelines are associated with this pull request.

Xavier Dupré (xadupre) · 2022-02-18T13:18:18Z

/azp Linux CPU CI Pipeline

azure-pipelines · 2022-02-18T13:18:23Z

Command 'Linux' is not supported by Azure Pipelines. Supported commands help: Get descriptions, examples and documentation about supported commands Example: help "command_name" list: List all pipelines for this repository using a comment. Example: "list" run: Run all pipelines or specific pipelines for this repository using a comment. Use this command by itself to trigger all related pipelines, or specify specific pipelines to run. Example: "run" or "run pipeline_name, pipeline_name, pipeline_name" where: Report back the Azure DevOps orgs that are related to this repository and org Example: "where" See additional documentation.

Xavier Dupré (xadupre) · 2022-02-18T14:27:17Z

/azp run Linux CPU CI Pipeline

azure-pipelines · 2022-02-18T14:27:29Z

Azure Pipelines successfully started running 1 pipeline(s).

…te build definition.

Edward Chen (edgchen1) · 2022-02-18T21:58:12Z

Here's the build I ran initially:
https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=586630&view=results

With some small changes it passes:
https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=586729&view=results

Here's a PR with the changes:
octoml#4

Move install_tvm_test_dependencies.sh out of Docker context dir, update build definition.

Alexey Gladyshev (KJlaccHoeUM9l) · 2022-02-19T09:31:33Z

Hello Edward Chen (@edgchen1)!
Thank you very much for your help! I have merged your changes. It seems that now we can run all pipelines?

Alexey Gladyshev (KJlaccHoeUM9l) · 2022-02-22T14:00:36Z

Hello Xavier Dupré (@xadupre)!
Could you run CI pipelines?

Xavier Dupré (xadupre) · 2022-02-22T15:49:30Z

/azp run Linux CPU CI Pipeline

azure-pipelines · 2022-02-22T15:49:45Z

Azure Pipelines successfully started running 1 pipeline(s).

Xavier Dupré (xadupre) · 2022-02-22T15:59:44Z

/azp run Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline

azure-pipelines · 2022-02-22T16:00:42Z

Azure Pipelines successfully started running 8 pipeline(s).

Xavier Dupré (xadupre) · 2022-02-22T17:00:52Z

/azp run Windows CPU CI Pipeline, Windows GPU CI Pipeline, Windows GPU TensorRT CI Pipeline, Windows WebAssembly CI Pipeline, orttraining-amd-gpu-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed, onnxruntime-python-checks-ci-pipeline

azure-pipelines · 2022-02-22T17:01:51Z

Azure Pipelines successfully started running 8 pipeline(s).

Edward Chen (edgchen1) · 2022-02-22T19:54:43Z

/azp run Linux TVM CI Pipeline

azure-pipelines · 2022-02-22T19:54:49Z

Azure Pipelines could not run because the pipeline triggers exclude this branch/path.

Edward Chen (edgchen1)

CI pipeline related changes look good to me

Alexey Gladyshev (KJlaccHoeUM9l) · 2022-02-24T15:19:19Z

Hello Xavier Dupré (@xadupre)!
Looks like we can merge this PR?

…soft#10505) * add support for bool type * add TVM EP support for tests * include TVM EP in python test pool * fix pylint * moved technical imports to a separate file * clean up post build actions & move _ld_preload.py extension to CMake level * add files for include TVM EP into CI * implement custom logger for TVM * replace TVM logging with ONNX RT logging * update link for TVM EP tutorial * clean up TVM EP cmake * add pybind auto enabling for TVM EP * fix blank spaces * code review fixes * replace print with comment * add list of EP without TVM EP * enable onnx tests * disable contrib ops and ml ops * reuse Dockerfile.ubuntu * Move install_tvm_test_dependencies.sh out of Docker context dir, update build definition. Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com> (cherry picked from commit 7dc7529)

Alexey Gladyshev (KJlaccHoeUM9l) changed the title ~~[TVM EP][WIP] Integrate tests for TVM EP into public onnxruntime CI~~ [TVM EP] Integrate tests for TVM EP into public onnxruntime CI Feb 10, 2022

This was referenced Feb 10, 2022

[TVM EP] upstream Apache/TVM after internal fixes #10515

Merged

[TVM EP] support of TVM Virtual Machine #10341

Merged

Edward Chen (edgchen1) reviewed Feb 12, 2022

View reviewed changes

Comment thread tools/ci_build/github/linux/docker/Dockerfile.ubuntu_tvm Outdated

Alexey Gladyshev (KJlaccHoeUM9l) force-pushed the agladyshev/dev/ci/integration branch from 2c3379c to 82ba76a Compare February 15, 2022 14:42

Alexey Gladyshev (KJlaccHoeUM9l) added 9 commits February 16, 2022 11:59

add support for bool type

705b10a

add TVM EP support for tests

6e9995f

include TVM EP in python test pool

b9ab525

fix pylint

52d98a6

moved technical imports to a separate file

76ffd54

clean up post build actions & move _ld_preload.py extension to CMake …

4076994

…level

add files for include TVM EP into CI

7cca3bc

implement custom logger for TVM

cce45cb

replace TVM logging with ONNX RT logging

ee61b1e

Alexey Gladyshev (KJlaccHoeUM9l) force-pushed the agladyshev/dev/ci/integration branch from 4a1350f to ee61b1e Compare February 16, 2022 09:16

Alexey Gladyshev (KJlaccHoeUM9l) added 3 commits February 16, 2022 12:21

update link for TVM EP tutorial

6c91555

clean up TVM EP cmake

4540477

add pybind auto enabling for TVM EP

aa7aa75

Edward Chen (edgchen1) reviewed Feb 16, 2022

View reviewed changes

Comment thread onnxruntime/core/providers/tvm/custom_logging.cc Outdated

Edward Chen (edgchen1) reviewed Feb 16, 2022

View reviewed changes

Edward Chen (edgchen1) requested review from Changming Sun (snnn) and Xavier Dupré (xadupre) February 16, 2022 18:28

Alexey Gladyshev (KJlaccHoeUM9l) added 2 commits February 17, 2022 10:14

fix blank spaces

d6a6046

code review fixes

7e4723e

Alexey Gladyshev (KJlaccHoeUM9l) force-pushed the agladyshev/dev/ci/integration branch from 5becef1 to 7e4723e Compare February 17, 2022 07:15

reuse Dockerfile.ubuntu

046c015

Move install_tvm_test_dependencies.sh out of Docker context dir, upda…

2c952c1

…te build definition.

Merge pull request #4 from microsoft/edgchen1/tvm_ep_ci_update

acec4dd

Move install_tvm_test_dependencies.sh out of Docker context dir, update build definition.

Edward Chen (edgchen1) approved these changes Feb 22, 2022

View reviewed changes

Valery Chernov (vvchernov) mentioned this pull request Feb 24, 2022

[TVM EP] code refactor #10655

Merged

Xavier Dupré (xadupre) merged commit 7dc7529 into microsoft:master Feb 24, 2022

	add_definitions(-DTVM_LOG_CUSTOMIZE=1)
	add_compile_definitions(TVM_LOG_CUSTOMIZE=1)

Conversation

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 16, 2022

Uh oh!

Thierry Moreau (tmoreau89) commented Feb 16, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Edward Chen (edgchen1) commented Feb 17, 2022

Uh oh!

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 18, 2022

Uh oh!

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 18, 2022

Uh oh!

azure-pipelines Bot commented Feb 18, 2022

Uh oh!

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 18, 2022

Uh oh!

Xavier Dupré (xadupre) commented Feb 18, 2022

Uh oh!

azure-pipelines Bot commented Feb 18, 2022

Uh oh!

Xavier Dupré (xadupre) commented Feb 18, 2022

Uh oh!

azure-pipelines Bot commented Feb 18, 2022

Uh oh!

Xavier Dupré (xadupre) commented Feb 18, 2022

Uh oh!

azure-pipelines Bot commented Feb 18, 2022

Uh oh!

Edward Chen (edgchen1) commented Feb 18, 2022

Uh oh!

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 19, 2022

Uh oh!

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 22, 2022

Uh oh!

Xavier Dupré (xadupre) commented Feb 22, 2022

Uh oh!

azure-pipelines Bot commented Feb 22, 2022

Uh oh!

Xavier Dupré (xadupre) commented Feb 22, 2022

Uh oh!

azure-pipelines Bot commented Feb 22, 2022

Uh oh!

Xavier Dupré (xadupre) commented Feb 22, 2022

Uh oh!

azure-pipelines Bot commented Feb 22, 2022

Uh oh!

Edward Chen (edgchen1) commented Feb 22, 2022

Uh oh!

azure-pipelines Bot commented Feb 22, 2022

Uh oh!

Edward Chen (edgchen1) left a comment

Choose a reason for hiding this comment

Uh oh!

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 24, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Alexey Gladyshev (KJlaccHoeUM9l) commented Feb 9, 2022 •

edited

Loading