[RFC]PyTorchTVM by Meteorix · Pull Request #25 · apache/tvm-rfcs

Meteorix · 2021-08-24T14:17:35Z

This RFC add a PyTorchTVM module to support: compile TorchScript to TVM and use accelerated module in PyTorch.

Discuss: https://discuss.tvm.apache.org/t/rfc-pytorchtvm-compile-torchscript-to-tvm-and-use-accelerated-module-in-pytorch/10873

masahi · 2021-08-24T20:28:38Z

cc @t-vi

junrushao · 2021-08-24T20:41:11Z

Thanks for the nice RFC @Meteorix! It is definitely super important direction to embed TVM into PyTorch. I will read it carefully later this week.

masahi · 2021-08-25T09:08:54Z

Is this supposed to be useful for accelerating PyTorch training?

Meteorix · 2021-08-26T03:36:00Z

Is this supposed to be useful for accelerating PyTorch training?

From my perspective, we are a little far from supporting PyTorch training. I can think of 2 possible ways:

use torchscript to train
use relay to train

It seems both ways are in early stage. Maybe we can further discuss the possibility, but not in the scope of this RFC.

t-vi · 2021-08-27T13:11:27Z

I wonder whether this would make the torch fallback op (apache/tvm#7401) more or less useful (it would depend on what you (plan to) do with unsupported ops). I am still pondering whether to close it or dust it off.

I should note that as far as I know NVidia has a TensorRT front end serving a similar goal and there also is one for ONNXRuntime-as-a-module (recently featured in the PyTorch blog). There may be useful design insights in how they work (or maybe they're too different to what you have in mind).

junrushao · 2021-08-30T00:08:02Z

+* Onnx cannot cover all models with dynamic control flow (e.g. for loop)
+* TensorRT can only accelerate some standard networks
+
+So we hope to use TVM to accelerate PyTorch model inference.


We should discuss more about the benefit of PyTorchTVM here

junrushao · 2021-08-30T00:16:44Z

+ images: List[torch.Tensor] = []
+ for path in image_path:
+ img = read_image(path)
+ images.append(img)
+ x = torch.stack(images).cuda().half()


Shall we conclude these lines to an abstract data loader to be consistent with our description? like, self.load_data()

maybe just keep the class simple?

junrushao · 2021-08-30T00:24:37Z

+# Prior art
+[prior-art]: #prior-art
+
+Our implementation is inspired by this RFC: https://discuss.tvm.apache.org/t/rfc-add-tensorflow-custom-op-to-embed-tvm-runtime-in-tensorflow-graph-and-session/4601


Suggested change
Our implementation is inspired by this RFC: https://discuss.tvm.apache.org/t/rfc-add-tensorflow-custom-op-to-embed-tvm-runtime-in-tensorflow-graph-and-session/4601
Our implementation is inspired by this RFC which allows TVM operators to be embedded into TensorFlow: https://discuss.tvm.apache.org/t/rfc-add-tensorflow-custom-op-to-embed-tvm-runtime-in-tensorflow-graph-and-session/4601. Beyond a single operator, our RFC further extends the scope to embedding arbitrary TVM graphs to TorchScript, and enables smooth integration and serving with both `torch.jit.trace` and `torch.jit.script` APIs.

To clarify: a major difference is that, the TF RFC only support embedding a TVM op, but not a TVM graph. And we also have an internal implementation that embeds TVM graph into TF.

Updated with clarification

junrushao · 2021-08-30T00:26:07Z

Thanks again for the contribution! I have no doubt that it is going to be hugely important piece of work. Just made some suggestions in terms of wording.

Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Meteorix · 2021-08-30T05:29:30Z

I wonder whether this would make the torch fallback op (apache/tvm#7401) more or less useful (it would depend on what you (plan to) do with unsupported ops). I am still pondering whether to close it or dust it off.

@t-vi It will help a lot. If we have the torch fallback op support, users will less likely get stuck in frontend conversion phase. We are looking forward to this feature. Could you complete the PR?

I should note that as far as I know NVidia has a TensorRT front end serving a similar goal and there also is one for ONNXRuntime-as-a-module (recently featured in the PyTorch blog). There may be useful design insights in how they work (or maybe they're too different to what you have in mind).

And thanks for this information. Actually, we are inspired by another NVIDIA work: TRTorch.

Co-authored-by: Junru Shao <junrushao1994@gmail.com>

suluner · 2021-09-09T07:51:16Z

Can this method split the graph automaticly given a torchscript, and convert the subgraph that tvm Relay IR can support or tvm outperforming pytorch?

junrushao

LGTM

junrushao · 2021-09-09T19:33:36Z

@masahi @t-vi Please take another look :-)

hogepodge

In general I support this effort. I'd like to see stronger integrations of TVM into adjacent projects like PyTorch. This looks like a good start.

One minor spelling point, but otherwise this looks good to me.

Laurawly

Great work!

tqchen · 2021-09-10T13:26:44Z

Thanks for the great work, a minor note on rendering. it would be great to add the codeblock type to the code so we have syntax highlights

Meteorix · 2021-09-13T08:59:45Z

@hogepodge @tqchen thanks for your advices. I have updated the PR.

junrushao · 2021-09-13T23:33:02Z

It seems that we all reach consensus and no more comments are raised for several days, let's get this RFC finally merged! 🔥 🎉

Create 0025-add-pytorch-tvm.md
fa4accf

masahi self-assigned this Aug 24, 2021

Meteorix mentioned this pull request Aug 25, 2021
[PyTorch]Add PyTorchTVM: compile torchscript to tvm and export as pytorch_op apache/tvm#8777
Merged

masahi reviewed Aug 27, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md

junrushao reviewed Aug 29, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

junrushao reviewed Aug 30, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

Meteorixand others added 3 commits August 30, 2021 13:06

Update rfcs/0025-add-pytorch-tvm.md
b473bc0
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Update rfcs/0025-add-pytorch-tvm.md
ce111a5
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Update rfcs/0025-add-pytorch-tvm.md
fd0a3bf
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Merge branch 'apache:main' into PyTorchTVM
9f340a9

Meteorixand others added 7 commits September 6, 2021 18:05

Update rfcs/0025-add-pytorch-tvm.md
308e600
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Update rfcs/0025-add-pytorch-tvm.md
6edf13a
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Update rfcs/0025-add-pytorch-tvm.md
f9b06f6

Update rfcs/0025-add-pytorch-tvm.md
29fe8ee
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Update rfcs/0025-add-pytorch-tvm.md
f66cee5
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Update rfcs/0025-add-pytorch-tvm.md
aa6e9e9
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

Update rfcs/0025-add-pytorch-tvm.md
a53a573
Co-authored-by: Junru Shao <junrushao1994@gmail.com>

junrushao approved these changes Sep 9, 2021
View reviewed changes

hogepodge suggested changes Sep 9, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

Laurawly approved these changes Sep 9, 2021
View reviewed changes

masahi approved these changes Sep 11, 2021
View reviewed changes

Update rfcs/0025-add-pytorch-tvm.md
5fab6c2

Meteorix commented Sep 13, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

Update rfcs/0025-add-pytorch-tvm.md
fa38e4e

Meteorix commented Sep 13, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

Update rfcs/0025-add-pytorch-tvm.md
4ed42b7

Meteorix commented Sep 13, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

Update rfcs/0025-add-pytorch-tvm.md
eeef5fe

Meteorix commented Sep 13, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

Update rfcs/0025-add-pytorch-tvm.md
3b31ffb

Meteorix commented Sep 13, 2021
View reviewed changes

Comment threadrfcs/0025-add-pytorch-tvm.md Outdated

Update rfcs/0025-add-pytorch-tvm.md
96e2165

junrushao merged commit 1ed4d70 into apache:mainSep 13, 2021

zhiqwang mentioned this pull request Nov 17, 2021
Roadmap of yolort zhiqwang/yolort#230
Open
10 tasks

masahi mentioned this pull request Jan 10, 2022
RFC: initial stab at TorchScript fallback apache/tvm#7401
Merged

Uh oh!

Conversation

Meteorix commented Aug 24, 2021

Uh oh!

masahi commented Aug 24, 2021

Uh oh!

junrushao commented Aug 24, 2021

Uh oh!

masahi commented Aug 25, 2021

Uh oh!

Meteorix commented Aug 26, 2021

Uh oh!

Uh oh!

t-vi commented Aug 27, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

junrushaoAug 30, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

junrushaoAug 30, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MeteorixSep 6, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

junrushaoAug 30, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MeteorixAug 30, 2021

Choose a reason for hiding this comment

Uh oh!

junrushaoSep 2, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

junrushao commented Aug 30, 2021

Uh oh!

Meteorix commented Aug 30, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

suluner commented Sep 9, 2021

Uh oh!

junrushao left a comment

Choose a reason for hiding this comment

Uh oh!

junrushao commented Sep 9, 2021

Uh oh!

hogepodge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Laurawly left a comment

Choose a reason for hiding this comment

Uh oh!

tqchen commented Sep 10, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Meteorix commented Sep 13, 2021

Uh oh!

junrushaoAug 30, 2021•
edited
Loading

junrushaoAug 30, 2021•
edited
Loading

Meteorix commented Aug 30, 2021•
edited
Loading