Introduce the Arm(R) Ethos(TM)-U Cascading Scheduler by mbaret · Pull Request #37 · apache/tvm-rfcs

mbaret · 2021-09-27T15:12:31Z

No description provided.

mbaret · 2021-09-27T15:18:28Z

cc @jacobbohlin @NicolaLancellotti @manupa-arm @csullivan @junrushao1994 @mbs-octoml @jroesch

junrushao · 2021-09-29T19:59:15Z

Thanks for the RFC!

The TE scheduling part (rolling-buffer) looks good to me.
Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you
We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

areusch

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:

a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
the Ethos-U-specific logic.

I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

areusch · 2021-09-29T16:11:39Z

-$$stripe_{in} = {M} \cdot {stripe_{out}}$$
+![meta-schedule-workflow](../resources/cascading-formula-1.png)

 Let's briefly consider how to derive such a transform matrix for a 3x3 unstrided, undilated and unpadded NHWC convolution. Immediately, the '3x3' kernel tells us something important: a single element in the output depends on 3x3 elements in the height/width of the input. If we were instead to consider a 2x2 region of the output in the height/width dimensions, we'd then need a 4x4 region in the input. So in general, the rule is that we need 2 more elements in height and width when calculating the dependencies of an output stripe. It can be shown that more generally this number is the kernel_size-1 in each axis. Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel. This is because the input channel axis is a reduction axis in a convolution, in a sense it isn't 'reflected' in the output. Combining these two observations, we arrive at the following transform matrix:


Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel.
just curious: what about depthwise convolutions?

In a depthwise you're right that this will be different, but depthwise will also use a different transform matrix. So here we're just working out the transform matrix for a standard 2D convolution.

mbaret · 2021-10-06T16:29:31Z

Thanks for taking a look @junrushao1994!

Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you

For Ethos-U, we have a different compilation flow to 'standard' TVM. In particular, those operators which are being offloaded onto Ethos-U don't go through FuseOps but instead are lowered into TE as one large graph. This is what allows us to avoid the problems of Relay level fusion. I agree that were we to want to generalize this technique further, we would need a solution for this.

We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

I hadn't actually noticed the affine analysis infra before, so thanks for pointing me towards this. From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR. Perhaps this could be better discussed on a draft PR to demonstrate specifically what we do with the affine transforms and whether the existing infra would be able to achieve the same effect.

junrushao · 2021-10-06T17:16:38Z

Thank you for your response @mbaret 😄

From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR

For sure, we are happy to work together and potentially improve the affine analysis infrastructure. Note that the iter-affine-map is located under tvm/arith/iter_affine_map.h, so it's not deeply coupled with TIR, but only reuses several TIR expressions.

CC'ed the author of affine analysis in TVM: @tqchen @spectrometerHBH @Hzfengsy

Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret · 2021-11-02T17:08:20Z

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:
1. a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
2. the Ethos-U-specific logic.
I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

Apologies for the late reply @areusch. Due to some of the uncertainties around implementing this generically (Relay fusion/conflict with AutoTVM/deriving matrices), I think it's more pragmatic to have everything initially scoped just to Ethos-U. But within that implementation, all the truly Ethos-U specific logic is quite self-contained (pretty much entirely in the performance heuristics and matrix definitions). Therefore, should this need reusing in future we could probably lift it out of the Ethos-U namespace. I think this would be very doable as the cascader itself is quite well decoupled from the rest of the Ethos-U compiler, exposing only a TE Schedule -> TE Schedule interface. With some effort, although hopefully not too much, that interface could likely be tweaked to be TensorIR -> TensorIR.

mbs-octoml

Hi Mike, thx for such a nicely written RFC, you're upping the bar.

I think we all agreed in our last live conversation:

Though the S-TIR machinery is now in main and could be a viable medium instead of TE that transition will be left to future work.
The transition to TE should hopefully not require duplicating the ScheduleBuilder visitor -- I think someone else at ARM was getting that in place. Sorry I can't keep up.
I'm hoping we can make constants first class so there's no need for any side inputs. Eg just look for them bound as globals in the IRModule. But that's a refactor which will probably come after this work.
There's an ongoing tension between working in Relay+TIR vs TIR-only, but the work-exclusively-in-TE/TIR approach is consistent with both the current AOT flow and the USMP analysis work.

If I got that right perhaps capture it in the 'alternates considered' or someplace.

I can't say anything about possible reuse of the affine x-form machinery when doing the abstract interpretation for the memory footprint. Sounds like it might be similar to the S-TIR situation: maybe not now but can go back later.

So LGTM from me.

areusch

thanks @mbaret we should merge this now :)

tqchen · 2022-01-27T14:17:03Z

To followup on this RFC, @mbaret would be great to start followup with a rolling-buffer or related primitive in TensorIR. So we can smoothly transition the solution as we start to migrate to TIR schedule

mbaret · 2022-02-01T10:23:09Z

We don't have any plans in the short term to look at extending the rolling buffer primitive into TensorIR, but I'd be happy to provide support/assistance to anyone interested in implementing it. I would hope the effort would not be too great as it's already implemented as a TIR pass.

mbaret · 2021-09-27T15:12:31Z

No description provided.

mbaret · 2021-09-27T15:18:28Z

cc @jacobbohlin @NicolaLancellotti @manupa-arm @csullivan @junrushao1994 @mbs-octoml @jroesch

junrushao · 2021-09-29T19:59:15Z

Thanks for the RFC!

The TE scheduling part (rolling-buffer) looks good to me.
Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you
We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

areusch

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:

a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
the Ethos-U-specific logic.

I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

areusch · 2021-09-29T16:11:39Z

-$$stripe_{in} = {M} \cdot {stripe_{out}}$$
+![meta-schedule-workflow](../resources/cascading-formula-1.png)

 Let's briefly consider how to derive such a transform matrix for a 3x3 unstrided, undilated and unpadded NHWC convolution. Immediately, the '3x3' kernel tells us something important: a single element in the output depends on 3x3 elements in the height/width of the input. If we were instead to consider a 2x2 region of the output in the height/width dimensions, we'd then need a 4x4 region in the input. So in general, the rule is that we need 2 more elements in height and width when calculating the dependencies of an output stripe. It can be shown that more generally this number is the kernel_size-1 in each axis. Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel. This is because the input channel axis is a reduction axis in a convolution, in a sense it isn't 'reflected' in the output. Combining these two observations, we arrive at the following transform matrix:


Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel.
just curious: what about depthwise convolutions?

In a depthwise you're right that this will be different, but depthwise will also use a different transform matrix. So here we're just working out the transform matrix for a standard 2D convolution.

mbaret · 2021-10-06T16:29:31Z

Thanks for taking a look @junrushao1994!

Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you

For Ethos-U, we have a different compilation flow to 'standard' TVM. In particular, those operators which are being offloaded onto Ethos-U don't go through FuseOps but instead are lowered into TE as one large graph. This is what allows us to avoid the problems of Relay level fusion. I agree that were we to want to generalize this technique further, we would need a solution for this.

We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

I hadn't actually noticed the affine analysis infra before, so thanks for pointing me towards this. From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR. Perhaps this could be better discussed on a draft PR to demonstrate specifically what we do with the affine transforms and whether the existing infra would be able to achieve the same effect.

junrushao · 2021-10-06T17:16:38Z

Thank you for your response @mbaret 😄

From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR

For sure, we are happy to work together and potentially improve the affine analysis infrastructure. Note that the iter-affine-map is located under tvm/arith/iter_affine_map.h, so it's not deeply coupled with TIR, but only reuses several TIR expressions.

CC'ed the author of affine analysis in TVM: @tqchen @spectrometerHBH @Hzfengsy

Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret · 2021-11-02T17:08:20Z

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:
1. a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
2. the Ethos-U-specific logic.
I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

Apologies for the late reply @areusch. Due to some of the uncertainties around implementing this generically (Relay fusion/conflict with AutoTVM/deriving matrices), I think it's more pragmatic to have everything initially scoped just to Ethos-U. But within that implementation, all the truly Ethos-U specific logic is quite self-contained (pretty much entirely in the performance heuristics and matrix definitions). Therefore, should this need reusing in future we could probably lift it out of the Ethos-U namespace. I think this would be very doable as the cascader itself is quite well decoupled from the rest of the Ethos-U compiler, exposing only a TE Schedule -> TE Schedule interface. With some effort, although hopefully not too much, that interface could likely be tweaked to be TensorIR -> TensorIR.

mbs-octoml

Hi Mike, thx for such a nicely written RFC, you're upping the bar.

I think we all agreed in our last live conversation:

Though the S-TIR machinery is now in main and could be a viable medium instead of TE that transition will be left to future work.
The transition to TE should hopefully not require duplicating the ScheduleBuilder visitor -- I think someone else at ARM was getting that in place. Sorry I can't keep up.
I'm hoping we can make constants first class so there's no need for any side inputs. Eg just look for them bound as globals in the IRModule. But that's a refactor which will probably come after this work.
There's an ongoing tension between working in Relay+TIR vs TIR-only, but the work-exclusively-in-TE/TIR approach is consistent with both the current AOT flow and the USMP analysis work.

If I got that right perhaps capture it in the 'alternates considered' or someplace.

I can't say anything about possible reuse of the affine x-form machinery when doing the abstract interpretation for the memory footprint. Sounds like it might be similar to the S-TIR situation: maybe not now but can go back later.

So LGTM from me.

areusch

thanks @mbaret we should merge this now :)

tqchen · 2022-01-27T14:17:03Z

To followup on this RFC, @mbaret would be great to start followup with a rolling-buffer or related primitive in TensorIR. So we can smoothly transition the solution as we start to migrate to TIR schedule

mbaret · 2022-02-01T10:23:09Z

We don't have any plans in the short term to look at extending the rolling buffer primitive into TensorIR, but I'd be happy to provide support/assistance to anyone interested in implementing it. I would hope the effort would not be too great as it's already implemented as a TIR pass.

mbaret · 2021-09-27T15:12:31Z

No description provided.

mbaret · 2021-09-27T15:18:28Z

cc @jacobbohlin @NicolaLancellotti @manupa-arm @csullivan @junrushao1994 @mbs-octoml @jroesch

junrushao · 2021-09-29T19:59:15Z

Thanks for the RFC!

The TE scheduling part (rolling-buffer) looks good to me.
Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you
We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

areusch

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:

a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
the Ethos-U-specific logic.

I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

areusch · 2021-09-29T16:11:39Z

-$$stripe_{in} = {M} \cdot {stripe_{out}}$$
+![meta-schedule-workflow](../resources/cascading-formula-1.png)

 Let's briefly consider how to derive such a transform matrix for a 3x3 unstrided, undilated and unpadded NHWC convolution. Immediately, the '3x3' kernel tells us something important: a single element in the output depends on 3x3 elements in the height/width of the input. If we were instead to consider a 2x2 region of the output in the height/width dimensions, we'd then need a 4x4 region in the input. So in general, the rule is that we need 2 more elements in height and width when calculating the dependencies of an output stripe. It can be shown that more generally this number is the kernel_size-1 in each axis. Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel. This is because the input channel axis is a reduction axis in a convolution, in a sense it isn't 'reflected' in the output. Combining these two observations, we arrive at the following transform matrix:


Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel.
just curious: what about depthwise convolutions?

In a depthwise you're right that this will be different, but depthwise will also use a different transform matrix. So here we're just working out the transform matrix for a standard 2D convolution.

mbaret · 2021-10-06T16:29:31Z

Thanks for taking a look @junrushao1994!

Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you

For Ethos-U, we have a different compilation flow to 'standard' TVM. In particular, those operators which are being offloaded onto Ethos-U don't go through FuseOps but instead are lowered into TE as one large graph. This is what allows us to avoid the problems of Relay level fusion. I agree that were we to want to generalize this technique further, we would need a solution for this.

We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

I hadn't actually noticed the affine analysis infra before, so thanks for pointing me towards this. From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR. Perhaps this could be better discussed on a draft PR to demonstrate specifically what we do with the affine transforms and whether the existing infra would be able to achieve the same effect.

junrushao · 2021-10-06T17:16:38Z

Thank you for your response @mbaret 😄

From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR

For sure, we are happy to work together and potentially improve the affine analysis infrastructure. Note that the iter-affine-map is located under tvm/arith/iter_affine_map.h, so it's not deeply coupled with TIR, but only reuses several TIR expressions.

CC'ed the author of affine analysis in TVM: @tqchen @spectrometerHBH @Hzfengsy

Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret · 2021-11-02T17:08:20Z

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:
1. a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
2. the Ethos-U-specific logic.
I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

Apologies for the late reply @areusch. Due to some of the uncertainties around implementing this generically (Relay fusion/conflict with AutoTVM/deriving matrices), I think it's more pragmatic to have everything initially scoped just to Ethos-U. But within that implementation, all the truly Ethos-U specific logic is quite self-contained (pretty much entirely in the performance heuristics and matrix definitions). Therefore, should this need reusing in future we could probably lift it out of the Ethos-U namespace. I think this would be very doable as the cascader itself is quite well decoupled from the rest of the Ethos-U compiler, exposing only a TE Schedule -> TE Schedule interface. With some effort, although hopefully not too much, that interface could likely be tweaked to be TensorIR -> TensorIR.

mbs-octoml

Hi Mike, thx for such a nicely written RFC, you're upping the bar.

I think we all agreed in our last live conversation:

Though the S-TIR machinery is now in main and could be a viable medium instead of TE that transition will be left to future work.
The transition to TE should hopefully not require duplicating the ScheduleBuilder visitor -- I think someone else at ARM was getting that in place. Sorry I can't keep up.
I'm hoping we can make constants first class so there's no need for any side inputs. Eg just look for them bound as globals in the IRModule. But that's a refactor which will probably come after this work.
There's an ongoing tension between working in Relay+TIR vs TIR-only, but the work-exclusively-in-TE/TIR approach is consistent with both the current AOT flow and the USMP analysis work.

If I got that right perhaps capture it in the 'alternates considered' or someplace.

I can't say anything about possible reuse of the affine x-form machinery when doing the abstract interpretation for the memory footprint. Sounds like it might be similar to the S-TIR situation: maybe not now but can go back later.

So LGTM from me.

areusch

thanks @mbaret we should merge this now :)

tqchen · 2022-01-27T14:17:03Z

To followup on this RFC, @mbaret would be great to start followup with a rolling-buffer or related primitive in TensorIR. So we can smoothly transition the solution as we start to migrate to TIR schedule

mbaret · 2022-02-01T10:23:09Z

We don't have any plans in the short term to look at extending the rolling buffer primitive into TensorIR, but I'd be happy to provide support/assistance to anyone interested in implementing it. I would hope the effort would not be too great as it's already implemented as a TIR pass.

mbaret · 2021-09-27T15:12:31Z

No description provided.

mbaret · 2021-09-27T15:18:28Z

cc @jacobbohlin @NicolaLancellotti @manupa-arm @csullivan @junrushao1994 @mbs-octoml @jroesch

junrushao · 2021-09-29T19:59:15Z

Thanks for the RFC!

The TE scheduling part (rolling-buffer) looks good to me.
Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you
We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

areusch

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:

a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
the Ethos-U-specific logic.

I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

areusch · 2021-09-29T16:11:39Z

-$$stripe_{in} = {M} \cdot {stripe_{out}}$$
+![meta-schedule-workflow](../resources/cascading-formula-1.png)

 Let's briefly consider how to derive such a transform matrix for a 3x3 unstrided, undilated and unpadded NHWC convolution. Immediately, the '3x3' kernel tells us something important: a single element in the output depends on 3x3 elements in the height/width of the input. If we were instead to consider a 2x2 region of the output in the height/width dimensions, we'd then need a 4x4 region in the input. So in general, the rule is that we need 2 more elements in height and width when calculating the dependencies of an output stripe. It can be shown that more generally this number is the kernel_size-1 in each axis. Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel. This is because the input channel axis is a reduction axis in a convolution, in a sense it isn't 'reflected' in the output. Combining these two observations, we arrive at the following transform matrix:


Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel.
just curious: what about depthwise convolutions?

In a depthwise you're right that this will be different, but depthwise will also use a different transform matrix. So here we're just working out the transform matrix for a standard 2D convolution.

mbaret · 2021-10-06T16:29:31Z

Thanks for taking a look @junrushao1994!

Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you

For Ethos-U, we have a different compilation flow to 'standard' TVM. In particular, those operators which are being offloaded onto Ethos-U don't go through FuseOps but instead are lowered into TE as one large graph. This is what allows us to avoid the problems of Relay level fusion. I agree that were we to want to generalize this technique further, we would need a solution for this.

We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

I hadn't actually noticed the affine analysis infra before, so thanks for pointing me towards this. From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR. Perhaps this could be better discussed on a draft PR to demonstrate specifically what we do with the affine transforms and whether the existing infra would be able to achieve the same effect.

junrushao · 2021-10-06T17:16:38Z

Thank you for your response @mbaret 😄

From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR

For sure, we are happy to work together and potentially improve the affine analysis infrastructure. Note that the iter-affine-map is located under tvm/arith/iter_affine_map.h, so it's not deeply coupled with TIR, but only reuses several TIR expressions.

CC'ed the author of affine analysis in TVM: @tqchen @spectrometerHBH @Hzfengsy

Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret · 2021-11-02T17:08:20Z

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:
1. a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
2. the Ethos-U-specific logic.
I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

Apologies for the late reply @areusch. Due to some of the uncertainties around implementing this generically (Relay fusion/conflict with AutoTVM/deriving matrices), I think it's more pragmatic to have everything initially scoped just to Ethos-U. But within that implementation, all the truly Ethos-U specific logic is quite self-contained (pretty much entirely in the performance heuristics and matrix definitions). Therefore, should this need reusing in future we could probably lift it out of the Ethos-U namespace. I think this would be very doable as the cascader itself is quite well decoupled from the rest of the Ethos-U compiler, exposing only a TE Schedule -> TE Schedule interface. With some effort, although hopefully not too much, that interface could likely be tweaked to be TensorIR -> TensorIR.

mbs-octoml

Hi Mike, thx for such a nicely written RFC, you're upping the bar.

I think we all agreed in our last live conversation:

Though the S-TIR machinery is now in main and could be a viable medium instead of TE that transition will be left to future work.
The transition to TE should hopefully not require duplicating the ScheduleBuilder visitor -- I think someone else at ARM was getting that in place. Sorry I can't keep up.
I'm hoping we can make constants first class so there's no need for any side inputs. Eg just look for them bound as globals in the IRModule. But that's a refactor which will probably come after this work.
There's an ongoing tension between working in Relay+TIR vs TIR-only, but the work-exclusively-in-TE/TIR approach is consistent with both the current AOT flow and the USMP analysis work.

If I got that right perhaps capture it in the 'alternates considered' or someplace.

I can't say anything about possible reuse of the affine x-form machinery when doing the abstract interpretation for the memory footprint. Sounds like it might be similar to the S-TIR situation: maybe not now but can go back later.

So LGTM from me.

areusch

thanks @mbaret we should merge this now :)

tqchen · 2022-01-27T14:17:03Z

To followup on this RFC, @mbaret would be great to start followup with a rolling-buffer or related primitive in TensorIR. So we can smoothly transition the solution as we start to migrate to TIR schedule

mbaret · 2022-02-01T10:23:09Z

We don't have any plans in the short term to look at extending the rolling buffer primitive into TensorIR, but I'd be happy to provide support/assistance to anyone interested in implementing it. I would hope the effort would not be too great as it's already implemented as a TIR pass.

mbaret · 2021-09-27T15:12:31Z

No description provided.

mbaret · 2021-09-27T15:18:28Z

cc @jacobbohlin @NicolaLancellotti @manupa-arm @csullivan @junrushao1994 @mbs-octoml @jroesch

junrushao · 2021-09-29T19:59:15Z

Thanks for the RFC!

The TE scheduling part (rolling-buffer) looks good to me.
Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you
We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

areusch

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:

a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
the Ethos-U-specific logic.

I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

areusch · 2021-09-29T16:11:39Z

-$$stripe_{in} = {M} \cdot {stripe_{out}}$$
+![meta-schedule-workflow](../resources/cascading-formula-1.png)

 Let's briefly consider how to derive such a transform matrix for a 3x3 unstrided, undilated and unpadded NHWC convolution. Immediately, the '3x3' kernel tells us something important: a single element in the output depends on 3x3 elements in the height/width of the input. If we were instead to consider a 2x2 region of the output in the height/width dimensions, we'd then need a 4x4 region in the input. So in general, the rule is that we need 2 more elements in height and width when calculating the dependencies of an output stripe. It can be shown that more generally this number is the kernel_size-1 in each axis. Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel. This is because the input channel axis is a reduction axis in a convolution, in a sense it isn't 'reflected' in the output. Combining these two observations, we arrive at the following transform matrix:


Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel.
just curious: what about depthwise convolutions?

In a depthwise you're right that this will be different, but depthwise will also use a different transform matrix. So here we're just working out the transform matrix for a standard 2D convolution.

mbaret · 2021-10-06T16:29:31Z

Thanks for taking a look @junrushao1994!

Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you

For Ethos-U, we have a different compilation flow to 'standard' TVM. In particular, those operators which are being offloaded onto Ethos-U don't go through FuseOps but instead are lowered into TE as one large graph. This is what allows us to avoid the problems of Relay level fusion. I agree that were we to want to generalize this technique further, we would need a solution for this.

We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

I hadn't actually noticed the affine analysis infra before, so thanks for pointing me towards this. From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR. Perhaps this could be better discussed on a draft PR to demonstrate specifically what we do with the affine transforms and whether the existing infra would be able to achieve the same effect.

junrushao · 2021-10-06T17:16:38Z

Thank you for your response @mbaret 😄

From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR

For sure, we are happy to work together and potentially improve the affine analysis infrastructure. Note that the iter-affine-map is located under tvm/arith/iter_affine_map.h, so it's not deeply coupled with TIR, but only reuses several TIR expressions.

CC'ed the author of affine analysis in TVM: @tqchen @spectrometerHBH @Hzfengsy

Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret · 2021-11-02T17:08:20Z

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:
1. a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
2. the Ethos-U-specific logic.
I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

Apologies for the late reply @areusch. Due to some of the uncertainties around implementing this generically (Relay fusion/conflict with AutoTVM/deriving matrices), I think it's more pragmatic to have everything initially scoped just to Ethos-U. But within that implementation, all the truly Ethos-U specific logic is quite self-contained (pretty much entirely in the performance heuristics and matrix definitions). Therefore, should this need reusing in future we could probably lift it out of the Ethos-U namespace. I think this would be very doable as the cascader itself is quite well decoupled from the rest of the Ethos-U compiler, exposing only a TE Schedule -> TE Schedule interface. With some effort, although hopefully not too much, that interface could likely be tweaked to be TensorIR -> TensorIR.

mbs-octoml

Hi Mike, thx for such a nicely written RFC, you're upping the bar.

I think we all agreed in our last live conversation:

Though the S-TIR machinery is now in main and could be a viable medium instead of TE that transition will be left to future work.
The transition to TE should hopefully not require duplicating the ScheduleBuilder visitor -- I think someone else at ARM was getting that in place. Sorry I can't keep up.
I'm hoping we can make constants first class so there's no need for any side inputs. Eg just look for them bound as globals in the IRModule. But that's a refactor which will probably come after this work.
There's an ongoing tension between working in Relay+TIR vs TIR-only, but the work-exclusively-in-TE/TIR approach is consistent with both the current AOT flow and the USMP analysis work.

If I got that right perhaps capture it in the 'alternates considered' or someplace.

I can't say anything about possible reuse of the affine x-form machinery when doing the abstract interpretation for the memory footprint. Sounds like it might be similar to the S-TIR situation: maybe not now but can go back later.

So LGTM from me.

areusch

thanks @mbaret we should merge this now :)

tqchen · 2022-01-27T14:17:03Z

To followup on this RFC, @mbaret would be great to start followup with a rolling-buffer or related primitive in TensorIR. So we can smoothly transition the solution as we start to migrate to TIR schedule

mbaret · 2022-02-01T10:23:09Z

We don't have any plans in the short term to look at extending the rolling buffer primitive into TensorIR, but I'd be happy to provide support/assistance to anyone interested in implementing it. I would hope the effort would not be too great as it's already implemented as a TIR pass.

mbaret · 2021-09-27T15:12:31Z

No description provided.

mbaret · 2021-09-27T15:18:28Z

cc @jacobbohlin @NicolaLancellotti @manupa-arm @csullivan @junrushao1994 @mbs-octoml @jroesch

junrushao · 2021-09-29T19:59:15Z

Thanks for the RFC!

The TE scheduling part (rolling-buffer) looks good to me.
Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you
We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

areusch

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:

a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
the Ethos-U-specific logic.

I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

areusch · 2021-09-29T16:11:39Z

-$$stripe_{in} = {M} \cdot {stripe_{out}}$$
+![meta-schedule-workflow](../resources/cascading-formula-1.png)

 Let's briefly consider how to derive such a transform matrix for a 3x3 unstrided, undilated and unpadded NHWC convolution. Immediately, the '3x3' kernel tells us something important: a single element in the output depends on 3x3 elements in the height/width of the input. If we were instead to consider a 2x2 region of the output in the height/width dimensions, we'd then need a 4x4 region in the input. So in general, the rule is that we need 2 more elements in height and width when calculating the dependencies of an output stripe. It can be shown that more generally this number is the kernel_size-1 in each axis. Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel. This is because the input channel axis is a reduction axis in a convolution, in a sense it isn't 'reflected' in the output. Combining these two observations, we arrive at the following transform matrix:


Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel.
just curious: what about depthwise convolutions?

In a depthwise you're right that this will be different, but depthwise will also use a different transform matrix. So here we're just working out the transform matrix for a standard 2D convolution.

mbaret · 2021-10-06T16:29:31Z

Thanks for taking a look @junrushao1994!

Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you

For Ethos-U, we have a different compilation flow to 'standard' TVM. In particular, those operators which are being offloaded onto Ethos-U don't go through FuseOps but instead are lowered into TE as one large graph. This is what allows us to avoid the problems of Relay level fusion. I agree that were we to want to generalize this technique further, we would need a solution for this.

We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

I hadn't actually noticed the affine analysis infra before, so thanks for pointing me towards this. From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR. Perhaps this could be better discussed on a draft PR to demonstrate specifically what we do with the affine transforms and whether the existing infra would be able to achieve the same effect.

junrushao · 2021-10-06T17:16:38Z

Thank you for your response @mbaret 😄

From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR

For sure, we are happy to work together and potentially improve the affine analysis infrastructure. Note that the iter-affine-map is located under tvm/arith/iter_affine_map.h, so it's not deeply coupled with TIR, but only reuses several TIR expressions.

CC'ed the author of affine analysis in TVM: @tqchen @spectrometerHBH @Hzfengsy

Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret · 2021-11-02T17:08:20Z

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:
1. a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
2. the Ethos-U-specific logic.
I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

Apologies for the late reply @areusch. Due to some of the uncertainties around implementing this generically (Relay fusion/conflict with AutoTVM/deriving matrices), I think it's more pragmatic to have everything initially scoped just to Ethos-U. But within that implementation, all the truly Ethos-U specific logic is quite self-contained (pretty much entirely in the performance heuristics and matrix definitions). Therefore, should this need reusing in future we could probably lift it out of the Ethos-U namespace. I think this would be very doable as the cascader itself is quite well decoupled from the rest of the Ethos-U compiler, exposing only a TE Schedule -> TE Schedule interface. With some effort, although hopefully not too much, that interface could likely be tweaked to be TensorIR -> TensorIR.

mbs-octoml

Hi Mike, thx for such a nicely written RFC, you're upping the bar.

I think we all agreed in our last live conversation:

Though the S-TIR machinery is now in main and could be a viable medium instead of TE that transition will be left to future work.
The transition to TE should hopefully not require duplicating the ScheduleBuilder visitor -- I think someone else at ARM was getting that in place. Sorry I can't keep up.
I'm hoping we can make constants first class so there's no need for any side inputs. Eg just look for them bound as globals in the IRModule. But that's a refactor which will probably come after this work.
There's an ongoing tension between working in Relay+TIR vs TIR-only, but the work-exclusively-in-TE/TIR approach is consistent with both the current AOT flow and the USMP analysis work.

If I got that right perhaps capture it in the 'alternates considered' or someplace.

I can't say anything about possible reuse of the affine x-form machinery when doing the abstract interpretation for the memory footprint. Sounds like it might be similar to the S-TIR situation: maybe not now but can go back later.

So LGTM from me.

areusch

thanks @mbaret we should merge this now :)

tqchen · 2022-01-27T14:17:03Z

To followup on this RFC, @mbaret would be great to start followup with a rolling-buffer or related primitive in TensorIR. So we can smoothly transition the solution as we start to migrate to TIR schedule

mbaret · 2022-02-01T10:23:09Z

We don't have any plans in the short term to look at extending the rolling buffer primitive into TensorIR, but I'd be happy to provide support/assistance to anyone interested in implementing it. I would hope the effort would not be too great as it's already implemented as a TIR pass.

mbaret · 2021-09-27T15:12:31Z

No description provided.

mbaret · 2021-09-27T15:18:28Z

cc @jacobbohlin @NicolaLancellotti @manupa-arm @csullivan @junrushao1994 @mbs-octoml @jroesch

junrushao · 2021-09-29T19:59:15Z

Thanks for the RFC!

The TE scheduling part (rolling-buffer) looks good to me.
Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you
We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

areusch

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:

a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
the Ethos-U-specific logic.

I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

areusch · 2021-09-29T16:11:39Z

-$$stripe_{in} = {M} \cdot {stripe_{out}}$$
+![meta-schedule-workflow](../resources/cascading-formula-1.png)

 Let's briefly consider how to derive such a transform matrix for a 3x3 unstrided, undilated and unpadded NHWC convolution. Immediately, the '3x3' kernel tells us something important: a single element in the output depends on 3x3 elements in the height/width of the input. If we were instead to consider a 2x2 region of the output in the height/width dimensions, we'd then need a 4x4 region in the input. So in general, the rule is that we need 2 more elements in height and width when calculating the dependencies of an output stripe. It can be shown that more generally this number is the kernel_size-1 in each axis. Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel. This is because the input channel axis is a reduction axis in a convolution, in a sense it isn't 'reflected' in the output. Combining these two observations, we arrive at the following transform matrix:


Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel.
just curious: what about depthwise convolutions?

In a depthwise you're right that this will be different, but depthwise will also use a different transform matrix. So here we're just working out the transform matrix for a standard 2D convolution.

mbaret · 2021-10-06T16:29:31Z

Thanks for taking a look @junrushao1994!

Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you

For Ethos-U, we have a different compilation flow to 'standard' TVM. In particular, those operators which are being offloaded onto Ethos-U don't go through FuseOps but instead are lowered into TE as one large graph. This is what allows us to avoid the problems of Relay level fusion. I agree that were we to want to generalize this technique further, we would need a solution for this.

We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

I hadn't actually noticed the affine analysis infra before, so thanks for pointing me towards this. From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR. Perhaps this could be better discussed on a draft PR to demonstrate specifically what we do with the affine transforms and whether the existing infra would be able to achieve the same effect.

junrushao · 2021-10-06T17:16:38Z

Thank you for your response @mbaret 😄

From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR

For sure, we are happy to work together and potentially improve the affine analysis infrastructure. Note that the iter-affine-map is located under tvm/arith/iter_affine_map.h, so it's not deeply coupled with TIR, but only reuses several TIR expressions.

CC'ed the author of affine analysis in TVM: @tqchen @spectrometerHBH @Hzfengsy

Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret · 2021-11-02T17:08:20Z

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:
1. a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
2. the Ethos-U-specific logic.
I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

Apologies for the late reply @areusch. Due to some of the uncertainties around implementing this generically (Relay fusion/conflict with AutoTVM/deriving matrices), I think it's more pragmatic to have everything initially scoped just to Ethos-U. But within that implementation, all the truly Ethos-U specific logic is quite self-contained (pretty much entirely in the performance heuristics and matrix definitions). Therefore, should this need reusing in future we could probably lift it out of the Ethos-U namespace. I think this would be very doable as the cascader itself is quite well decoupled from the rest of the Ethos-U compiler, exposing only a TE Schedule -> TE Schedule interface. With some effort, although hopefully not too much, that interface could likely be tweaked to be TensorIR -> TensorIR.

mbs-octoml

Hi Mike, thx for such a nicely written RFC, you're upping the bar.

I think we all agreed in our last live conversation:

Though the S-TIR machinery is now in main and could be a viable medium instead of TE that transition will be left to future work.
The transition to TE should hopefully not require duplicating the ScheduleBuilder visitor -- I think someone else at ARM was getting that in place. Sorry I can't keep up.
I'm hoping we can make constants first class so there's no need for any side inputs. Eg just look for them bound as globals in the IRModule. But that's a refactor which will probably come after this work.
There's an ongoing tension between working in Relay+TIR vs TIR-only, but the work-exclusively-in-TE/TIR approach is consistent with both the current AOT flow and the USMP analysis work.

If I got that right perhaps capture it in the 'alternates considered' or someplace.

I can't say anything about possible reuse of the affine x-form machinery when doing the abstract interpretation for the memory footprint. Sounds like it might be similar to the S-TIR situation: maybe not now but can go back later.

So LGTM from me.

areusch

thanks @mbaret we should merge this now :)

tqchen · 2022-01-27T14:17:03Z

To followup on this RFC, @mbaret would be great to start followup with a rolling-buffer or related primitive in TensorIR. So we can smoothly transition the solution as we start to migrate to TIR schedule

mbaret · 2022-02-01T10:23:09Z

We don't have any plans in the short term to look at extending the rolling buffer primitive into TensorIR, but I'd be happy to provide support/assistance to anyone interested in implementing it. I would hope the effort would not be too great as it's already implemented as a TIR pass.

mbaret · 2021-09-27T15:12:31Z

No description provided.

mbaret · 2021-09-27T15:18:28Z

cc @jacobbohlin @NicolaLancellotti @manupa-arm @csullivan @junrushao1994 @mbs-octoml @jroesch

junrushao · 2021-09-29T19:59:15Z

Thanks for the RFC!

The TE scheduling part (rolling-buffer) looks good to me.
Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you
We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

areusch

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:

a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
the Ethos-U-specific logic.

I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

areusch · 2021-09-29T16:11:39Z

-$$stripe_{in} = {M} \cdot {stripe_{out}}$$
+![meta-schedule-workflow](../resources/cascading-formula-1.png)

 Let's briefly consider how to derive such a transform matrix for a 3x3 unstrided, undilated and unpadded NHWC convolution. Immediately, the '3x3' kernel tells us something important: a single element in the output depends on 3x3 elements in the height/width of the input. If we were instead to consider a 2x2 region of the output in the height/width dimensions, we'd then need a 4x4 region in the input. So in general, the rule is that we need 2 more elements in height and width when calculating the dependencies of an output stripe. It can be shown that more generally this number is the kernel_size-1 in each axis. Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel. This is because the input channel axis is a reduction axis in a convolution, in a sense it isn't 'reflected' in the output. Combining these two observations, we arrive at the following transform matrix:


Now to consider the channels, in a convolution no matter how many output elements you are computing you'll always need every input channel.
just curious: what about depthwise convolutions?

In a depthwise you're right that this will be different, but depthwise will also use a different transform matrix. So here we're just working out the transform matrix for a standard 2D convolution.

mbaret · 2021-10-06T16:29:31Z

Thanks for taking a look @junrushao1994!

Grouping operators together might be challenging in Relay fusion and lowering, and would love to hear more thoughts from you

For Ethos-U, we have a different compilation flow to 'standard' TVM. In particular, those operators which are being offloaded onto Ethos-U don't go through FuseOps but instead are lowered into TE as one large graph. This is what allows us to avoid the problems of Relay level fusion. I agree that were we to want to generalize this technique further, we would need a solution for this.

We have some basic affine analysis infra (iter-affine-map) in TVM, and it would be great if the infra could be reused and improved upon the usecases in your RFC

I hadn't actually noticed the affine analysis infra before, so thanks for pointing me towards this. From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR. Perhaps this could be better discussed on a draft PR to demonstrate specifically what we do with the affine transforms and whether the existing infra would be able to achieve the same effect.

junrushao · 2021-10-06T17:16:38Z

Thank you for your response @mbaret 😄

From my initial look over it, I suspect there may be elements we could reuse, but the representations are unfamiliar to me as it how to work with them from outside TIR

For sure, we are happy to work together and potentially improve the affine analysis infrastructure. Note that the iter-affine-map is located under tvm/arith/iter_affine_map.h, so it's not deeply coupled with TIR, but only reuses several TIR expressions.

CC'ed the author of affine analysis in TVM: @tqchen @spectrometerHBH @Hzfengsy

Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret · 2021-11-02T17:08:20Z

@mbaret thanks for the detailed RFC! i think my main questions here are whether or not this could be implemented in roughly two pieces similar to USMP:
1. a piece which could be reused if we e.g. wanted to attempt to apply this to non-Ethos-U kernels
2. the Ethos-U-specific logic.
I understand it may not make sense to integrate with AutoTVM--it might make more sense to wait til TensorIR has fully landed. but just curious to understand in planning terms what the lift might be.

Apologies for the late reply @areusch. Due to some of the uncertainties around implementing this generically (Relay fusion/conflict with AutoTVM/deriving matrices), I think it's more pragmatic to have everything initially scoped just to Ethos-U. But within that implementation, all the truly Ethos-U specific logic is quite self-contained (pretty much entirely in the performance heuristics and matrix definitions). Therefore, should this need reusing in future we could probably lift it out of the Ethos-U namespace. I think this would be very doable as the cascader itself is quite well decoupled from the rest of the Ethos-U compiler, exposing only a TE Schedule -> TE Schedule interface. With some effort, although hopefully not too much, that interface could likely be tweaked to be TensorIR -> TensorIR.

mbs-octoml

Hi Mike, thx for such a nicely written RFC, you're upping the bar.

I think we all agreed in our last live conversation:

Though the S-TIR machinery is now in main and could be a viable medium instead of TE that transition will be left to future work.
The transition to TE should hopefully not require duplicating the ScheduleBuilder visitor -- I think someone else at ARM was getting that in place. Sorry I can't keep up.
I'm hoping we can make constants first class so there's no need for any side inputs. Eg just look for them bound as globals in the IRModule. But that's a refactor which will probably come after this work.
There's an ongoing tension between working in Relay+TIR vs TIR-only, but the work-exclusively-in-TE/TIR approach is consistent with both the current AOT flow and the USMP analysis work.

If I got that right perhaps capture it in the 'alternates considered' or someplace.

I can't say anything about possible reuse of the affine x-form machinery when doing the abstract interpretation for the memory footprint. Sounds like it might be similar to the S-TIR situation: maybe not now but can go back later.

So LGTM from me.

areusch

thanks @mbaret we should merge this now :)

tqchen · 2022-01-27T14:17:03Z

To followup on this RFC, @mbaret would be great to start followup with a rolling-buffer or related primitive in TensorIR. So we can smoothly transition the solution as we start to migrate to TIR schedule

mbaret · 2022-02-01T10:23:09Z

We don't have any plans in the short term to look at extending the rolling buffer primitive into TensorIR, but I'd be happy to provide support/assistance to anyone interested in implementing it. I would hope the effort would not be too great as it's already implemented as a TIR pass.

areusch reviewed Sep 28, 2021
View reviewed changes

Comment threadrfcs/0037-arm-ethosu-cascading-planner.md Outdated

NicolaLancellotti reviewed Sep 30, 2021
View reviewed changes

Comment threadrfcs/0037-arm-ethosu-cascading-planner.md Outdated
Comment threadrfcs/0037-arm-ethosu-cascading-planner.md Outdated

areusch reviewed Oct 4, 2021
View reviewed changes

mbaret mentioned this pull request Nov 2, 2021
[Tracking Issue] Arm(R) Ethos(TM)-U Cascading Scheduler apache/tvm#9429
Closed
12 tasks

mbaret added 3 commits November 2, 2021 16:22

Introduce the Arm(R) Ethos(TM)-U Cascading Planner
cbc81de
Change-Id: I34d0acf303391157cf3ddab9ae771b3f6c667967

Render formulas
7d94dc6
Change-Id: Ie4b6fbbc0cc060da6a733c06f276f2c22d2cbc93

Rename planner -> scheduler/cascader
afb0448
Change-Id: I89e9128a7e18791671fb4a7baeee9b64e65174c4

mbaret force-pushed the cascading-rfc branch from 5947693 to afb0448Compare November 4, 2021 16:55

mbaret changed the title ~~Introduce the Arm(R) Ethos(TM)-U Cascading Planner~~Introduce the Arm(R) Ethos(TM)-U Cascading SchedulerNov 4, 2021

mbs-octoml approved these changes Nov 4, 2021
View reviewed changes

This was referenced Nov 5, 2021
[microNPU][1] Add affine analysis structures for the cascader apache/tvm#9458
Merged
[microNPU][2a] Add CascaderGraph for cascading analysis apache/tvm#9469
Merged
[microNPU][2b] Create CascaderGraphs from TE graphs apache/tvm#9471
Merged

This was referenced Dec 20, 2021
[microNPU][2c] Add performance modelling to cascader apache/tvm#9778
Merged
[microNPU][2d] Add more Part matchers to cascader apache/tvm#9785
Merged

mbaret mentioned this pull request Jan 10, 2022
[microNPU][3] Plan generation for the cascader apache/tvm#9890
Merged

mbaret mentioned this pull request Jan 18, 2022
[microNPU][4] Add the cascader Proposal generator apache/tvm#9959
Merged

mbaret mentioned this pull request Jan 25, 2022
[microNPU][5] Convert Proposals to te.Schedules apache/tvm#10062
Merged

areusch approved these changes Jan 25, 2022
View reviewed changes

areusch merged commit f9fa824 into apache:mainJan 25, 2022

mbaret mentioned this pull request Feb 24, 2022
[microNPU][6] Migrate codegen to use cascader apache/tvm#10377
Closed

driazati mentioned this pull request Jul 14, 2022
TVM v0.9.0.rc0 Release Candidate Notes apache/tvm#12102
Closed