[RFC]TECompiler: Staged refactor and removal of compile engine

[csullivan]

This is a staged removal of the use of compile engine. The motivation is to bring TIR compilation into the main flow of the compiler rather than producing and compiling it via a callback into the compile engine. By replacing Relay primitive function calls with TIR prim function calls that contain the lowered TIR we enable,

• An intermediate stage in the lowering process where Relay and TIR coexist.
• The ability to add passes at this intermediate stage,
  • For example memory planning which can infer user provided information from TE and the resulting TIR.

We are starting with a proof of concept by refactoring the GraphRuntimeCodegen to use an introduced TIR/TE compiler instead of the compile engine directly. In the new flow,

• The TE/TIR compiler lowers TE in the LowerTensorExpr pass
• Replaces relay.Function(attr:primitive) with a PrimFnCall that contains the lowered TIR
• Runs GraphPlanMemory
• Finally runs GraphRuntimeCodegen::VisitExpr to lower to graph JSON

[RFC][Relay] TECompiler: Rewrite existing compile engine to match updated compiler flow

[jroesch]

Need to port fix from #7703 but otherwise ready for review.

[jroesch]

Modulo some left over polish work and documentation I think this is ready for review @icemelon9 @comaniac@csullivan@tkonolige@rkimball@junrushao1994@areusch@mehrdadh

[jroesch]

cc @mbaret

[areusch]

did an initial pass

[areusch]

at some point we should support overriding this. it seems like you'd need to build a C++ impl right now to do that, so I don't know that we need this hook now. but, we should plan to put some customization back in the future?

[jroesch]

I think we should ask what do we allow customization of what and where ? I think allowing people to wholesale replace the lowering/memory/planning/etc in an ad-hoc way "seems bad", which effectively is all of these hooks (that I'm removing) do right now. They allow you to just arbitrarily overload parts of the compiler with custom code which is pretty non-compositional and hard to reason about since in certain cases, i.e Python loaded or not the behavior of the compiler is completely different.

[areusch]

I don't think it makes sense to allow people to overload GraphPlanMemory with its current implementation. If GraphPlanMemory were to support memory pools and tensor pinning, then I think there will be cases where people want to implement custom algorithms to place tensors in different memory pools.

I also think the reason this "seems bad" is that there is no good interface/data structure defined here. There's no reason we can't define one--in fact, I think we are hurting compiler readability by not doing so. Here we are keeping the interface to a sort of Pythonic "informal set of lists" in place of a proper data structure. Here and downstream of here, there are many instances where both TVM code and people trying to integrate with TVM code would benefit from effectively exporting StorageToken as a user-facing data structure and clearly defining its meaning.

[jroesch]

Yeah I think we need to move up the stack to address this, because everyone is only thinking about their own memory planning needs and not really looking at the large picture.

We talked