Support for NVRTC kernels

[timmoon10]

This PR adds basic support for run-time compilation of kernels with the goals of reducing binary size, reducing compilation time, and enabling more configurable kernel optimizations.

Changes:

• NVRTC infrastructure (influenced by MXNet and nvFuser)
• NVRTC kernel for transpose using similar perf optimizations as Use 4B vector loads/stores in cast-transpose kernel for small matrices #101
• Convenience functions for string manipulation, accessing environment variables, CUDA API calls, and more

Tasks:

• Implementation
• Evaluate performance
• Documentation
• Unit tests for NVRTC infrastructure
• Unit tests for convenience functions

Future work:

• NVRTC kernels for other transpose operations
• NVRTC kernels for layer norm
• Refactor error checking infrastructure to use convenience functions

[ptrendx]

Cool trick :-)

[timmoon10]

Evaluating FP8 transposes on an H100 SXM for square matrices with nice dims (power of two or halfway between powers of two) and ugly dims (primes 20, 40, ..., 400):

Nice dims

Nice and ugly dims

The NVRTC kernel gets better performance than the baseline with small sizes (dim<2k), although we are clearly hitting up against kernel launch overheads. I do observe a ~1% perf regression at large scale though. We see the generalized implementation gets much worse perf than the NVRTC kernel, likely due to the large amount of bounds checking, but the baseline implementation doesn't support these sizes at all.

[timmoon10]

/te-ci

[timmoon10]

/te-ci

[timmoon10]

/te-ci

[ptrendx]

LGTM