thedatamates

TokenSpeed is a speed-of-light LLM inference engine.

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit and 4-bit floating point (FP8 and FP4) precision on Hopper, Ada and Blackwell GPUs, to provide better performance with lower memory utilization in both training and inference.

Programmable chat templates for LLM training and inference.

Scalable toolkit for efficient model reinforcement

Fast and memory-efficient exact attention

🚀 Pytorch Distributed native training library for LLMs/VLMs with OOTB Hugging Face support

Ongoing research training transformer models at scale

Training library for Megatron-based models with bidirectional Hugging Face conversion capability

A high-throughput and memory-efficient inference and serving engine for LLMs

FlashInfer: Kernel Library for LLM Serving