Foundation Model Inference

FlexLLMGen Public archive

Running large language models on a single GPU for throughput-oriented scenarios.

Python 9,359Apache-2.0 590 52 (3 issues need help) 6 Updated Oct 28, 2024

H2O Public

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

Python 529 86 35 1 Updated Aug 1, 2024

DejaVu Public

Python 359 45 27 0 Updated Apr 2, 2024

Navigation Menu