g023's TurboXInf 🚀: 2x+ faster inference for Qwen3-1.77B or Qwen3.5-2B on RTX 3060! Custom Triton INT8 GEMV kernels halve memory traffic by fusing dequantization, paired with torch.compile. Hits 113 tok/s (vs 56.4 baseline) with no quality loss with INT8 even better results for INT4. MIT License.
vLLM serving poolside Laguna S 2.1 (118B MoE, 8B active, INT4) on AMD Strix Halo (Ryzen AI Max+ 395, gfx1151, 128GB unified) via TheRock ROCm nightly. OpenAI-compatible, 256K context, DFlash speculative decoding.
Three hand-written Triton kernels for LLM inference (fused RMSNorm plus residual, online softmax, INT4 g128 GEMV) benchmarked on NVIDIA Blackwell against PyTorch eager and torch.compile, with every raw CUDA-event sample, measured device ceiling, and Nsight Compute report committed and CI-verified.
Trajectory-aware mixed-precision quantization for Stable Diffusion XL, perceptually lossless inference under 8GB VRAM, no retraining.
Fully local desktop automation agent. Five OpenVINO INT4 agents see the screen, plan, act, and verify their own work. No cloud calls, no data leaves the machine.
To associate your repository with the int4-quantization topic, visit your repo's landing page and select "manage topics."