Pinned Loading

1. vllm-project/vllm-metalvllm-project/vllm-metalPublic

   Community maintained hardware plugin for vLLM on Apple Silicon

   Python 1.6k 224

2. inference-engine-benchmark-systeminference-engine-benchmark-systemPublic

   Production-grade benchmark harness comparing vLLM vs SGLang LLM inference engines across latency, throughput, KV-cache, structured generation, and speculative decoding on NVIDIA A10G (14 models, 2B…

   HTML 1

3. guardrail-compliance-engineguardrail-compliance-enginePublic

   Scan Terraform, CloudFormation & Kubernetes for compliance violations. Local checks + AWS Bedrock AI reasoning. SOC 2, CIS, PCI-DSS, HIPAA policies included.

   Python

4. sagemaker-llm-inference-optimizersagemaker-llm-inference-optimizerPublic

   Benchmark and optimize LLM inference backends on Amazon SageMaker for latency, throughput, and cost.

   HTML

5. kv-transfer-taxkv-transfer-taxPublic

   Measuring the KV-transfer tax of disaggregated prefill/decode LLM serving (vLLM + NIXL, 4x A10G): on PCIe-only hardware, disagg loses to plain data-parallel replication — measured, committed, repro…

   Python

6. triton-kernel-lab-for-llm-inferencetriton-kernel-lab-for-llm-inferencePublic

   Triton GPU kernels for LLM inference: FlashAttention-2 forward, RMSNorm, RoPE, SwiGLU, INT8 matmul, KV-cache ops. Correctness-tested against PyTorch and rooflined on real GPUs (A10G, T4). No invent…

   Python