prefix-sharing

Here are 3 public repositories matching this topic...

JohnScheuer / pipeline-parallel-kv-bench

Analytical benchmark connecting pipeline parallelism and KV cache management: bubble-time overlap for disaggregation, prefix reuse tax of PP stages, and tiering speedup across NVLink, InfiniBand, and PCIe

performance-engineering benchmark simulation inference systems infiniband disaggregation pcie nvlink analytical-model tiering pipeline-parallelism kv-cache llm llm-serving machine-learning-infrastructure prefix-sharing kv-cache-management bubble-time

Updated Jul 26, 2026
Python

JohnScheuer / tensor-parallel-kv-cache-bench

Star

Analytical benchmark connecting tensor parallelism and KV cache management in LLM serving: GQA sharding regimes, all-gather vs recompute tradeoffs, PCIe viability, and operational decision matrix across NVLink, InfiniBand, and PCIe

Updated Jul 26, 2026
Python

JohnScheuer / sliding-window-attention-kv-bench

Star

Analytical benchmark for sliding window attention KV cache management: quality vs window size tradeoffs, SWA vs eviction comparison, prefix sharing interaction, and operational window recommendations across four attention distributions

performance-engineering benchmark simulation inference systems attention mistral eviction analytical-model kv-cache llm long-context llm-serving machine-learning-infrastructure sliding-window-attention prefix-sharing serving-systems kv-cache-management

Updated Jul 26, 2026
Python

Add this topic to your repo

To associate your repository with the prefix-sharing topic, visit your repo's landing page and select "manage topics."

Learn more