Thank you for the excellent work! I have a few questions regarding the top-k KV cache selection mechanism, and I would greatly appreciate your clarification.
Is the top-k mechanism applied during training as well, or is it only used during inference?
As I understand it, the top-k mechanism is only used to select which tokens participate in the actual attention computation, while the historical KV cache itself still remains in memory without eviction. If so, the memory usage would still grow linearly with the generated sequence length. Is this understanding accurate?
Thank you for your time.
Thank you for the excellent work! I have a few questions regarding the top-k KV cache selection mechanism, and I would greatly appreciate your clarification.
Is the top-k mechanism applied during training as well, or is it only used during inference?
As I understand it, the top-k mechanism is only used to select which tokens participate in the actual attention computation, while the historical KV cache itself still remains in memory without eviction. If so, the memory usage would still grow linearly with the generated sequence length. Is this understanding accurate?
Thank you for your time.