Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization

Paper · Code · Dataset · Issue

Create an environment (requires Conda installation):

Use the following command to create a new Conda environment named robustgymnasium with Python 3.10:

conda create -n paperbench python=3.10

Activate the newly created environment:

conda activate paperbench

Install dependency packages:

pip install -r requirements.txt

Run VLLM server

export CUDA_VISIBLE_DEVICES=4,5,6,7 python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-4-Scout-17B-16E-Instruct \
--host 0.0.0.0 \
--port 8002 \
--tensor-parallel-size 4 \
--max-model-len 328816 \
--dtype auto \
--gpu-memory-utilization 0.90 \
--trust-remote-code

For personalization evaluation

bash paperbench/personalization/eva_personalization.sh

For privacy evaluation

bash paperbench/privacy/eva_privacy.sh

Citation

If you use PAPerBench in your research, please cite:

@article{gu2026long,
title={Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization},
author={Gu, Shangding},
journal={arXiv preprint arXiv:2602.15028},
year={2026}
}