Yuhui Zeng1,4, Xinyu Mao2,4, Xiaokun Liu4, Xin Tao4, Jinfa Huang3, Jiayi Ji1, Xiawu Zheng1
1Xiamen University 2The Chinese University of Hong Kong 3University of Rochester 4Kling Team, Kuaishou Technology
- 2026-07-20: Released the TimePLE codec, Qwen3-VL integration, SFT, data-curation pipeline, and public training configurations.
- Available: The TimePLE checkpoint, training annotations, benchmark annotations, and inference/evaluation suite are available now.
Video temporal grounding (VTG) aims to localize the continuous video interval described by a natural-language query. Existing VLM-based methods commonly represent an interval through two endpoint outputs, either as timestamp tokens or continuous boundary coordinates.
TimePLE reformulates VTG as interval-native prediction. It maps every valid temporal interval to a point in a canonical position-duration square and predicts a single joint distribution over this space. A generated <|TIMESPAN|> token provides the latent output interface, while input-side <|TIMESTAMP|> tokens encode the temporal coverage of sampled visual units using the same interval geometry.
video + query
|
| temporal anchors encoded by <|TIMESTAMP|>
v
VLM hidden states
|
| generated <|TIMESPAN|>
v
joint distribution over the canonical position-duration square
|
| expectation decoding + duration-aware residual refinement
v
continuous temporal interval [start, end]
The released codec uses a 128 x 128 canonical grid, Gaussian bandwidth sigma_u = sigma_v = 0.015, and duration-adaptive residual scale alpha = 0.02.
| Component | Status | Entry point |
|---|---|---|
| TimePLE interval codec | ✅ | src/timeple/models |
| Geometry pretraining | ✅ | src/timeple/geometry_pretrain |
| Qwen3-VL / Transformers integration | ✅ | integrations/transformers |
| SFT / ms-swift integration | ✅ | integrations/ms_swift |
| GRPO / EasyR1 integration | ✅ | integrations/easyr1 |
| Training-data curation | ✅ | data_pipeline/train_building |
| Benchmark human-review tools | ✅ | data_pipeline/bench_cleaning |
| TimePLE-8B checkpoint | ✅ | KlingTeam/TimePLE |
| Training and benchmark annotations | ✅ | KlingTeam/TimePLE-Dataset |
| Benchmark inference and evaluation | ✅ | evaluation |
- Installation
- Data Preparation
- Training TimePLE
- Data Curation
- Charades-TimePLE
- Inference and Evaluation
- Implementation Notes
Clone the repository and enter the project directory:
git clone https://github.com/KlingAIResearch/TimePLE.git
cd TimePLETimePLE uses uv to reproduce exact upstream environments. Build the environment required by the stage you want to run:
# Supervised fine-tuning
bash scripts/setup_env.sh sft
# Reinforcement-learning post-training
bash scripts/setup_env.sh rl
# Data curation
bash scripts/setup_env.sh data-pipelineThe setup script installs the exact upstream versions recorded in uv.lock, validates their versions and source hashes, and then applies the TimePLE integration patches. The repository does not contain complete copies of Transformers, ms-swift, or EasyR1 source files.
The SFT and RL extras are intentionally separate because accelerator stacks often require platform-specific dependency pins. uv.lock records the reference resolution.
The Hugging Face checkpoint is a weights-and-assets repository. The executable TimePLE implementation is provided by this installable source package rather than duplicated in the model repository. Import timeple once to register the custom configuration, model, and processor with Transformers; Hub-hosted Python code and trust_remote_code=True are not required.
importtorchimporttimeplefromtransformersimportAutoModelForImageTextToText, AutoProcessormodel_id="KlingTeam/TimePLE"processor=AutoProcessor.from_pretrained(model_id)
model=AutoModelForImageTextToText.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
).eval()Training and benchmark annotations are released separately as KlingTeam/TimePLE-Dataset. Licensed videos, pretrained weights, and generated checkpoints are not redistributed in this GitHub repository. Place local artifacts under the following repository-relative structure:
TimePLE/
├── checkpoints/
│ ├── base-model/ # Qwen3-VL-8B-Instruct
│ ├── geometry/ # selected geometry artifacts
│ ├── sft-stage1/ # selected stage-1 checkpoint
│ └── TimePLE-8B/ # final release-ready SFT model
└── data/
├── TimePLE-Dataset/ # full local annotation package; ignored by Git
├── sft.jsonl # committed schema examples
└── rl_train.jsonl # committed schema examples
All committed YAML files use paths relative to the TimePLE repository root. The launchers export TIMEPLE_ROOT automatically, so machine-specific absolute paths do not need to be committed.
The committed sft.jsonl and rl_train.jsonl records are 100 text-only schema examples with placeholder video names. They demonstrate the training interface and are not the paper training set. The SFT configs read the full local package from data/TimePLE-Dataset/train/timeple_train.jsonl.
Convert local temporal annotations into the TimePLE SFT and EasyR1 schemas with:
uv run python scripts/data/prepare_sft.py \
--input /path/to/temporal_annotations.jsonl \
--output data/sft.jsonl \
--count 100 \
--seed 2
uv run python scripts/data/prepare_rl.py \
--input data/sft.jsonl \
--output data/rl_train.jsonlSee data/README.md and scripts/data/README.md for the record schemas.
TimePLE training consists of three independently launched runs:
- interval geometry pretraining;
- SFT stage 1 with the interval codec frozen;
- SFT stage 2 with the interval codec unfrozen.
The repository does not automatically promote checkpoints between stages. After each run, inspect the validation results and expose the selected artifact at the stable repository-relative path expected by the next config.
uv run python -m timeple.geometry_pretrain \
--config configs/model/geometry_pretrain_v1.yamlEach run is saved under a timestamped directory in outputs/geometry_pretrain_v1/. After selecting a run, provide:
checkpoints/geometry/best.pt <- best_timeple_codec_state_dict.pt
checkpoints/geometry/codec_config.json <- codec_config_resolved.json
You may copy or link the selected files, or update the stage-1 config to another repository-relative location.
bash scripts/sft/train_stage1.shStage 1 loads the selected geometry state, freezes the TimePLE encoder and decoder, and trains the language-side temporal interface. Its validation set is created from the full local annotation package through the deterministic split configured in configs/sft/timeple_sft_stage1.yaml.
After training, expose the selected checkpoint as checkpoints/sft-stage1/ or update the stage-2 model field.
bash scripts/sft/train_stage2.shStage 2 starts from the selected stage-1 model and unfreezes the TimePLE encoder and decoder. It independently creates a deterministic validation split from data/TimePLE-Dataset/train/timeple_train.jsonl.
The final release-ready SFT model is stored locally at checkpoints/TimePLE-8B/.
# GRPO used in the paper's post-SFT study
bash scripts/rl/train_grpo.sh
# Experimental repository extensions
bash scripts/csdo/train_csdo.sh
bash scripts/tr_spd/train_tr_spd.shCSDO and TR-SPD are experimental extensions and are not required for the paper's main TimePLE-SFT result.
The training-data pipeline combines heterogeneous teacher VLMs to verify existing temporal annotations and construct additional grounded samples from cross-model event consensus.
The public implementation provides:
- deterministic preprocessing for Charades-STA and ActivityNet-Captions;
- Gemini and local-vLLM teacher backends;
- agreement-based filtering of existing samples;
- temporal and semantic consensus for newly constructed samples;
- conversion into TimePLE SFT and RL supervision formats.
Start from data_pipeline/README.md. Teacher checkpoints, API credentials, raw annotations, and licensed videos are supplied locally through copied configuration templates and are never hard-coded in the repository.
Charades-TimePLE is the human-verified corrected benchmark described in the paper. Its annotations are distributed together with the TimePLE training set in the unified Hugging Face dataset repository:
Important
Dataset:KlingTeam/TimePLE-Dataset
The dataset repository provides the training annotations, corrected benchmark annotations, WebDataset video shards, and integrity metadata.
The human-review interface and annotation-application tools are already available under data_pipeline/bench_cleaning.
The public evaluation suite supports Charades-STA, ActivityNet-Captions, and QVHighlights with layered dataset/model/prompt profiles, resumable prediction files, distributed sharding, and duration-stratified metrics.
Install the evaluation environment and render a suite without loading the model:
bash scripts/setup_env.sh eval
uv run python evaluation/src/run_eval_suite.py \
--suite evaluation/configs/suites/charades_sta.yamlRun TimePLE on a benchmark with:
SUITE=charades_sta bash scripts/eval/run_suite.sh --models timeple_8b
SUITE=activitynet_captions bash scripts/eval/run_suite.sh
SUITE=qvhighlights bash scripts/eval/run_suite.shSee evaluation/README.md for the expected annotation schema, data layout, output files, and distributed evaluation workflow.
Note
The default evaluation profile follows the paper setting of 2 FPS, at most 200 frames, and at most 64 visual tokens per frame. These inference settings are independent of the public training YAML files.
For a normalized interval [s, e] with duration d = e - s, TimePLE uses:
u = s / (1 - d)
v = d
Every point (u, v) in the canonical square maps back to a valid interval. The output decoder predicts a joint distribution over the square, computes its expected coordinate, and applies a duration-aware bounded residual before recovering continuous boundaries.
When <|TIMESTAMP|> and <|TIMESPAN|> are newly added, their input and output embeddings are initialized using the empirical mean and covariance statistics of the existing vocabulary embeddings. Temporal-token rows already present in a resumed TimePLE checkpoint are preserved.
| Path | Description |
|---|---|
src/timeple/models | Canonical transform, codec, losses, and interface adapters |
src/timeple/geometry_pretrain | Synthetic geometry training and diagnostics |
configs | Model, SFT, RL, and DeepSpeed configurations |
integrations/transformers | Versioned Qwen3-VL integration patch and manifest |
integrations/ms_swift | Versioned SFT integration patch and manifest |
integrations/easyr1 | GRPO, CSDO, and TR-SPD integration |
data_pipeline | Training-data curation and benchmark correction |
rewards | Temporal localization and format rewards |
Run the lightweight repository checks without launching distributed training:
bash scripts/setup_env.sh dev
uv run pytest
uv run python -m compileall -q src integrations rewards scripts tests
bash -n scripts/sft/*.sh scripts/rl/*.sh scripts/csdo/*.sh scripts/tr_spd/*.shIf you find TimePLE useful for your research, please consider citing our work:
@article{zeng2026timeple,
title = {TimePLE: Rethinking Temporal Representation for Video Temporal Grounding},
author = {Zeng, Yuhui and Mao, Xinyu and Liu, Xiaokun and Tao, Xin and Huang, Jinfa and Ji, Jiayi and Zheng, Xiawu},
journal = {arXiv preprint},
year = {2026}
}The citation entry will be updated with the final arXiv identifier.
TimePLE is built upon the following open-source projects:
See THIRD_PARTY_NOTICES.md for integration details and upstream licenses.
TimePLE is released under the Apache License 2.0. This repository does not redistribute third-party datasets, licensed videos, or pretrained model weights.