Skip to content

docs(training-hub): QLoRA + CPT guides + e2e (8.2.3) #270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
typhoonzero merged 1 commit into from
Jul 6, 2026
Merged

docs(training-hub): QLoRA + CPT guides + e2e (8.2.3) #270

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
338 changes: 338 additions & 0 deletions docs/en/training_guides/cpt-comprehensive-tutorial.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,338 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Comprehensive Continued Pre-training (CPT) Tutorial\n",
"\n",
"This notebook covers **continued pre-training (CPT)** \u2014 also called *continual* or\n",
"*domain-adaptive* pre-training \u2014 with the\n",
"[`training_hub`](https://github.com/Red-Hat-AI-Innovation-Team/training_hub) library on\n",
"Alauda AI.\n",
"\n",
"CPT keeps the original **next-token prediction** objective but runs it over a **raw-text\n",
"corpus** from your domain (medical notes, legal contracts, code, a new language, ...). It\n",
"teaches the model new *knowledge and vocabulary* before any instruction tuning, unlike SFT\n",
"which teaches *behavior* from chat data.\n",
"\n",
"`training_hub` runs CPT through the same `sft(...)` entrypoint with **`is_pretraining=True`**:\n",
"the loss is computed over *all* tokens (no assistant-only masking) and documents are packed\n",
"into fixed-size `block_size` windows."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## When to use CPT\n",
"\n",
"A typical domain-adaptation pipeline is **CPT -> SFT -> alignment**:\n",
"\n",
"1. **CPT** on raw domain text \u2014 inject knowledge / vocabulary.\n",
"2. **SFT** (or **OSFT**) on chat data \u2014 teach the model to follow instructions.\n",
"3. Optional preference/RL alignment.\n",
"\n",
"| | CPT | SFT | OSFT |\n",
"|---|---|---|---|\n",
"| Data | raw text | chat (instruction/response) | chat |\n",
"| Objective | next-token over all tokens | next-token over assistant turns | next-token, orthogonal subspace |\n",
"| Teaches | knowledge / vocabulary | behavior | behavior, forgetting-resistant |\n",
"\n",
"CPT can cause **catastrophic forgetting** of general ability. Mitigate by mixing in some\n",
"general-domain text, keeping the learning rate low, and running a follow-up SFT pass. If\n",
"forgetting is the main concern, consider **OSFT** instead (see\n",
"[Training Hub fine-tuning](../training-hub-fine-tuning.mdx))."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import os\n",
"\n",
"# CPT trains on RAW TEXT: one document per line under a \"text\" field.\n",
"data_path = \"./test_cpt_data.jsonl\"\n",
"if not os.path.exists(data_path):\n",
" print(f\"Creating dummy raw-text corpus at {data_path}\")\n",
" docs = [\n",
" \"Alauda AI is an MLOps platform that runs fine-tuning and inference workloads on Kubernetes.\",\n",
" \"Continued pre-training adapts a base language model to a new domain using unlabeled text.\",\n",
" \"The training_hub library wraps SFT, OSFT, LoRA, QLoRA and continued pre-training behind one interface.\",\n",
" \"Kubeflow Trainer v2 submits distributed TrainJobs that schedule onto GPU or NPU nodes.\",\n",
" ] * 8\n",
" with open(data_path, \"w\") as f:\n",
" for d in docs:\n",
" f.write(json.dumps({\"text\": d}) + \"\\n\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data format requirements\n",
"\n",
"CPT data is **raw text**, not chat turns. Each JSONL line is a document under a single\n",
"text column (default `\"text\"`):\n",
"\n",
"```json\n",
"{\"text\": \"A paragraph or document of raw domain text ...\"}\n",
"{\"text\": \"Another document ...\"}\n",
"```\n",
"\n",
"`training_hub` concatenates and chunks the documents into fixed-length `block_size`\n",
"windows and trains next-token prediction over every token (full unmasking).\n",
"\n",
"- `document_column_name` \u2014 the JSONL field holding the text (default `\"text\"`).\n",
"- `block_size` \u2014 the packed context-window length (e.g. 1024-4096)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Model configuration examples\n",
"\n",
"CPT typically starts from a **base** (not already instruction-tuned) checkpoint. Provide a\n",
"HuggingFace name or a local path; pre-download to a local directory on air-gapped clusters."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# =============================================================================\n",
"# MODEL CONFIGURATION EXAMPLES\n",
"# =============================================================================\n",
"\n",
"# Example 1: Qwen 3 0.6B Base (tiny - good for a smoke test)\n",
"qwen_small_example = {\n",
" \"model_name\": \"Qwen 3 0.6B Base\",\n",
" \"model_path\": \"/opt/app-root/src/Qwen3-0.6B-Base\",\n",
" \"example_block_size\": 1024,\n",
" \"example_batch_size\": 8,\n",
" \"example_learning_rate\": 5e-6,\n",
"}\n",
"\n",
"# Example 2: Qwen 2.5 7B (base) - domain adaptation target\n",
"qwen_7b_example = {\n",
" \"model_name\": \"Qwen 2.5 7B\",\n",
" \"model_path\": \"Qwen/Qwen2.5-7B\",\n",
" \"example_block_size\": 4096,\n",
" \"example_batch_size\": 128,\n",
" \"example_learning_rate\": 5e-6,\n",
"}\n",
"\n",
"# =============================================================================\n",
"# SELECT YOUR CONFIGURATION\n",
"# =============================================================================\n",
"selected_example = qwen_small_example\n",
"print(f\"Selected model: {selected_example['model_name']} ({selected_example['model_path']})\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## CPT parameter reference\n",
"\n",
"CPT reuses every `sft(...)` parameter and adds three pre-training controls:\n",
"\n",
"- `is_pretraining=True` \u2014 switch the data path to raw-text pre-training (no chat masking).\n",
"- `block_size` \u2014 packed sequence length for next-token prediction.\n",
"- `document_column_name` \u2014 JSONL field holding the raw text.\n",
"\n",
"Use a **low learning rate** (CPT is sensitive \u2014 `1e-6`-`1e-5`) to limit forgetting."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# =============================================================================\n",
"# COMPLETE CPT PARAMETER CONFIGURATION\n",
"# =============================================================================\n",
"from datetime import datetime\n",
"\n",
"experiment_name = \"cpt_comprehensive_example\"\n",
"timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
"full_experiment_name = f\"{experiment_name}_{timestamp}\"\n",
"\n",
"# ---- Required ----\n",
"model_path = selected_example[\"model_path\"]\n",
"data_path = \"./test_cpt_data.jsonl\"\n",
"ckpt_output_dir = f\"checkpoints/{full_experiment_name}\"\n",
"\n",
"# ---- Continued pre-training controls ----\n",
"is_pretraining = True\n",
"block_size = selected_example[\"example_block_size\"]\n",
"document_column_name = \"text\"\n",
"\n",
"# ---- Core training ----\n",
"num_epochs = 1\n",
"effective_batch_size = selected_example[\"example_batch_size\"]\n",
"learning_rate = selected_example[\"example_learning_rate\"] # keep low to limit forgetting\n",
"max_seq_len = block_size\n",
"max_tokens_per_gpu = 4096 # per-GPU token budget; auto micro-batch sizing\n",
"warmup_steps = 10\n",
"\n",
"print(\"CPT configuration:\")\n",
"print(f\" pretraining: is_pretraining={is_pretraining}, block_size={block_size}, column='{document_column_name}'\")\n",
"print(f\" train: epochs={num_epochs}, ebs={effective_batch_size}, lr={learning_rate}, max_tokens_per_gpu={max_tokens_per_gpu}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Distributed configuration\n",
"\n",
"CPT uses the same distributed topology knobs as SFT. Raw-text corpora are usually large,\n",
"so multi-GPU / multi-node is common \u2014 set `nproc_per_node` to the GPU count per node."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"nproc_per_node = 1 # raise to the per-node GPU count for real corpora\n",
"nnodes = 1\n",
"node_rank = 0\n",
"rdzv_id = 200\n",
"rdzv_endpoint = \"127.0.0.1:29500\"\n",
"print(f\"distributed: nproc_per_node={nproc_per_node}, nnodes={nnodes}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Execute training\n",
"\n",
"The final cell calls `training_hub.sft(..., is_pretraining=True)` \u2014 continued pre-training\n",
"over the raw-text corpus."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from training_hub import sft\n",
"\n",
"result = sft(\n",
" # --- required ---\n",
" model_path=model_path,\n",
" data_path=data_path,\n",
" ckpt_output_dir=ckpt_output_dir,\n",
" # --- continued pre-training ---\n",
" is_pretraining=is_pretraining,\n",
" block_size=block_size,\n",
" document_column_name=document_column_name,\n",
" # --- core training ---\n",
" num_epochs=num_epochs,\n",
" effective_batch_size=effective_batch_size,\n",
" learning_rate=learning_rate,\n",
" max_seq_len=max_seq_len,\n",
" max_tokens_per_gpu=max_tokens_per_gpu,\n",
" warmup_steps=warmup_steps,\n",
" checkpoint_at_epoch=True,\n",
" # --- distributed ---\n",
" nproc_per_node=nproc_per_node,\n",
" nnodes=nnodes,\n",
" node_rank=node_rank,\n",
" rdzv_id=rdzv_id,\n",
" rdzv_endpoint=rdzv_endpoint,\n",
")\n",
"print(f\"CPT finished: {result!r}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Post-training analysis\n",
"\n",
"CPT writes a **full checkpoint** (the whole model is updated). The natural next step is an\n",
"SFT / OSFT pass on instruction data so the domain-adapted model can follow instructions."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"hf_dir = os.path.join(ckpt_output_dir, \"hf_format\")\n",
"print(f\"Checkpoint dir: {hf_dir}\")\n",
"if os.path.isdir(hf_dir):\n",
" for ckpt in sorted(os.listdir(hf_dir)):\n",
" print(\" checkpoint:\", ckpt)\n",
"\n",
"print(\"\\nNext step: run sft() / osft() on instruction data, pointing model_path at this checkpoint.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Parameter reference summary\n",
"\n",
"### CPT-specific parameters\n",
"\n",
"| Parameter | Description | Typical |\n",
"|---|---|---|\n",
"| `is_pretraining` | Switch to raw-text pre-training (no chat masking) | `True` |\n",
"| `block_size` | Packed context-window length | 1024-4096 |\n",
"| `document_column_name` | JSONL field with raw text | `\"text\"` |\n",
"\n",
"### Common training parameters\n",
"\n",
"| Parameter | Description |\n",
"|---|---|\n",
"| `model_path`, `data_path`, `ckpt_output_dir` | Required I/O |\n",
"| `num_epochs`, `effective_batch_size`, `learning_rate` | Core training (use a **low** LR) |\n",
"| `max_seq_len`, `max_tokens_per_gpu` | Sequence length / memory budget |\n",
"| `nproc_per_node`, `nnodes`, `node_rank`, `rdzv_id`, `rdzv_endpoint` | Distributed topology |\n",
"\n",
"> **NPU note:** Full-parameter continued pre-training also runs on Huawei Ascend NPU via\n",
"> the `MindSpeed-LLM` runtime (`pretrain_gpt.py`). See\n",
"> [Fine-tune and Pretrain on Ascend NPU](../fine-tune-and-pretrain-llms-on-ascend-npu.mdx)\n",
"> and the `qwen25_pretrain_verify.ipynb` recipe for the Megatron-style pre-training path."
]
}
],
"metadata": {
"kernelspec": {
"display_name": ".venv",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.11"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading