Skip to content

docs(kubeflow): add Trainer v2 training runtime catalog #255

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
typhoonzero merged 1 commit into from
Jun 5, 2026
Merged

docs(kubeflow): add Trainer v2 training runtime catalog #255

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions ...beflow/how_to/assets/training-runtimes/llamafactory0.9-cann8.5-arm64-trainingruntime.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
name: llamafactory0.9-cann8.5-arm64
namespace: kubeflow-admin-cpaas-io
labels:
trainer.kubeflow.org/framework: torch
alauda.io/training-runtime: llamafactory0.9-cann8.5-arm64
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
backoffLimit: 0
template:
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
containers:
- name: node
image: docker.io/alaudadockerhub/llamafactory0.9-cann8.5-arm64:v0.1.0
command:
- bash
- -c
- |
set -ex
python -c "import torch, torch_npu, llamafactory; print('torch', torch.__version__, 'torch_npu', torch_npu.__version__, 'lf', llamafactory.__version__, 'npu_available', torch_npu.npu.is_available())"
llamafactory-cli version || true
resources:
requests:
cpu: "1"
memory: 4Gi
limits:
cpu: "4"
memory: 8Gi
huawei.com/Ascend910B4: "1"
huawei.com/Ascend910B4-memory: "8192"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: dshm, mountPath: /dev/shm }
volumes:
- { name: workspace, emptyDir: {} }
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 2Gi
81 changes: 81 additions & 0 deletions ...kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
name: llamafactory0.9-cu126-amd64
namespace: kubeflow-admin-cpaas-io
labels:
trainer.kubeflow.org/framework: torch
alauda.io/training-runtime: llamafactory0.9-cu126-amd64
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
backoffLimit: 0
template:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
containers:
- name: node
image: docker.io/alaudadockerhub/llamafactory0.9-cu126-amd64:v0.1.0
command:
- bash
- -c
- |
set -ex
# Synthetic identity SFT smoke. Writes a 4-row alpaca dataset, fine-tunes
# qwen-style chat template against an empty config to exercise the
# LLaMA-Factory CLI without downloading a real model.
Comment on lines +39 to +41

@coderabbitai coderabbitai Bot Jun 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Align the smoke comment with executed behavior.

Lines 39-41 say this block fine-tunes, but the script only prepares data and checks versions. This mismatch can mislead debugging and runbook expectations.

Suggested fix 
-                          # Synthetic identity SFT smoke. Writes a 4-row alpaca dataset, fine-tunes
-                          # qwen-style chat template against an empty config to exercise the
-                          # LLaMA-Factory CLI without downloading a real model.
+                          # Synthetic smoke: writes a 4-row alpaca dataset and validates
+                          # LLaMA-Factory/PyTorch imports + CLI availability without downloading
+                          # a real model or running full fine-tuning.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Synthetic identity SFT smoke. Writes a 4-row alpaca dataset, fine-tunes
# qwen-style chat template against an empty config to exercise the
# LLaMA-Factory CLI without downloading a real model.
# Synthetic smoke: writes a 4-row alpaca dataset and validates
# LLaMA-Factory/PyTorch imports + CLI availability without downloading
# a real model or running full fine-tuning.
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml`
around lines 39 - 41, The comment block starting with "Synthetic identity SFT
smoke" (the lines that say "Writes a 4-row alpaca dataset, fine-tunes qwen-style
chat template...") is misleading because the YAML step only prepares data and
performs version checks; update that comment to accurately describe the executed
behavior (e.g., "prepares a 4-row alpaca dataset and validates LLaMA-Factory
CLI/model versions; does not run a fine-tune"), or alternatively modify the
associated step that invokes the LLaMA-Factory CLI to actually kick off a
fine-tune if the original intention was to run one; ensure the comment and the
step invoking the LLaMA-Factory CLI / data preparation are consistent.

mkdir -p /tmp/lf
cat >/tmp/lf/dataset_info.json <<'JSON'
{"identity_smoke": {"file_name": "identity_smoke.json"}}
JSON
cat >/tmp/lf/identity_smoke.json <<'JSON'
[
{"instruction":"Who are you?","input":"","output":"I am Alauda AI."},
{"instruction":"Where do you run?","input":"","output":"On Kubeflow Trainer v2."},
{"instruction":"What framework powers you?","input":"","output":"LLaMA-Factory on PyTorch."},
{"instruction":"What device?","input":"","output":"NVIDIA GPU via CUDA 12.6."}
]
JSON
python -c "import llamafactory, torch; print('lf', llamafactory.__version__, 'torch', torch.__version__, 'cuda', torch.cuda.is_available())"
llamafactory-cli version || true

@coderabbitai coderabbitai Bot Jun 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Do not swallow llamafactory-cli failures in smoke validation.

Line 55 masks command failures with || true, so this runtime can report success even when the CLI is missing/broken.

Suggested fix 
-                          llamafactory-cli version || true
+                          llamafactory-cli version
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
llamafactory-cli version || true
llamafactory-cli version
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml`
at line 55, The smoke-check currently masks failures by appending "|| true" to
the "llamafactory-cli version" invocation, allowing the runtime to report
success even when the CLI is absent or failing; remove the "|| true" from the
"llamafactory-cli version" call (or replace it with an explicit
existence/version check that fails the step and surfaces/logs the error) so that
the command's non-zero exit propagates and the validation fails when the CLI is
missing or broken.

resources:
requests:
cpu: "1"
memory: 4Gi
limits:
cpu: "4"
memory: 8Gi
nvidia.com/gpualloc: 1
nvidia.com/gpucores: 50
nvidia.com/gpumem: "8192"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: dshm, mountPath: /dev/shm }
volumes:
- { name: workspace, emptyDir: {} }
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 2Gi
66 changes: 66 additions & 0 deletions ...kubeflow/how_to/assets/training-runtimes/mindspeed-llm-cann8.5-arm64-trainingruntime.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
name: mindspeed-llm-cann8.5-arm64
namespace: kubeflow-admin-cpaas-io
labels:
trainer.kubeflow.org/framework: torch
alauda.io/training-runtime: mindspeed-llm-cann8.5-arm64
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
backoffLimit: 0
template:
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
containers:
- name: node
image: docker.io/alaudadockerhub/mindspeed-llm-cann8.5-arm64:v0.1.0
command:
- bash
- -c
- |
set -ex
python -c "import torch, torch_npu, megatron, mindspeed, mindspeed_llm; print('torch', torch.__version__, 'torch_npu', torch_npu.__version__, 'megatron', megatron.__version__, 'mindspeed_llm', mindspeed_llm.__version__)"
resources:
requests:
cpu: "1"
memory: 4Gi
limits:
cpu: "4"
memory: 8Gi
huawei.com/Ascend910B4: "1"
huawei.com/Ascend910B4-memory: "8192"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: dshm, mountPath: /dev/shm }
volumes:
- { name: workspace, emptyDir: {} }
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 2Gi
77 changes: 77 additions & 0 deletions docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cann8.5-arm64-trainingruntime.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
name: torch2.6-cann8.5-arm64
namespace: kubeflow-admin-cpaas-io
labels:
trainer.kubeflow.org/framework: torch
alauda.io/training-runtime: torch2.6-cann8.5-arm64
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
backoffLimit: 0
template:
spec:
schedulerName: hami-scheduler
runtimeClassName: ascend
securityContext:
runAsNonRoot: true
runAsUser: 1001
runAsGroup: 1001
fsGroup: 1001
containers:
- name: node
image: docker.io/alaudadockerhub/torch2.6-cann8.5-arm64:v0.1.0
command:
- python
- -c
- |
import os, torch
try:
import torch_npu
import torch_npu.utils
print(f"torch={torch.__version__} torch_npu={torch_npu.__version__}")
print(f"npu_available={torch_npu.npu.is_available()} count={torch_npu.npu.device_count()}")
if torch_npu.npu.is_available():
x = torch.randn(1024, 1024).npu()
y = x @ x.T
print(f"matmul ok shape={tuple(y.shape)} mean={y.mean().item():.4f}")
except Exception as e:
print(f"torch_npu probe failed: {e}")
raise
resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "4"
memory: 8Gi
huawei.com/Ascend910B4: "1"
huawei.com/Ascend910B4-memory: "8192"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: dshm, mountPath: /dev/shm }
volumes:
- { name: workspace, emptyDir: {} }
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 1Gi
70 changes: 70 additions & 0 deletions docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cu126-amd64-trainingruntime.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
apiVersion: trainer.kubeflow.org/v1alpha1
kind: TrainingRuntime
metadata:
name: torch2.6-cu126-amd64
namespace: kubeflow-admin-cpaas-io
labels:
trainer.kubeflow.org/framework: torch
alauda.io/training-runtime: torch2.6-cu126-amd64
spec:
mlPolicy:
numNodes: 1
torch:
numProcPerNode: auto
template:
spec:
replicatedJobs:
- name: node
template:
metadata:
labels:
trainer.kubeflow.org/trainjob-ancestor-step: trainer
spec:
backoffLimit: 0
template:
spec:
securityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
containers:
- name: node
image: docker.io/alaudadockerhub/torch2.6-cu126-amd64:v0.1.0
command:
- python
- -c
- |
import torch, os
print(f"torch={torch.__version__} cuda={torch.cuda.is_available()} devices={torch.cuda.device_count()}")
print(f"PET_NNODES={os.environ.get('PET_NNODES')} PET_NODE_RANK={os.environ.get('PET_NODE_RANK')}")
if torch.cuda.is_available():
x = torch.randn(1024, 1024, device='cuda')
y = x @ x.T
print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
Comment on lines +41 to +44

@coderabbitai coderabbitai Bot Jun 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fail fast when CUDA is unavailable in the smoke runtime.

Line 41 currently makes GPU validation optional; if CUDA is unavailable, the container still exits successfully, so the smoke TrainJob can pass with a broken GPU runtime.

Suggested fix 
-                          if torch.cuda.is_available():
-                              x = torch.randn(1024, 1024, device='cuda')
-                              y = x @ x.T
-                              print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
+                          if not torch.cuda.is_available() or torch.cuda.device_count() < 1:
+                              raise SystemExit("CUDA is not available; smoke check failed")
+                          x = torch.randn(1024, 1024, device='cuda')
+                          y = x @ x.T
+                          print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if torch.cuda.is_available():
x = torch.randn(1024, 1024, device='cuda')
y = x @ x.T
print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
if not torch.cuda.is_available() or torch.cuda.device_count() < 1:
raise SystemExit("CUDA is not available; smoke check failed")
x = torch.randn(1024, 1024, device='cuda')
y = x @ x.T
print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cu126-amd64-trainingruntime.yaml`
around lines 41 - 44, The GPU smoke-check currently skips failure when CUDA is
missing because it only runs the matmul under if torch.cuda.is_available();
change this to fail fast: detect CUDA unavailability with
torch.cuda.is_available() and immediately print a clear error and exit non‑zero
(or raise an exception) so the container fails the smoke TrainJob; keep the
existing CUDA matmul path (creating x on device='cuda', computing y = x @ x.T
and printing the success message) but ensure the missing-CUDA branch aborts the
process.

resources:
requests:
cpu: "1"
memory: 2Gi
limits:
cpu: "2"
memory: 4Gi
nvidia.com/gpualloc: 1
nvidia.com/gpucores: 50
nvidia.com/gpumem: "8192"
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: [ALL]
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
volumeMounts:
- { name: workspace, mountPath: /workspace }
- { name: dshm, mountPath: /dev/shm }
volumes:
- { name: workspace, emptyDir: {} }
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 1Gi
Loading