Skip to content

docs(kubeflow): add Trainer v2 training runtime catalog - #255

Merged
typhoonzero merged 1 commit into
masterfrom
codex/training-runtimes
Jun 5, 2026
Merged

typhoonzero merged 1 commit into
masterfrom
codex/training-runtimes

Conversation

@typhoonzero

@typhoonzero typhoonzero commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Six curated TrainingRuntime images (PyTorch + LLaMA-Factory + TrainingHub on NVIDIA CUDA, plus torch_npu / LLaMA-Factory / MindSpeed-LLM on Huawei Ascend NPU via CANN) so users can submit TrainJobs without building a custom image.

  • training-runtimes.mdx: device + framework catalog, HAMI vGPU / vNPU resource model, NPU host-driver caveat, smoke TrainJob walkthrough.
  • assets/training-runtimes/*.yaml: ready-to-apply TrainingRuntime for each image plus a shared smoke TrainJob template.

Source Containerfiles, multi-arch build helper, e2e harness, and post-fix security scan evidence live in alauda/kubeflow-plugin under training-runtimes/.

Summary by CodeRabbit

  • Documentation
    • Added a comprehensive Kubeflow TrainingRuntimes guide with usage, caveats, and smoke-test flow.
    • Published runtime pages for PyTorch, LLaMA-Factory, TrainingHub, and MindSpeed-LLM with v0.1.0 container images.
  • New Features
    • Delivered ready-to-apply runtime templates for NVIDIA CUDA (amd64) and Huawei Ascend NPU (arm64), including resource and shared-memory defaults.
    • Included hardened pod/container security settings in templates and device-specific resource examples for GPU/NPU deployment.

@coderabbitai

coderabbitai Bot commented Jun 4, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Adds six TrainingRuntime manifests (CUDA and Ascend), a TrainJob smoke-test example, and a documentation page describing runtime selection, device resource models, smoke-test steps, and image-specific caveats.

Changes

CUDA GPU Training Runtimes

Layer / File(s) Summary
LLaMA-Factory CUDA Runtime
docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml
Adds llamafactory0.9-cu126-amd64 with single-node Torch setup, synthetic dataset creation, llamafactory-cli smoke test command, hardened non-root security context, and GPU resource requests/limits.
PyTorch 2.6 CUDA Runtime
docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cu126-amd64-trainingruntime.yaml
Adds torch2.6-cu126-amd64 with inline CUDA availability probe and GPU matrix-multiplication smoke test, non-root security hardening, and GPU resource configuration.
TRL/PEFT/DeepSpeed Training Hub Runtime
docs/en/kubeflow/how_to/assets/training-runtimes/traininghub0.1-cu126-amd64-trainingruntime.yaml
Adds traininghub0.1-cu126-amd64 with framework version probing, conditional CUDA/LoRA initialization, restrictive security context, and GPU resource limits.
Smoke Test Example
docs/en/kubeflow/how_to/assets/training-runtimes/trainjob-smoke.yaml
Provides a concrete TrainJob manifest with inline documentation listing available runtimes and demonstrating runtime reference for verification.

Ascend NPU Training Runtimes

Layer / File(s) Summary
LLaMA-Factory Ascend Runtime
docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cann8.5-arm64-trainingruntime.yaml
Adds llamafactory0.9-cann8.5-arm64 targeting hami-scheduler and runtimeClassName: ascend, with torch_npu availability probing, llamafactory-cli smoke test, Ascend 910B4 device requests, and memory-backed /dev/shm.
PyTorch 2.6 Ascend Runtime
docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cann8.5-arm64-trainingruntime.yaml
Adds torch2.6-cann8.5-arm64 with torch_npu probe and NPU matmul smoke test, HAMI scheduler/Ascend runtime integration, Ascend device and memory resource limits, and error handling with logging.
MindSpeed LLM Ascend Runtime
docs/en/kubeflow/how_to/assets/training-runtimes/mindspeed-llm-cann8.5-arm64-trainingruntime.yaml
Adds mindspeed-llm-cann8.5-arm64 with multi-framework version reporting (torch, torch_npu, megatron, mindspeed, mindspeed_llm), HAMI scheduler and Ascend runtime class configuration, and hardened container security.

Training Runtime Documentation and Guide

Layer / File(s) Summary
Runtime Selection and Usage Guide
docs/en/kubeflow/how_to/training-runtimes.mdx (intro, matrix, usage)
Introduces the runtime catalog and tag scheme, provides decision rules for selecting runtimes by training type and accelerator, and documents kubectl commands for deployment and pod inspection.
Device Resource Configuration and Troubleshooting
docs/en/kubeflow/how_to/training-runtimes.mdx (resource model section)
Documents GPU and NPU resource request/limit models, HAMI vGPU slicing support, runtimeClassName: ascend configuration, device-plugin interaction, and pending pod troubleshooting for driver/label/registration issues.
Framework Caveats and Build Information
docs/en/kubeflow/how_to/training-runtimes.mdx (caveats and build sections)
Lists image-specific limitations (DeepSpeed JIT requiring CUDA toolkit, MindSpeed/MindSpeed-LLM version compatibility), documents build artifact locations, and explains the derivation model from base images.

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly Related PRs

  • alauda/aml-docs#201: Adds a TrainingRuntime for mindspeed-llm-cann8.5-arm64, which closely relates to the MindSpeed-LLM Ascend runtime added here.

Poem

🐰 I hopped through YAML, volumes, and schedulers bright,

I mounted /dev/shm and checked GPUs by night,
For CUDA and Ascend the manifests sing,
Now trainers wake up and happily spring,
🥕 logs stream, smoke-tests pass — a deploy done right.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: adding a catalog of Kubeflow Trainer v2 training runtimes to the documentation.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/training-runtimes

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jun 4, 2026

Copy link
Copy Markdown

Deploying alauda-ai with  Cloudflare Pages  Cloudflare Pages

Latest commit: 32f075a
Status: ✅  Deploy successful!
Preview URL: https://8f69221d.alauda-ai.pages.dev
Branch Preview URL: https://codex-training-runtimes.alauda-ai.pages.dev

View logs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🧹 Nitpick comments (1)
docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cann8.5-arm64-trainingruntime.yaml (1)

42-42: ⚡ Quick win

Consider removing || true to catch CLI failures.

The || true suppresses failures from llamafactory-cli version, which means the smoke test will succeed even if the CLI tool is broken or misconfigured. For a smoke test runtime, failing fast when core tooling doesn't work provides clearer diagnostics.

If the CLI check is intentionally optional (e.g., known flakiness), consider adding a comment explaining why failures are suppressed.

🔧 Proposed change to fail fast on CLI issues
           set -ex
           python -c "import torch, torch_npu, llamafactory; print('torch', torch.__version__, 'torch_npu', torch_npu.__version__, 'lf', llamafactory.__version__, 'npu_available', torch_npu.npu.is_available())"
-          llamafactory-cli version || true
+          llamafactory-cli version
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cann8.5-arm64-trainingruntime.yaml`
at line 42, The smoke-test line currently runs "llamafactory-cli version ||
true", which silences failures; remove the "|| true" so the step fails fast on
CLI errors and surfaces diagnostics, or if suppression is deliberate add an
inline comment explaining the known flakiness and why the failure is acceptable;
update the line that contains the exact token "llamafactory-cli version || true"
accordingly and ensure the job/step will fail when "llamafactory-cli version"
exits non‑zero.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml`:
- Line 55: The smoke-check currently masks failures by appending "|| true" to
the "llamafactory-cli version" invocation, allowing the runtime to report
success even when the CLI is absent or failing; remove the "|| true" from the
"llamafactory-cli version" call (or replace it with an explicit
existence/version check that fails the step and surfaces/logs the error) so that
the command's non-zero exit propagates and the validation fails when the CLI is
missing or broken.
- Around line 39-41: The comment block starting with "Synthetic identity SFT
smoke" (the lines that say "Writes a 4-row alpaca dataset, fine-tunes qwen-style
chat template...") is misleading because the YAML step only prepares data and
performs version checks; update that comment to accurately describe the executed
behavior (e.g., "prepares a 4-row alpaca dataset and validates LLaMA-Factory
CLI/model versions; does not run a fine-tune"), or alternatively modify the
associated step that invokes the LLaMA-Factory CLI to actually kick off a
fine-tune if the original intention was to run one; ensure the comment and the
step invoking the LLaMA-Factory CLI / data preparation are consistent.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cu126-amd64-trainingruntime.yaml`:
- Around line 41-44: The GPU smoke-check currently skips failure when CUDA is
missing because it only runs the matmul under if torch.cuda.is_available();
change this to fail fast: detect CUDA unavailability with
torch.cuda.is_available() and immediately print a clear error and exit non‑zero
(or raise an exception) so the container fails the smoke TrainJob; keep the
existing CUDA matmul path (creating x on device='cuda', computing y = x @ x.T
and printing the success message) but ensure the missing-CUDA branch aborts the
process.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/traininghub0.1-cu126-amd64-trainingruntime.yaml`:
- Around line 41-44: The current smoke check wraps CUDA-dependent logic in an if
torch.cuda.is_available(): block so a no-CUDA environment still passes; change
this to require CUDA by checking torch.cuda.is_available() and failing fast
(raise a RuntimeError or call sys.exit(1)) if False before importing peft and
instantiating LoraConfig (symbols: torch.cuda.is_available, LoraConfig, cfg,
print), so the GPU runtime image fails validation when CUDA is not present.

In `@docs/en/kubeflow/how_to/assets/training-runtimes/trainjob-smoke.yaml`:
- Line 13: The manifest hardcodes namespace: kubeflow-admin-cpaas-io which
breaks runtime resolution because spec.runtimeRef.name must exist in the same
namespace; remove the fixed "namespace" line (or replace it with a
template/placeholder that uses the target Profile namespace) so users can apply
the smoke manifest into their own Profile namespace, and add a brief note near
spec.runtimeRef.name to remind users that the referenced runtime must be created
in the same namespace as the manifest.

In `@docs/en/kubeflow/how_to/training-runtimes.mdx`:
- Around line 39-40: The doc line describing the image tag scheme
(v<semver>-build.<YYYYMMDDHHMMSS>) conflicts with the TrainingRuntime manifests
that use simple semantic tags like v0.1.0; update the sentence in the docs to
either (a) match the manifests by stating that manifests currently use plain
semantic tags (e.g., v0.1.0) or (b) clarify both formats and when each is used
by adding a short note that timestamped tags (v<semver>-build.<YYYYMMDDHHMMSS>)
are produced for rebuilt images while released manifests are pinned to simple
semantic tags (e.g., v0.1.0) — edit the paragraph that contains the tag scheme
text to implement one of these options and ensure the example tags in the doc
align with the TrainingRuntime YAMLs.
- Around line 71-73: The current sentence incorrectly tells users to "Replace
the placeholder image tag..." even though the YAMLs already contain concrete
tags; update the prose around TrainingRuntime/metadata.namespace and TrainJob to
note that manifests are pinned to a specific version and instruct users to
update the image: field if they need a different release (suggested phrasing:
"The manifests are pinned to a specific version; update image: if you need a
different release.").

---

Nitpick comments:
In
`@docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cann8.5-arm64-trainingruntime.yaml`:
- Line 42: The smoke-test line currently runs "llamafactory-cli version ||
true", which silences failures; remove the "|| true" so the step fails fast on
CLI errors and surfaces diagnostics, or if suppression is deliberate add an
inline comment explaining the known flakiness and why the failure is acceptable;
update the line that contains the exact token "llamafactory-cli version || true"
accordingly and ensure the job/step will fail when "llamafactory-cli version"
exits non‑zero.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 1db28062-bf1e-4cd2-a2bc-8c200768bc07

📥 Commits

Reviewing files that changed from the base of the PR and between da357ca and 9751aa0.

📒 Files selected for processing (8)
  • docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cann8.5-arm64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/mindspeed-llm-cann8.5-arm64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cann8.5-arm64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cu126-amd64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/traininghub0.1-cu126-amd64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/trainjob-smoke.yaml
  • docs/en/kubeflow/how_to/training-runtimes.mdx

Comment on lines +39 to +41
# Synthetic identity SFT smoke. Writes a 4-row alpaca dataset, fine-tunes
# qwen-style chat template against an empty config to exercise the
# LLaMA-Factory CLI without downloading a real model.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Align the smoke comment with executed behavior.

Lines 39-41 say this block fine-tunes, but the script only prepares data and checks versions. This mismatch can mislead debugging and runbook expectations.

Suggested fix
-                          # Synthetic identity SFT smoke. Writes a 4-row alpaca dataset, fine-tunes
-                          # qwen-style chat template against an empty config to exercise the
-                          # LLaMA-Factory CLI without downloading a real model.
+                          # Synthetic smoke: writes a 4-row alpaca dataset and validates
+                          # LLaMA-Factory/PyTorch imports + CLI availability without downloading
+                          # a real model or running full fine-tuning.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
# Synthetic identity SFT smoke. Writes a 4-row alpaca dataset, fine-tunes
# qwen-style chat template against an empty config to exercise the
# LLaMA-Factory CLI without downloading a real model.
# Synthetic smoke: writes a 4-row alpaca dataset and validates
# LLaMA-Factory/PyTorch imports + CLI availability without downloading
# a real model or running full fine-tuning.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml`
around lines 39 - 41, The comment block starting with "Synthetic identity SFT
smoke" (the lines that say "Writes a 4-row alpaca dataset, fine-tunes qwen-style
chat template...") is misleading because the YAML step only prepares data and
performs version checks; update that comment to accurately describe the executed
behavior (e.g., "prepares a 4-row alpaca dataset and validates LLaMA-Factory
CLI/model versions; does not run a fine-tune"), or alternatively modify the
associated step that invokes the LLaMA-Factory CLI to actually kick off a
fine-tune if the original intention was to run one; ensure the comment and the
step invoking the LLaMA-Factory CLI / data preparation are consistent.

]
JSON
python -c "import llamafactory, torch; print('lf', llamafactory.__version__, 'torch', torch.__version__, 'cuda', torch.cuda.is_available())"
llamafactory-cli version || true

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Do not swallow llamafactory-cli failures in smoke validation.

Line 55 masks command failures with || true, so this runtime can report success even when the CLI is missing/broken.

Suggested fix
-                          llamafactory-cli version || true
+                          llamafactory-cli version
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
llamafactory-cli version || true
llamafactory-cli version
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml`
at line 55, The smoke-check currently masks failures by appending "|| true" to
the "llamafactory-cli version" invocation, allowing the runtime to report
success even when the CLI is absent or failing; remove the "|| true" from the
"llamafactory-cli version" call (or replace it with an explicit
existence/version check that fails the step and surfaces/logs the error) so that
the command's non-zero exit propagates and the validation fails when the CLI is
missing or broken.

Comment on lines +41 to +44
if torch.cuda.is_available():
x = torch.randn(1024, 1024, device='cuda')
y = x @ x.T
print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fail fast when CUDA is unavailable in the smoke runtime.

Line 41 currently makes GPU validation optional; if CUDA is unavailable, the container still exits successfully, so the smoke TrainJob can pass with a broken GPU runtime.

Suggested fix
-                          if torch.cuda.is_available():
-                              x = torch.randn(1024, 1024, device='cuda')
-                              y = x @ x.T
-                              print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
+                          if not torch.cuda.is_available() or torch.cuda.device_count() < 1:
+                              raise SystemExit("CUDA is not available; smoke check failed")
+                          x = torch.randn(1024, 1024, device='cuda')
+                          y = x @ x.T
+                          print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if torch.cuda.is_available():
x = torch.randn(1024, 1024, device='cuda')
y = x @ x.T
print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
if not torch.cuda.is_available() or torch.cuda.device_count() < 1:
raise SystemExit("CUDA is not available; smoke check failed")
x = torch.randn(1024, 1024, device='cuda')
y = x @ x.T
print(f"matmul ok, shape={tuple(y.shape)}, mean={y.mean().item():.4f}")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cu126-amd64-trainingruntime.yaml`
around lines 41 - 44, The GPU smoke-check currently skips failure when CUDA is
missing because it only runs the matmul under if torch.cuda.is_available();
change this to fail fast: detect CUDA unavailability with
torch.cuda.is_available() and immediately print a clear error and exit non‑zero
(or raise an exception) so the container fails the smoke TrainJob; keep the
existing CUDA matmul path (creating x on device='cuda', computing y = x @ x.T
and printing the success message) but ensure the missing-CUDA branch aborts the
process.

Comment on lines +41 to +44
if torch.cuda.is_available():
from peft import LoraConfig
cfg = LoraConfig(r=4, lora_alpha=8, lora_dropout=0.05, target_modules=["q_proj","v_proj"])
print("peft LoraConfig ok", cfg)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Require CUDA for this GPU smoke runtime instead of treating it as optional.

At Line 41, CUDA-dependent checks are conditional, so a no-CUDA environment still succeeds. That weakens smoke validation for a CUDA runtime image.

Suggested fix
-                          if torch.cuda.is_available():
-                              from peft import LoraConfig
-                              cfg = LoraConfig(r=4, lora_alpha=8, lora_dropout=0.05, target_modules=["q_proj","v_proj"])
-                              print("peft LoraConfig ok", cfg)
+                          if not torch.cuda.is_available() or torch.cuda.device_count() < 1:
+                              raise SystemExit("CUDA is not available; smoke check failed")
+                          from peft import LoraConfig
+                          cfg = LoraConfig(r=4, lora_alpha=8, lora_dropout=0.05, target_modules=["q_proj","v_proj"])
+                          print("peft LoraConfig ok", cfg)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if torch.cuda.is_available():
from peft import LoraConfig
cfg = LoraConfig(r=4, lora_alpha=8, lora_dropout=0.05, target_modules=["q_proj","v_proj"])
print("peft LoraConfig ok", cfg)
if not torch.cuda.is_available() or torch.cuda.device_count() < 1:
raise SystemExit("CUDA is not available; smoke check failed")
from peft import LoraConfig
cfg = LoraConfig(r=4, lora_alpha=8, lora_dropout=0.05, target_modules=["q_proj","v_proj"])
print("peft LoraConfig ok", cfg)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@docs/en/kubeflow/how_to/assets/training-runtimes/traininghub0.1-cu126-amd64-trainingruntime.yaml`
around lines 41 - 44, The current smoke check wraps CUDA-dependent logic in an
if torch.cuda.is_available(): block so a no-CUDA environment still passes;
change this to require CUDA by checking torch.cuda.is_available() and failing
fast (raise a RuntimeError or call sys.exit(1)) if False before importing peft
and instantiating LoraConfig (symbols: torch.cuda.is_available, LoraConfig, cfg,
print), so the GPU runtime image fails validation when CUDA is not present.

kind: TrainJob
metadata:
generateName: smoke-
namespace: kubeflow-admin-cpaas-io

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Avoid hardcoded namespace in the shared smoke manifest.

Line 13 fixes the namespace to kubeflow-admin-cpaas-io; users often apply examples in their own Profile namespace, which then breaks runtime resolution (spec.runtimeRef.name must exist in the same namespace).

Suggested fix
 metadata:
   generateName: smoke-
-  namespace: kubeflow-admin-cpaas-io
+  # Replace with your Profile namespace.
+  namespace: <your-namespace>
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/en/kubeflow/how_to/assets/training-runtimes/trainjob-smoke.yaml` at line
13, The manifest hardcodes namespace: kubeflow-admin-cpaas-io which breaks
runtime resolution because spec.runtimeRef.name must exist in the same
namespace; remove the fixed "namespace" line (or replace it with a
template/placeholder that uses the target Profile namespace) so users can apply
the smoke manifest into their own Profile namespace, and add a brief note near
spec.runtimeRef.name to remind users that the referenced runtime must be created
in the same namespace as the manifest.

Comment on lines +39 to +40
Tag scheme is `v<semver>-build.<YYYYMMDDHHMMSS>`; each rebuild gets a unique
tag so scan evidence stays correlated.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Clarify the tag scheme or update example tags.

The documentation describes a tag scheme with build timestamps (v<semver>-build.<YYYYMMDDHHMMSS>), but the actual TrainingRuntime YAML files use simple semantic versions like v0.1.0. This inconsistency may confuse users about which tag format to expect when pinning versions.

Consider either:

  • Updating the description to match the actual tag format used in the manifests
  • Or adding a note explaining when timestamped tags vs simple semantic versions are used
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/en/kubeflow/how_to/training-runtimes.mdx` around lines 39 - 40, The doc
line describing the image tag scheme (v<semver>-build.<YYYYMMDDHHMMSS>)
conflicts with the TrainingRuntime manifests that use simple semantic tags like
v0.1.0; update the sentence in the docs to either (a) match the manifests by
stating that manifests currently use plain semantic tags (e.g., v0.1.0) or (b)
clarify both formats and when each is used by adding a short note that
timestamped tags (v<semver>-build.<YYYYMMDDHHMMSS>) are produced for rebuilt
images while released manifests are pinned to simple semantic tags (e.g.,
v0.1.0) — edit the paragraph that contains the tag scheme text to implement one
of these options and ensure the example tags in the doc align with the
TrainingRuntime YAMLs.

Comment thread docs/en/kubeflow/how_to/training-runtimes.mdx Outdated
Six curated TrainingRuntime images (PyTorch + LLaMA-Factory + TrainingHub on
NVIDIA CUDA, plus torch_npu / LLaMA-Factory / MindSpeed-LLM on Huawei Ascend
NPU via CANN) so users can submit TrainJobs without building a custom image.

- training-runtimes.mdx: device + framework catalog, HAMI vGPU / vNPU
  resource model, NPU host-driver caveat, smoke TrainJob walkthrough.
- assets/training-runtimes/*.yaml: ready-to-apply TrainingRuntime for each
  image plus a shared smoke TrainJob template.

Source Containerfiles, multi-arch build helper, e2e harness, and
post-fix security scan evidence live in alauda/kubeflow-plugin under
training-runtimes/.
@typhoonzero
typhoonzero force-pushed the codex/training-runtimes branch from 9751aa0 to 32f075a Compare June 5, 2026 01:37

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
docs/en/kubeflow/how_to/training-runtimes.mdx (1)

156-164: 💤 Low value

Terminology inconsistency: "adaptor" vs "adapter".

Line 160 uses "megatron_adaptor_v2" (likely the actual module name) while line 164 uses "adapter shim." While "adaptor" may be the correct module name, using both "adaptor" and "adapter" in the same paragraph could confuse readers about whether these refer to the same concept.

Consider adding a brief clarification, e.g., "...without going through the MindSpeed-LLM adaptor shim" to maintain consistent terminology.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/en/kubeflow/how_to/training-runtimes.mdx` around lines 156 - 164, The
paragraph mixes "adaptor" and "adapter" which is inconsistent; update the text
to use the module-correct spelling "adaptor" everywhere (e.g., change "adapter
shim" to "adaptor shim") so readers know "megatron_adaptor_v2" and the
MindSpeed-LLM shim refer to the same thing, and ensure mentions of mindspeed_llm
and MindSpeed-LLM use consistent casing/terminology.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@docs/en/kubeflow/how_to/training-runtimes.mdx`:
- Around line 156-164: The paragraph mixes "adaptor" and "adapter" which is
inconsistent; update the text to use the module-correct spelling "adaptor"
everywhere (e.g., change "adapter shim" to "adaptor shim") so readers know
"megatron_adaptor_v2" and the MindSpeed-LLM shim refer to the same thing, and
ensure mentions of mindspeed_llm and MindSpeed-LLM use consistent
casing/terminology.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 06b32e25-623d-4ee2-a869-8d8e2c135d04

📥 Commits

Reviewing files that changed from the base of the PR and between 9751aa0 and 32f075a.

📒 Files selected for processing (8)
  • docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cann8.5-arm64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/mindspeed-llm-cann8.5-arm64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cann8.5-arm64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cu126-amd64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/traininghub0.1-cu126-amd64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/trainjob-smoke.yaml
  • docs/en/kubeflow/how_to/training-runtimes.mdx
✅ Files skipped from review due to trivial changes (1)
  • docs/en/kubeflow/how_to/assets/training-runtimes/trainjob-smoke.yaml
🚧 Files skipped from review as they are similar to previous changes (5)
  • docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cann8.5-arm64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/traininghub0.1-cu126-amd64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/mindspeed-llm-cann8.5-arm64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/llamafactory0.9-cu126-amd64-trainingruntime.yaml
  • docs/en/kubeflow/how_to/assets/training-runtimes/torch2.6-cu126-amd64-trainingruntime.yaml

@typhoonzero
typhoonzero merged commit 0c97d8f into master Jun 5, 2026
3 checks passed
@typhoonzero
typhoonzero deleted the codex/training-runtimes branch June 5, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant