Skip to content

K8s/XCalibur Support - #579

Draft
sjayaram-nv wants to merge 8 commits into
NVIDIA-NeMo:mainfrom
sjayaram-nv:sjayaram/k8s-excalibur
Draft

K8s/XCalibur Support#579
sjayaram-nv wants to merge 8 commits into
NVIDIA-NeMo:mainfrom
sjayaram-nv:sjayaram/k8s-excalibur

Conversation

@sjayaram-nv

Copy link
Copy Markdown

This branch contains changes for adding the XCalibur executor backend to enable llm benchmarking runs in K8s.

Comment threadnemo_run/run/torchx_backend/schedulers/xcalibur.py
Comment threadnemo_run/core/execution/xcalibur.py Outdated
msg="",
)

def log_iter(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this return all the logs or just rank 0? Kubeflow plugin has a specific check to avoid anything that isn't rank0.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log_iter returns logs for all ranks/pods. It uses kubectl logs -l jobset.sigs.k8s.io/jobset-name= with --prefix — the label selector matches all pods in the JobSet (all nodes, all ranks), and --prefix prepends each line with the pod name so we can tell which rank it's from. The --max-log-requests is set to num_nodes * 2 to accommodate all pods simultaneously.

Comment threadnemo_run/run/torchx_backend/schedulers/xcalibur.py
Comment threadnemo_run/core/execution/xcalibur.py Outdated
launch_path = os.path.join(self.job_dir, "launch.sh")
with open(launch_path, "w") as f:
f.write(script)
os.chmod(launch_path, 0o755)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed it to 0o555 now.

Comment threadnemo_run/core/execution/xcalibur.py Outdated
val = labels.get("excalibur.nvidia.com/job")
if val and val != workloadrun_name:
return val
except json.JSONDecodeError:

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed the "pass" to a DEBUG log statement.

sjayaram-nvand others added 7 commits August 11, 2026 17:40
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
Signed-off-by: Ajay <abalasa@nvidia.com>
…VIDIA-NeMo#595)
* chore: add build constraint for setuptools version in pyproject.toml
Added a build constraint to specify that setuptools must be less than version 82. This change ensures compatibility with existing dependencies.
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
* chore: add build constraint for setuptools version in uv.lock
Included a build constraint to specify that setuptools must be less than version 82, ensuring compatibility with existing dependencies.
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
---------
Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@sjayaram-nv@sudostock@github-advanced-security@balasaajay