Uh oh!
There was an error while loading. Please reload this page.
K8s/XCalibur Support - #579
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| msg="", | ||
| ) | ||
| def log_iter( |
There was a problem hiding this comment.
Does this return all the logs or just rank 0? Kubeflow plugin has a specific check to avoid anything that isn't rank0.
There was a problem hiding this comment.
log_iter returns logs for all ranks/pods. It uses kubectl logs -l jobset.sigs.k8s.io/jobset-name= with --prefix — the label selector matches all pods in the JobSet (all nodes, all ranks), and --prefix prepends each line with the pod name so we can tell which rank it's from. The --max-log-requests is set to num_nodes * 2 to accommodate all pods simultaneously.
Uh oh!
There was an error while loading. Please reload this page.
| launch_path = os.path.join(self.job_dir, "launch.sh") | ||
| with open(launch_path, "w") as f: | ||
| f.write(script) | ||
| os.chmod(launch_path, 0o755) |
| val = labels.get("excalibur.nvidia.com/job") | ||
| if val and val != workloadrun_name: | ||
| return val | ||
| except json.JSONDecodeError: |
There was a problem hiding this comment.
Changed the "pass" to a DEBUG log statement.
Address review comments
Signed-off-by: Smitha Jayaram <sjayaram@nvidia.com>
Signed-off-by: Ajay <abalasa@nvidia.com>
…VIDIA-NeMo#595) * chore: add build constraint for setuptools version in pyproject.toml Added a build constraint to specify that setuptools must be less than version 82. This change ensures compatibility with existing dependencies. Signed-off-by: Ajay Balasa <abalasa@nvidia.com> * chore: add build constraint for setuptools version in uv.lock Included a build constraint to specify that setuptools must be less than version 82, ensuring compatibility with existing dependencies. Signed-off-by: Ajay Balasa <abalasa@nvidia.com> --------- Signed-off-by: Ajay Balasa <abalasa@nvidia.com>
This branch contains changes for adding the XCalibur executor backend to enable llm benchmarking runs in K8s.