[NVVM] Support - Followup enhancements

[abhilash1910]

Description

Issue Link - #981

Changes to be addressed in this WIP PR:

• LTO IR testing
• Is there a way to add multiple modules?
  {If / when it is possible to add multiple modules, a test with code that uses something from libdevice is probably a good idea.
  It's also useful to be able to lazily add a module}
• apply bitcode pattern input for libnvvm

cc @leofang

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[leofang]

Thanks, @abhilash1910! Any ETA to wrap this up?

[abhilash1910]

pre-commit.ci autofix

[abhilash1910]

pre-commit.ci autofix

[leofang]

Thanks, Abhilash! Leaving a few early feedbacks.

[abhilash1910]

pre-commit.ci autofix

[rwgk]

Compiling the same Program instance multiple times re-adds libdevice here multiple times. Is that a concern? Could it make sense to add call-once logic?

[rwgk]

The below is Cursor-generated, based on my suspicion. I'll continue to work on Option 1 with Cursor and will post an update here (hopefully very quick).

---

I think there is still a potential race in the current call-once logic.

Current pattern:

ifself._use_libdeviceandnotself._libdevice_added:
...
withnogil:
nvvmLazyAddModuleToProgram(...)
self._libdevice_added=True

Why this can race:

• with nogil releases the GIL.
• Another thread can enter Program.compile() on the same Program instance while the first thread is inside the nogil block.
• If that second thread checks _libdevice_added before the first thread stores True, both threads can pass the guard and both call nvvmLazyAddModuleToProgram(...).

So:

• Single-threaded usage is fine.
• Concurrent compile on the same Program can still double-add libdevice.

Potential fixes:

1. Per-instance lock (strongest)
   Guard the add/compile section with a lock to serialize concurrent compile calls per Program.

2. Set-before-call with rollback (minimal change)
   Set _libdevice_added = True before entering with nogil, and if add fails, reset it to False in except.
   This removes the check/set gap that currently exists around the GIL release.

[rwgk]

Sorry, the order of options got shuffled, I meant Option 1. Corrected above.

[rwgk]

/ok to test f7b9c4f

[rwgk]

@abhilash1910 I powered through all the pathfinder-related things I could think of. I think that part is in merge-ready shape. I left a few comments on the cuda_core program changes; filtered things Cursor found.

[abhilash1910]

pre-commit.ci autofix

[abhilash1910]

pre-commit.ci autofix

[abhilash1910]

/ok to test 51f6009

[abhilash1910]

Thanks @rwgk for the additions, addressed the suggestions.

[rwgk]

I pushed commit 701f54b to take care of the race. I'm testing locally. As soon as I see that passing (5-10 minutes) I'll trigger the CI here.

[rwgk]

/ok to test 701f54b

[rwgk]

Assuming the CI passes, this looks merge-ready to me.

[rwgk]

The one build failure is this:

 sccache: error: Timed out waiting for server startup. Maybe the remote service is unreachable?
Run with SCCACHE_LOG=debug SCCACHE_NO_DAEMON=1 to get more information
error: command '/host//home/runner/_work/_tool/sccache/0.14.0/arm64/sccache' failed with exit code 2

We need to rerun after the first attempt is done.