fix(gelu): compute exact GELU via fp32 erf polynomial in cuTile and Tile C++

[nayanp38]

Description

Problem:
When using approximate="none", the tanh approximation of GELU is calculated rather than the exact GELU in both cuTile (standard_normal_cdf_ct) and tilecpp (normal_cdf_f32).

Further, the cuTile kernel calculates intermediate values in the storage dtype while tilecpp casts to fp32. This causes noticeable cancellations on negative inputs on bf16/fp16:

x = [-2.0, -2.5, -3.0, -3.5, -4.0] (bf16)
exact (fp64): [-0.04550, -0.01552, -0.00405, -0.00081, -0.00013]
cutile: [-0.04688, -0.01465, -0.00586, 0.00000, 0.00000] <-
tilecpp: [-0.04541, -0.01508, -0.00363, -0.00062, -0.00007]

The defect seems to be invisible to test_gelu.py, whose atol is too loose under the inputs spanning [-2.3, -1.8].

Changes:

• Implement erf via Abramowitz–Stegun 7.1.26 in both backends
• Upcast to fp32 in cuTile backend to match tilecpp
• Widen inputs to test_gelu.py and tighten tolerances
• Add bench_gelu.py, structured nearly identically to bench_silu_and_mul.py

Tanh mode is unchanged (but now recieves fp32 input in cuTile, resolving some fails on the tightened test_gelu.py found below).

Results:
Max error vs fp64 reference on inputs from [-6, 6]:

backend / mode / dtype	before	after
both / none / fp32	4.7e-4 (22% pct error)	4.7e-7 (0.022% pct error)
cutile / none / bf16	1.1e-2 (100%)	7.8e-3 (0.4%)
cutile / tanh / bf16	9.3e-3 (100%)	7.8e-3 (0.4%)
cutile / none / fp16	2.8e-3 (45%)	9.8e-4 (0.06%)

Now, we have backend agreement within <= 1 ULP on all inputs. I also tightened the tolerances and widened input ranges under test_gelu.py to reflect these changes, giving 2 output ULPs per dtype. The tests pass here but fail on the previous (current main) branch in 8 cases:

	none/fp32	none/fp16	none/bf16	tanh/fp32	tanh/fp16	tanh/bf16
cutile	✗	✗	✗	✓	✗	✗
tilecpp	✗	✗	✗	✓	✓	✓

No performance losses were measured on my RTX 3060ti (all measurements within +/- 1% before vs. after). Ran on the following env:
WSL2 Ubuntu 24.04
CUDA Toolkit 13.3
cuda-tile 1.5.0,
torch 2.9.1+cu130
triton 3.5.1

Not addressed here: running tanh on fp16 with M=1024 resulted in ~15-35 GB/s for both backends, both before and after these changes. All other tests were in the 380-400 GB/s range on my hardware However, in isolation (with a warmup), it runs at 385 GB/s (same as pyTorch). I'm unsure about why this specifically happens in this case, but possibly a point for future investigation or changes to bench_gelu.py to add a warmup

CI Configuration

config:
  build: true
  # valid options are "ops", "benchmark", and "sanity"
  test: [ops, benchmark]

Checklist

• Code formatted and imports sorted via repo specifications (./format.sh)
• Documentation updated (if needed)
• CI configuration reviewed

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[nayanp38]

Rebased onto current main and tests still pass.

Something I noticed after the rebase: cuTile geglu imports standard_normal_cdf_ct, and thus inherits the fix here. However, tilecpp geglu has its own normal_cdf_f32, which is tanh-based (so the backends differ there). I can mirror the fix there if helpful, but I'll wait for feedback here first. Thank you!

[hannahli-nv]

Hi @nayanp38, thanks for digging into this!
Could I ask you to hold for a few days before we merge? We had already scheduled the cuTile GELU backward for TileGym this week, and it lands in the same file. Once that's in, could you rebase and extend the change to the backward path?
I'll ping you here as soon as the backward commit is merged. Thank you for your understanding regarding the inconvenience.

[nayanp38]

Of course! Thank you for letting me know and I'll make those changes when the backward lands.

[nayanp38]

Hey @hannahli-nv, I've been looking over the GELU forward/backward pipeline with the new commits landing. Doesn't seem like any of them are relevant to this pr, but I had two quick questions on things I found:

1. The existing tilecpp gelu_bwd_kernel seems to be computing the exact GELU derivative in both modes. This means that using the approx path calculates the exact derivative on the backward pass.
2. cuTile geglu.py imports the standard_normal_cdf_ct from the gelu.py I just modified, so it's automatically fixed. But, tilecpp's geglu.cuh copies the normal_cdf_f32 helper, and therefore still uses approx GELU for both exact and approx paths.

Just wanted to ask if the above two are intended? I'm happy to apply fixes when I rebase, just let me know. Thank you!

[hannahli-nv]

Hi @nayanp38, sorry for the late reply, and thank you for digging into this. Both of the things you spotted are real bugs, not intentional.

• The fix for tilecpp gelu_bwd_kernel ignoring its mode parameter is essentially ready and should land in TileGym on next Monday, together with the cuTile GELU backward. Apologies for making you wait on it.

• geglu.cuh's own tanh-based normal_cdf_f32, we've left untouched. Could you please also fix it in this PR?

Thanks again for flagging both.