`scan_exclusive_sum` regresses in CCCL 3.4 on Ampere

[bernhardmgruber]

[bernhardmgruber]

The regression:

## [0] NVIDIA A100 80GB PCIe

|  T{ct}  |  OffsetT{ct}  |  Elements{io}  |   Ref Time |   Ref Noise |   Cmp Time |   Cmp Noise |       Diff |   %Diff |  Status  |
|---------|---------------|----------------|------------|-------------|------------|-------------|------------|---------|----------|
|   I8    |      I32      |      2^16      |  10.267 us |       4.27% |  10.181 us |       3.73% |  -0.086 us |  -0.84% |  🔵 SAME  |
|   I8    |      I32      |      2^20      |  15.171 us |       8.99% |  15.360 us |      11.92% |   0.189 us |   1.25% |  🔵 SAME  |
|   I8    |      I32      |      2^24      |  69.645 us |       1.75% |  69.675 us |       1.75% |   0.030 us |   0.04% |  🔵 SAME  |
|   I8    |      I32      |      2^28      | 939.473 us |       0.34% | 936.451 us |       0.31% |  -3.022 us |  -0.32% |  🟢 FAST  |
|   I8    |      I64      |      2^16      |  10.685 us |       4.93% |  10.718 us |       5.38% |   0.033 us |   0.31% |  🔵 SAME  |
|   I8    |      I64      |      2^20      |  15.159 us |       3.79% |  15.280 us |       7.91% |   0.121 us |   0.80% |  🔵 SAME  |
|   I8    |      I64      |      2^24      |  69.706 us |       1.81% |  81.486 us |       1.17% |  11.780 us |  16.90% |  🔴 SLOW  |
|   I8    |      I64      |      2^28      | 940.342 us |       0.33% |   1.128 ms |       0.26% | 187.927 us |  19.98% |  🔴 SLOW  |
|   I8    |      I64      |      2^32      |  14.886 ms |       0.18% |  17.893 ms |       0.16% |   3.007 ms |  20.20% |  🔴 SLOW  |
|   I16   |      I32      |      2^16      |  11.186 us |       3.78% |  11.192 us |       3.63% |   0.006 us |   0.05% |  🔵 SAME  |
|   I16   |      I32      |      2^20      |  17.396 us |      14.61% |  16.997 us |      10.56% |  -0.399 us |  -2.29% |  🔵 SAME  |
|   I16   |      I32      |      2^24      |  74.877 us |       2.23% |  86.547 us |       1.46% |  11.670 us |  15.59% |  🔴 SLOW  |
|   I16   |      I32      |      2^28      | 994.368 us |       0.29% |   1.179 ms |       0.35% | 185.131 us |  18.62% |  🔴 SLOW  |
|   I16   |      I64      |      2^16      |  11.051 us |       4.34% |  11.247 us |       4.37% |   0.196 us |   1.77% |  🔵 SAME  |
|   I16   |      I64      |      2^20      |  16.723 us |      15.40% |  16.618 us |      13.61% |  -0.105 us |  -0.63% |  🔵 SAME  |
|   I16   |      I64      |      2^24      |  74.020 us |       2.05% |  85.880 us |       1.51% |  11.860 us |  16.02% |  🔴 SLOW  |
|   I16   |      I64      |      2^28      | 988.783 us |       0.35% |   1.181 ms |       0.28% | 191.942 us |  19.41% |  🔴 SLOW  |
|   I16   |      I64      |      2^32      |  15.696 ms |       0.17% |  18.755 ms |       0.18% |   3.060 ms |  19.49% |  🔴 SLOW  |
|   I32   |      I32      |      2^16      |  11.072 us |       4.24% |  11.075 us |       4.17% |   0.003 us |   0.03% |  🔵 SAME  |
|   I32   |      I32      |      2^20      |  18.323 us |       4.43% |  18.337 us |       4.63% |   0.014 us |   0.08% |  🔵 SAME  |
|   I32   |      I32      |      2^24      | 102.912 us |       2.05% | 108.508 us |       2.17% |   5.596 us |   5.44% |  🔴 SLOW  |
|   I32   |      I32      |      2^28      |   1.433 ms |       1.20% |   1.514 ms |       1.13% |  81.037 us |   5.66% |  🔴 SLOW  |
|   I32   |      I64      |      2^16      |  11.053 us |       4.50% |  11.101 us |       4.28% |   0.048 us |   0.44% |  🔵 SAME  |
|   I32   |      I64      |      2^20      |  17.591 us |       4.74% |  17.805 us |       4.07% |   0.214 us |   1.22% |  🔵 SAME  |
|   I32   |      I64      |      2^24      | 100.918 us |       2.17% | 106.189 us |       2.03% |   5.271 us |   5.22% |  🔴 SLOW  |
|   I32   |      I64      |      2^28      |   1.424 ms |       0.78% |   1.505 ms |       0.76% |  80.533 us |   5.65% |  🔴 SLOW  |
|   I32   |      I64      |      2^32      |  22.823 ms |       0.52% |  24.197 ms |       0.53% |   1.374 ms |   6.02% |  🔴 SLOW  |
|   I64   |      I32      |      2^16      |  11.889 us |       4.63% |  11.846 us |       4.33% |  -0.043 us |  -0.36% |  🔵 SAME  |
|   I64   |      I32      |      2^20      |  23.606 us |       5.62% |  24.204 us |      10.55% |   0.598 us |   2.53% |  🔵 SAME  |
|   I64   |      I32      |      2^24      | 184.656 us |       1.23% | 191.123 us |       1.41% |   6.467 us |   3.50% |  🔴 SLOW  |
|   I64   |      I32      |      2^28      |   2.718 ms |       1.36% |   2.796 ms |       1.19% |  78.252 us |   2.88% |  🔴 SLOW  |
|   I64   |      I64      |      2^16      |  11.844 us |       4.41% |  11.939 us |       4.33% |   0.096 us |   0.81% |  🔵 SAME  |
|   I64   |      I64      |      2^20      |  23.636 us |       6.31% |  23.933 us |       5.78% |   0.297 us |   1.25% |  🔵 SAME  |
|   I64   |      I64      |      2^24      | 180.878 us |       1.49% | 188.025 us |       1.76% |   7.147 us |   3.95% |  🔴 SLOW  |
|   I64   |      I64      |      2^28      |   2.676 ms |       0.57% |   2.754 ms |       0.61% |  77.775 us |   2.91% |  🔴 SLOW  |
|   I64   |      I64      |      2^32      |  42.813 ms |       0.10% |  44.181 ms |       0.16% |   1.367 ms |   3.19% |  🔴 SLOW  |
|  I128   |      I32      |      2^16      |  19.288 us |       2.51% |  19.246 us |       2.51% |  -0.042 us |  -0.22% |  🔵 SAME  |
|  I128   |      I32      |      2^20      |  66.547 us |       1.93% |  57.789 us |       1.56% |  -8.757 us | -13.16% |  🟢 FAST  |
|  I128   |      I32      |      2^24      | 579.237 us |       0.41% | 641.352 us |       0.20% |  62.115 us |  10.72% |  🔴 SLOW  |
|  I128   |      I32      |      2^28      |   8.741 ms |       0.32% |   9.935 ms |       0.06% |   1.194 ms |  13.66% |  🔴 SLOW  |
|  I128   |      I64      |      2^16      |  19.094 us |       2.82% |  19.042 us |       2.80% |  -0.053 us |  -0.28% |  🔵 SAME  |
|  I128   |      I64      |      2^20      |  66.029 us |       2.12% |  57.960 us |       1.58% |  -8.070 us | -12.22% |  🟢 FAST  |
|  I128   |      I64      |      2^24      | 579.649 us |       0.50% | 641.118 us |       0.23% |  61.469 us |  10.60% |  🔴 SLOW  |
|  I128   |      I64      |      2^28      |   8.782 ms |       0.09% |   9.936 ms |       0.06% |   1.154 ms |  13.15% |  🔴 SLOW  |
|   F32   |      I32      |      2^16      |  10.678 us |       4.87% |  10.673 us |       4.75% |  -0.005 us |  -0.05% |  🔵 SAME  |
|   F32   |      I32      |      2^20      |  18.320 us |       3.05% |  18.317 us |       2.76% |  -0.003 us |  -0.02% |  🔵 SAME  |
|   F32   |      I32      |      2^24      | 103.104 us |       1.93% | 106.319 us |       2.28% |   3.215 us |   3.12% |  🔴 SLOW  |
|   F32   |      I32      |      2^28      |   1.462 ms |       0.43% |   1.530 ms |       0.41% |  68.243 us |   4.67% |  🔴 SLOW  |
|   F32   |      I64      |      2^16      |  10.564 us |       4.51% |  10.549 us |       4.51% |  -0.016 us |  -0.15% |  🔵 SAME  |
|   F32   |      I64      |      2^20      |  18.449 us |       2.93% |  18.287 us |       3.17% |  -0.162 us |  -0.88% |  🔵 SAME  |
|   F32   |      I64      |      2^24      | 103.120 us |       1.90% | 106.137 us |       2.22% |   3.017 us |   2.93% |  🔴 SLOW  |
|   F32   |      I64      |      2^28      |   1.475 ms |       0.43% |   1.545 ms |       0.46% |  69.334 us |   4.70% |  🔴 SLOW  |
|   F32   |      I64      |      2^32      |  23.002 ms |       0.52% |  24.095 ms |       0.56% |   1.093 ms |   4.75% |  🔴 SLOW  |
|   F64   |      I32      |      2^16      |  11.481 us |       3.85% |  11.464 us |       3.69% |  -0.017 us |  -0.15% |  🔵 SAME  |
|   F64   |      I32      |      2^20      |  31.127 us |      32.64% |  30.481 us |      32.36% |  -0.647 us |  -2.08% |  🔵 SAME  |
|   F64   |      I32      |      2^24      | 196.618 us |       1.34% | 196.386 us |       1.25% |  -0.232 us |  -0.12% |  🔵 SAME  |
|   F64   |      I32      |      2^28      |   2.881 ms |       1.04% |   2.877 ms |       1.05% |  -4.030 us |  -0.14% |  🔵 SAME  |
|   F64   |      I64      |      2^16      |  11.480 us |       3.71% |  11.466 us |       3.88% |  -0.013 us |  -0.12% |  🔵 SAME  |
|   F64   |      I64      |      2^20      |  25.094 us |       2.29% |  24.950 us |       2.03% |  -0.144 us |  -0.57% |  🔵 SAME  |
|   F64   |      I64      |      2^24      | 194.245 us |       1.60% | 194.073 us |       1.51% |  -0.172 us |  -0.09% |  🔵 SAME  |
|   F64   |      I64      |      2^28      |   2.863 ms |       0.52% |   2.860 ms |       0.49% |  -2.530 us |  -0.09% |  🔵 SAME  |
|   F64   |      I64      |      2^32      |  45.897 ms |       0.06% |  45.841 ms |       0.06% | -56.107 us |  -0.12% |  🟢 FAST  |
|   C32   |      I32      |      2^16      |  13.366 us |       3.78% |  13.398 us |       3.64% |   0.032 us |   0.24% |  🔵 SAME  |
|   C32   |      I32      |      2^20      |  31.411 us |       2.22% |  31.744 us |       0.00% |   0.333 us |   1.06% |  🔴 SLOW  |
|   C32   |      I32      |      2^24      | 278.289 us |       0.78% | 293.789 us |       0.72% |  15.500 us |   5.57% |  🔴 SLOW  |
|   C32   |      I32      |      2^28      |   4.276 ms |       0.61% |   4.499 ms |       0.59% | 222.993 us |   5.21% |  🔴 SLOW  |
|   C32   |      I64      |      2^16      |  13.697 us |       3.90% |  13.691 us |       4.04% |  -0.007 us |  -0.05% |  🔵 SAME  |
|   C32   |      I64      |      2^20      |  31.331 us |       2.21% |  31.599 us |       2.05% |   0.267 us |   0.85% |  🔵 SAME  |
|   C32   |      I64      |      2^24      | 275.081 us |       0.89% | 290.196 us |       0.81% |  15.116 us |   5.49% |  🔴 SLOW  |
|   C32   |      I64      |      2^28      |   4.231 ms |       0.27% |   4.454 ms |       0.25% | 223.073 us |   5.27% |  🔴 SLOW  |
|   C32   |      I64      |      2^32      |  68.719 ms |       0.06% |  72.213 ms |       0.08% |   3.493 ms |   5.08% |  🔴 SLOW  |

was introduced with #6811

Comparing the commit at #6811 with branch/3.4.x still has the following regression:

|   F32   |      I32      |      2^16      |  10.673 us |       4.75% |  10.900 us |       4.57% |      0.227 us |   2.13% |  🔵 SAME  |
|   F32   |      I32      |      2^20      |  18.317 us |       2.76% |  18.790 us |       3.77% |      0.473 us |   2.58% |  🔵 SAME  |
|   F32   |      I32      |      2^24      | 106.319 us |       2.28% | 110.930 us |       1.73% |      4.610 us |   4.34% |  🔴 SLOW  |
|   F32   |      I32      |      2^28      |   1.530 ms |       0.41% |   1.604 ms |       0.33% |     73.863 us |   4.83% |  🔴 SLOW  |
|   F32   |      I64      |      2^16      |  10.549 us |       4.51% |  10.971 us |       4.44% |      0.423 us |   4.01% |  🔵 SAME  |
|   F32   |      I64      |      2^20      |  18.287 us |       3.17% |  18.073 us |       3.21% |     -0.215 us |  -1.17% |  🔵 SAME  |
|   F32   |      I64      |      2^24      | 106.137 us |       2.22% | 111.127 us |       1.84% |      4.990 us |   4.70% |  🔴 SLOW  |
|   F32   |      I64      |      2^28      |   1.545 ms |       0.46% |   1.626 ms |       0.38% |     81.417 us |   5.27% |  🔴 SLOW  |
|   F32   |      I64      |      2^32      |  24.095 ms |       0.56% |  25.467 ms |       0.50% |      1.373 ms |   5.70% |  🔴 SLOW  |
|   F64   |      I32      |      2^16      |  11.464 us |       3.69% |  11.756 us |       4.42% |      0.292 us |   2.55% |  🔵 SAME  |
|   F64   |      I32      |      2^20      |  30.481 us |      32.36% |  26.534 us |      13.72% |     -3.947 us | -12.95% |  🔵 SAME  |
|   F64   |      I32      |      2^24      | 196.386 us |       1.25% | 196.332 us |       1.35% |     -0.054 us |  -0.03% |  🔵 SAME  |
|   F64   |      I32      |      2^28      |   2.877 ms |       1.05% |   2.878 ms |       1.04% |      0.892 us |   0.03% |  🔵 SAME  |
|   F64   |      I64      |      2^16      |  11.466 us |       3.88% |  11.522 us |       3.86% |      0.055 us |   0.48% |  🔵 SAME  |
|   F64   |      I64      |      2^20      |  24.950 us |       2.03% |  28.023 us |      21.94% |      3.073 us |  12.32% |  🔴 SLOW  |
|   F64   |      I64      |      2^24      | 194.073 us |       1.51% | 193.241 us |       1.51% |     -0.832 us |  -0.43% |  🔵 SAME  |
|   F64   |      I64      |      2^28      |   2.860 ms |       0.49% |   2.860 ms |       0.51% |      0.255 us |   0.01% |  🔵 SAME  |
|   F64   |      I64      |      2^32      |  45.841 ms |       0.06% |  45.844 ms |       0.06% |      2.536 us |   0.01% |  🔵 SAME  |

So #6811 may not be the only offender.

[bernhardmgruber]

#6811 introduced _CCCL_GRID_CONSTANT annotations for the scan kernel :S

[bernhardmgruber]

Removing grid annotations recovers performance.