Kokkos::View related segfaults when using the SYCL backend

[ThomasJannaudCAM]

Hi all,

In Cambridge we just received some new Intel Ponte Vecchio GPUs. With a software engineer, we have been trying to run IDEFIX on them. She was able to compile Idefix on a number of the test cases using the SYCL backend provided by Kokkos, but at runtime it segfaults with an error about Kokkos::View. We are wondering if this is a fault with Idefix or with Kokkos. She has been able to reproduce it on two different PVC systems (each with slightly different environments - see below).

Thanks in advance !

Below is her description of how to reproduce the error:

In the test/MHD/disk directory type:

cmake ${IDEFIX_DIR} -DIdefix_MPI=ON -DKokkos_ENABLE_SYCL=ON -DKokkos_ARCH_INTEL_PVC=ON -DCMAKE_CXX_COMPILER=mpiicpx

then when it has finished configuring the build type ‘make'.

Run the Idefix executable (I used 4 MPI ranks but it doesn’t really matter, I’ve also tried with 1):

mpirun -np 4 ./idefix

This is the output:

jk945@sw-pvc03:~/idefix/test/MHD/disk $ mpirun -np 4 ./idefix
[0] MPI startup(): ===== GPU topology on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): NUMA nodes : 2
[0] MPI startup(): GPUs : 4
[0] MPI startup(): Stacks (Tiles) : 8
[0] MPI startup(): Hierarchy mode : flat
[0] MPI startup(): NUMA Id
GPU Id Stacks (tiles) Ranks on this NUMA
[0] MPI startup(): 0 0,1 (0,1)(2,3) 0,1
[0] MPI startup(): 1 2,3 (4,5)(6,7) 2,3
[0] MPI startup(): ===== GPU pinning on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): Rank
Pin stack (tile)
[0] MPI startup(): 0
{0,1}
[0] MPI startup(): 1
{2,3}
[0] MPI startup(): 2
{4,5}
[0] MPI startup(): 3
{6,7}
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
[0] MPI startup(): Intel(R) MPI Library, Version 2021.13 Build 20240515 (id: df72937)
[0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation. All rights reserved.
[0] MPI startup(): library kind: release
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
[0] MPI startup(): libfabric version: 1.20.1-impi
[0] MPI startup(): libfabric provider: mlx
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx_400_x1.dat" not found
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx_400.dat" not found
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx.dat" not found
[0] MPI startup(): Load tuning file: "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi.dat"
[0] MPI startup(): Number of NICs: 1 [0] MPI startup(): ===== NIC pinning on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): Rank Thread id Pin nic
[0] MPI startup(): 0 0 mlx
[0] MPI startup(): 1 0 mlx
[0] MPI startup(): 2 0 mlx
[0] MPI startup(): 3 0 mlx
[0] MPI startup(): ===== CPU pinning =====
[0] MPI startup(): Rank Pid Node name Pin cpu
[0] MPI startup(): 0 2296614 [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27}
[0] MPI startup(): 1 2296615 [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) {28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54
,55}
[0] MPI startup(): 2 2296616 [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) {56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82
,83}
[0] MPI startup(): 3 2296617 [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) {84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,
108,109,110,111}
.:HMMMMHn:. ..:n..
.H*'`` `'%HM'''''!x.
:x x*` .(MH: `#h.
x.`M M> :nMMMMMMMh. `n.
*kXk.. XL nnx:.XMMMMMMMMMMML .. 4X.
)MMMMMx 'M `^?M*MMMMMMMMMMMM:HMMMHHMM.
MMMMMMMX ?k 'X ..'*MMMMMMM.#MMMMMMMMMx
XMMMMMMMX 4: M:MhHxxHHHx`MMx`MMMMMMMMM>
XM!` ?M `x 4MM'`''``HHhMMX 'MMMMMMMM
4M M `: *> `` .('MX '*MMMM'
MX `X.nnx.. ..XMx` 'M*X
?h. ''```^'*!Hx. :Mf xHMh M**MMM 4L`
`*Mx `'*n.x. 4M> :M` `` 'M ` %
'% ``*MHMX X> !
:! `#MM> X> ` :x
:M ?M `X . ..'M
XX .!*X `x XM( MMx`h
'M>:: `M: `+ MMX XMM `:
'M> M 'X 'MMX ?MMk.Xx..
'M> ?L ...:! MMX.H**'MMMM*h
M> #L :!'`MM. . X*`.xHMMMMMnMk.
`! #h. :L XM'*hxHMM*MhHMMMMMMMMMM'#h
+ XMh: 4! x :f MM' `*MMMMMMMMMM% `X
M Mf``tHhxHM M> 4k xxX' `#MMMMMMMf `M .>
:f M `MMMMM: M> M!MMM: '*MMf' 'MH*
! Xf 'MMMMMX `X X>'h.` :P*Mx. .d*~..
:M X 4MMMMM> ! X~ `Mh. .nHL..M#'%nnMhH!'`
XM d> 'X`'**h 'h M ^'MMHH+*'` '''' `'**'
%nxM> *x+x.:. XL.. `k `::X
:nMMHMMM:. X> Mn`*MMMMMHM: `: ?MMn.
`'**MML M> 'MMhMMMMMMMM # `M:^*x
^*MMttnnMMMMMMMMMMMH>. M:.4X
`MMMM>X ( .MMM:MM! .
`'''4x.dX +^ `''MMMMHM?L..
``' `'`'`'`
Idefix version 2.2.00-54482d58
Built against Kokkos 40500
Compiled on Mar 5 2025 at 17:00:09
Main: initialization stage.
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
[0x6c5ab0] [0x69f3e2] [0x421166] [0x4210fd] [0x5c27b2] [0x48f765] [0x67d06f] [0x14acd1a29d90] [0x14acd1a29e40] __libc_start_main
[0x40eec5] [0x6c5ab0] [0x69f3e2] [0x421166] [0x4210fd] [0x5c27b2] [0x48f765] [0x67d06f] [0x15455c029d90] [0x15455c029e40] __libc_start_main
[0x40eec5] [0x6c5ab0] [0x69f3e2] [0x421166] [0x4210fd] [0x5c27b2] [0x48f765] [0x67d06f] [0x14f2af629d90] [0x14f2af629e40] __libc_start_main
[0x40eec5] [0x6c5ab0] [0x69f3e2] [0x421166] [0x4210fd] [0x5c27b2] [0x48f765] [0x67d06f] [0x151e84429d90] [0x151e84429e40] __libc_start_main
[0x40eec5] 

I do not see this error with the CUDA backend.

Here are the versions of Intel oneAPI packages that I have loaded:

jk945@sw-pvc03:~/idefix/test/MHD/disk $ module list
Currently Loaded Modulefiles:
1) intel/tbb/2021.13 2) intel/compiler-rt/2024.2.0 3) intel/compiler-intel-llvm/2024.2.0 4) intel/mpi/2021.13 5) intel/mkl/2024.2 6) glibc/2.35/gcc/spmhgmal 7) intel-oneapi-dpl/2022.5.0/oneapi/jxepuwvw 8) ncurses/6.2/gcc-7.2.0-awbmlbc 9) cmake/3.22.3/gcc-7.2.0-ffrdpvs

[glesur]

can't really help much as we don't have any intel gpu here. A few thoughts:

• you should try to run in serial first (ie without MPI enabled)
• try to enable IDEFIX_Debug=ON (in addition to KOKKOS_Debug, as done above)
• can you tell me the output of cmake (when called from a clean build directory)

[glesur]

additional thoughts: maybe add -DKokkos_ENABLE_SERIAL=ON to cmake, not sure it is enabled by default with syscl.

[julianakwan]

Hi @glesur - thanks very much for your suggestions.
I have tried a serial build with both Idefix and Kokkos debugging enabled but the same error with Kokkos::View still occurs.
This is the line I use to compile:
cmake ${IDEFIX_DIR} -DKokkos_ENABLE_SYCL=ON -DKokkos_ARCH_INTEL_PVC=ON -DKokkos_ENABLE_DEBUG=ON -DIdefix_DEBUG=ON

Here is the output from cmake

-- The C compiler identification is IntelLLVM 2025.0.0
-- The CXX compiler identification is IntelLLVM 2025.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icx - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icpx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting default Kokkos CXX standard to 17
-- Kokkos version: 4.5.0
-- The project name is: Kokkos
-- Using internal gtest for testing
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=gnu++17 for C++17 extensions as feature
-- Looking for SYCL_EXT_ONEAPI_DEVICE_GLOBAL
-- Looking for SYCL_EXT_ONEAPI_DEVICE_GLOBAL - found
-- Looking for SYCL_EXT_ONEAPI_GRAPH
-- Looking for SYCL_EXT_ONEAPI_GRAPH - found
-- Built-in Execution Spaces:
-- Device Parallel: Kokkos::SYCL
-- Host Parallel: NoTypeDefined
-- Host Serial: SERIAL
-- -- Architectures:
-- INTEL_PVC
-- Found TPLLIBDL: /usr/include -- Looking for C++ include oneapi/dpl/execution
-- Looking for C++ include oneapi/dpl/execution - found
-- Looking for C++ include oneapi/dpl/algorithm
-- Looking for C++ include oneapi/dpl/algorithm - found
-- Performing Test KOKKOS_NO_TBB_CONFLICT
-- Performing Test KOKKOS_NO_TBB_CONFLICT - Success
-- Using internal desul_atomics copy
-- Experimental mdspan support is enabled
-- Looking for C++ include experimental/mdspan
-- Looking for C++ include experimental/mdspan - not found
-- Looking for C++ include mdspan
-- Looking for C++ include mdspan - not found
-- Using internal mdspan directory /lus/flare/projects/grteclyn/idefix/src/kokkos/tpls/mdspan/include
-- Kokkos Backends: SERIAL;SYCL
-- Including problem-specific CMakeLists in '/lus/flare/projects/grteclyn/idefix/test/MHD/sod'
-- Explicitely setting Idefix_MHD = ON
-- Idefix final configuration
-- MHD: ON
-- MPI: OFF
-- HDF5: OFF
-- Python: OFF
-- Reconstruction: Linear
-- Precision: Double
-- Version: 2.2.00-54482d58
-- Problem definitions: 'definitions.hpp'
-- Configuring done (27.4s)
-- Generating done (2.8s)

Then I build using make and everything is fine at this stage.
When I run the executable, this is the output:

./idefix
----> Profiler::Init...
----> ...returned
.:HMMMMHn:. ..:n..
.H*'`` `'%HM'''''!x.
:x x*` .(MH: `#h.
x.`M M> :nMMMMMMMh. `n.
*kXk.. XL nnx:.XMMMMMMMMMMML .. 4X.
)MMMMMx 'M `^?M*MMMMMMMMMMMM:HMMMHHMM.
MMMMMMMX ?k 'X ..'*MMMMMMM.#MMMMMMMMMx
XMMMMMMMX 4: M:MhHxxHHHx`MMx`MMMMMMMMM>
XM!` ?M `x 4MM'`''``HHhMMX 'MMMMMMMM
4M M `: *> `` .('MX '*MMMM'
MX `X.nnx.. ..XMx` 'M*X
?h. ''```^'*!Hx. :Mf xHMh M**MMM 4L`
`*Mx `'*n.x. 4M> :M` `` 'M ` %
'% ``*MHMX X> !
:! `#MM> X> ` :x
:M ?M `X . ..'M
XX .!*X `x XM( MMx`h
'M>:: `M: `+ MMX XMM `:
'M> M 'X 'MMX ?MMk.Xx..
'M> ?L ...:! MMX.H**'MMMM*h
M> #L :!'`MM. . X*`.xHMMMMMnMk.
`! #h. :L XM'*hxHMM*MhHMMMMMMMMMM'#h
+ XMh: 4! x :f MM' `*MMMMMMMMMM% `X
M Mf``tHhxHM M> 4k xxX' `#MMMMMMMf `M .>
:f M `MMMMM: M> M!MMM: '*MMf' 'MH*
! Xf 'MMMMMX `X X>'h.` :P*Mx. .d*~..
:M X 4MMMMM> ! X~ `Mh. .nHL..M#'%nnMhH!'`
XM d> 'X`'**h 'h M ^'MMHH+*'` '''' `'**'
%nxM> *x+x.:. XL.. `k `::X
:nMMHMMM:. X> Mn`*MMMMMHM: `: ?MMn.
`'**MML M> 'MMhMMMMMMMM # `M:^*x
^*MMttnnMMMMMMMMMMMH>. M:.4X
`MMMM>X ( .MMM:MM! .
`'''4x.dX +^ `''MMMMHM?L..
``' `'`'`'`
Idefix version 2.2.00-54482d58
Built against Kokkos 40500
Compiled on Mar 11 2025 at 11:14:32
Main: initialization stage.
----> Grid::Grid(Input)...
----> ...returned
----> GridHost::GridHost(Grid)...
----> ...returned
----> GridHost::MakeGrid...
----> ...returned
----> GridHost::SyncToDevice...
----> ...returned
----> DataBlock::DataBlock...
--------> GridHost::GridHost(Grid)...
--------> ...returned
--------> GridHost::SyncFromDevice...
--------> ...returned
--------> DataBlock::ExtractSubdomain...
------------> idefix_for(coordinates)...
------------> ...returned
------------> idefix_for(coordinates)...
------------> ...returned
------------> idefix_for(coordinates)...
------------> ...returned
--------> ...returned
--------> DataBlock::MakeGeometry()...
------------> idefix_for(Volumes)...
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_dx0")
Backtrace:
[0x6069d0] [0x5e02d2] [0x411d76] [0x41af1d] [0x51e5c8] [0x45879b] [0x5d0c2a] [0x14d60757124d] __libc_start_main
[0x40ef0a] Aborted

[julianakwan]

Hi @glesur - I'm happy to report that that worked! There was a lot of output because I compiled on debug so let me know if you want to see it. I also checked with MPI (but always 1 MPI rank to 1 GPU) and it was fine as well. Thanks so much!

[ThomasJannaudCAM]

Thank you both very much !

[glesur]

Excellent, I'll propagate the fix to the develop branch. @ThomasJannaudCAM , whenever you have time, I'd like to know how it performs on Intel PVCs on the OT3D test with 128^3 points and no tracer.

[ThomasJannaudCAM]

What would you want ? Frequency of cell updates and parallel efficiency ?

[glesur]

yes, cell update/s (don't really care about the parallel efficiency for now), as in https://idefix.readthedocs.io/latest/performances.html

[glesur]

A fix for this has been merged into develop (#331) and will be part of the next Idefix release.