std.Build.Watch: key fanotify file descriptors by mount id

[kevinboulain]

I hope it's okay to take over the inactive pull request #20672. The patch is very similar but I've taken into account the review comments: since indices correspond between the 'public' DirTable and 'private' (to the Linux implementation) HandleTable I've stashed the mount point information in there. However, the Windows implementation is quite similar and shares markAllFilesDirty so I had to add a small edge case, not sure what you'll think of it.
I've been using Zig for the past two weeks so apologies if I'm not following conventions. I've tested my change by following the wiki and since I'm on NixOS I'm hitting this problem immediately (same error message as #20670).

Fixes#20670.
Closes#20672.

[andrewrk]

Nice work, thanks!

[Profpatsch]

I’m still getting the error because it re-mounts /nix/store as read-only bind-mount here and fanotify does not like that:

> ~/kot/zig/zls/result/bin/zig build check -
fincremental --watch
Build Summary: 2/2 steps succeeded
check success
└─ compile exe sqlitefs-check Debug native success 362ms
error: unable to watch /nix/store/0vkz54zg4lmfr6rzqkmnznp2mfckb4r3-zig-0.15.0-dev.936+fc2c1883b/lib/std: NotSameFileSystem
error: the following build command failed with exit code 1:
.zig-cache/o/1c616aff675c3ca94f65a3d541a2f645/build /nix/store/0vkz54zg4lmfr6rzqkmnznp2mfckb4r3-zig-0.15.0-dev.936+fc2c1883b/bin/zig /nix/store/0vkz54zg4lmfr6rzqkmnznp2mfckb4r3-zig-0.15.0-dev.936+fc2c1883b/lib /home/philip/depot/users/Profpatsch/sqlitefs .zig-cache /home/philip/.cache/zig --seed 0xf78133c2 -Z10a067d96f509605 check -fincremental --watch

> mount
…
/dev/mapper/cryptroot on /nix/store type btrfs (ro,relatime,ssd,space_cache,subvolid=666,subvol=/root)
/dev/mapper/cryptroot on / type btrfs (rw,relatime,ssd,space_cache,subvolid=666,subvol=/root)
…

[kevinboulain]

(I'm out for the next few weeks so I won't be able to take a closer look for now.)

I'm not sure I understand your setup with the information you've pasted. Do I understand correctly that your source files are under /nix/store (Zig's stdlib) and your project, its dependencies and cache all are under /home/philip (i.e.: Zig's local cache, your project sources ~/kot/zig/zls, and what appears to be dependencies /home/philip/depot/users/Profpatsch/sqlitefs)?
Then I assume that neither your /home nor anything under it is a separate mountpoint and it's all part of / (as in /dev/mapper/cryptroot on /)?
Maybe you could confirm that the mount ID is the same for /nix/store and / then? That would mean it's not enough to serve as a key.

[Profpatsch]

How do I see the mount ID for a mountpoint?

[Profpatsch]

Ah, after reading the manual: /proc/self/mountinfo

31 1 0:28 /root / rw,relatime shared:1 - btrfs /dev/mapper/cryptroot rw,ssd,space_cache,subvolid=666,subvol=/root
30 31 0:28 /root/nix/store /nix/store ro,relatime shared:14 - btrfs /dev/mapper/cryptroot rw,ssd,space_cache,subvolid=666,subvol=/root

So the bind-mount has a different MountID

[Profpatsch]

The error description of EXDEV in fanotify_mark is quite

EXDEV The filesystem object indicated by pathname resides within a filesystem subvolume (e.g., btrfs(5)) which uses a different fsid than its root superblock. This error can be returned only with an fanotify group that identifies filesystem objects by file handles.

But I think it gets thrown because I’m using btrfs, and as always btrfs is a special little snowflake when it comes to syscall behaviour on Linux. Maybe a solution here could be to create a different watch group?

[Profpatsch]

I think I figured it out; I wasn’t aware that fanotify was a relatively recent kernel API, and btrfs support was only added in 6.8 (fsnotify/fsnotify#114 (comment)). I’m currently on 6.6 still.

SO I think the way to do this is to catch that EXDEV error and provide an error message that includes a hint, or alternatively fall back to the inotify interface.

[kevinboulain]

If it helps to pinpoint the root cause on your side, I've tested this patch with Zig's stdlib in the Nix store on its own Btrfs subvolume and my project's source directory on another Btrfs subvolume, and, IIRC, under Linux 6.15. That's why I asked about your setup because yours doesn't seem to actually involve multiple filesystems so it's a bit confusing you're hitting this error (it might not even be necessary to key by mount id since it seems to be the same underlying filesystem in your case).
The issue you've linked does sound like the likely culprit and if it is, I think your suggestion

SO I think the way to do this is to catch that EXDEV error and provide an error message that includes a hint

is fair since there's already a precedent:

zig/lib/std/Build/Watch.zig

Line 187 in 0cb558b

error.UnsupportedFlags=>fatal("fanotify_init failed due to old kernel; requires 5.17+", .{}),

[Profpatsch]

Okay, so, I updated to a newer kernel and now the fanotify watch works as expected.