Skip to content

Take the desktop picker id through a flat varying, not gl_PrimitiveID - #6773

Closed
Fedr wants to merge 1 commit into
fedr/guard-picked-pointfrom
fedr/flat-pick-id
Closed

Take the desktop picker id through a flat varying, not gl_PrimitiveID#6773
Fedr wants to merge 1 commit into
fedr/guard-picked-pointfrom
fedr/flat-pick-id

Conversation

@Fedr

@Fedr Fedr commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Stacked on #6772 (the picked-point guard), so this PR's diff is the shader change alone. GitHub will retarget it to master once #6772 merges.

What was wrong

The Ubuntu 26.04 UI test picked face -1090332440 out of a 512-face mesh, which is what #6772's new guard reports:

[warning] pointOnObjectToPickedPoint: not a valid mesh pick: face=-1090332440, faceSize=512

On desktop, RenderMeshObject::renderPicker uses MeshDesktopPicker (MRRenderMeshObject.cpp:157), whose fragment shader took the primitive id from uint(gl_PrimitiveID). That is the channel that came back wrong. The decisive detail is what came back right: uniGeomId, a plain uniform written into the neighbouring channel of the very same pick pixel, resolved to a real mesh object — otherwise the guard would never have reached the mesh branch. One channel of one fragment correct, the other garbage, points at gl_PrimitiveID itself rather than at the FBO, the read-back or the decode.

The read-back is not at fault: GL_RGBA32UI texture, glClearBufferuiv to 0xffffffff, glReadPixels(..., GL_RGBA_INTEGER, GL_UNSIGNED_INT, ...) — internally consistent. Nor is the id arithmetic: every renderer that draws for picking sets primBucketSize (mesh 3 and 1, points 1, label 1; features delegate through RenderObjectCombinator), so there is no division by zero.

What this does

The picker vertex shader already computes uint primId = uint(gl_VertexID) / primBucketSize for the corner-mode path. This passes that same value down a flat integer varying and reads it in place of gl_PrimitiveID:

  • flat out uint primitiveIdFlat; / primitiveIdFlat = primId; in the picker vertex shader, desktop only.
  • the non-corner-mode fragment path reads primitiveIdFlat.

Emscripten keeps gl_PrimitiveID untouched behind #ifndef __EMSCRIPTEN__, and the varying is declared only in the mode that reads it, so LinesPicker/LinesJointPicker keep linking against vertex shaders that do not provide it.

There is precedent for the platform split a few files over: getMeshFragmentShaderColoringBlock() already picks gl_PrimitiveID on desktop and the interpolated float pair on Emscripten.

Evidence it works

Same MeshInspector UI suite, same ubuntu26 image, with this commit in the MeshLib gitlink:

before after
guard warning face=-1090332440 none
crashes SIGSEGV in toTriPoint, then a second in findVisibleFaces 0
scenarios died on the first pick of create_feature_sphere features and select suites run to completion, 5 of 6 suites pass

The remaining failure is unrelated: the text scenario misses its mesh-similarity threshold by 0.0009 (0.9941 vs 0.995), which looks like a freetype/font difference on Resolute and is a baseline question, not a pick one.

MRViewer.vcxproj builds clean locally and MeshViewer -hidden -noEventLoop -unloadPluginsAtEnd exits 0 on Windows — though that only exercises the C++, since the picker shader is compiled lazily on the first pick.

Note on scope

No disable-build-* labels: this changes shader generation shared by every desktop GL platform, so Windows, macOS and the vcpkg legs all want to run, and the Emscripten leg is what proves the untouched ES branch still compiles.

The Ubuntu 26.04 UI test picked face -1090332440 out of a 512-face mesh.
On desktop RenderMeshObject::renderPicker uses MeshDesktopPicker, whose
fragment shader took the id from gl_PrimitiveID, and that is the one
channel of the pick pixel that came back wrong - uniGeomId, a uniform in
the neighbouring channel, was right. llvmpipe 21.1.8 / Mesa 26.0.8 is the
only stack where it happens; 22.04 and 24.04 pass.

The vertex shader already computes the id as uint(gl_VertexID) /
primBucketSize for the corner-mode path, so pass that same value down a
flat integer varying and read it instead. Emscripten keeps gl_PrimitiveID
untouched behind #ifndef, and the varying is declared only in the mode
that reads it so the lines pickers keep linking.
@Fedr

Fedr commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #6774, which carries the same commit rebased onto master so it no longer depends on #6772. Force-push is blocked on this repo's branches, so the rebase had to go to a new branch; fedr/flat-pick-id stays alive because MeshInspectorCode#7746 currently pins its MeshLib gitlink to 00cf6e9 for testing.

@Fedr Fedr closed this Sep 3, 2026
@Fedr
Fedr deleted the fedr/flat-pick-id branch September 7, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant