hal: C++ API (hal.hh) and pybind11 bindings on the new HAL API

[grandixximo]

@rene-dev@BsAtHome - this is the unification pass we discussed in the #4099 meeting: Rene's C++/pybind11 interface rebuilt on Bertho's getter/setter API, satisfying both constraints (no HAL internals access; simple, low-maintenance, self-documenting C++ surface).

Based on master, fully additive: _hal/hal.py are untouched and nothing is replaced. The by-name query functions and HAL_PORT creation are compiled out when the query API header is absent and activate automatically when #4247 lands.

What this adds

• src/hal/hal.hh - type-safe C++ layer, header-only, in namespace linuxcnc::hal:
  • hal::pin<T> typed handles via traits<T> over the rtapi_ types: rtapi_s32/rtapi_sint are distinct C++ types, so 32/64 accessor selection happens at compile time with zero cost. Handles re-read the shmem slot per access, so hal_link() rewrites stay visible (same semantics as C pin pointers). Handles are move-only; HAL objects are unique.
  • hal::pin_t variant + hal::anypin: runtime-typed access multiplexes on the variant tag (the stored-type pattern required for name-keyed collections), with the tag supplied by the query API.
  • hal::component: pins + params, newpin/newparam (typed and runtime-typed), add_pin for the struct-member idiom, item access, prefix handling.
  • ULAPI by-name functions (get_value, set_value, set_signal, component_exists, pin_has_writer, signal management) implemented entirely on hal_get_p/hal_set_p/hal_get_s/hal_set_s/hal_comp_by_name. No hal_priv.h, no direct shmem access, no re-implemented parsers.
• src/hal/halpybind.cc - pybind11 module (halpp.so), adapted from @rene-dev's bindings, extended with params and error paths. Built alongside_hal/hal.py; module replacement is a separate (3.0) discussion.
• src/emc/task/taskclass.cc - converted to the new API (it was the only hal.hh consumer).
• tests/halpp/ - Python and native C++ smoke suites (both green), plus python3-pybind11 build dep.

Safety properties worth noting

• Query callbacks are strictly exception-free: conversion errors are reported by flag and thrown after the library releases the HAL mutex. (Throwing through a callback wedges the session-wide mutex; found the hard way, and tested.)
• Set-side coercion uses the item's type with range checks; an overflow raises instead of truncating.
• Typed access compiles to the same two-load sequence as the C API and the old pointer dereference, so there is no hot-path cost.

Relationship to the original pybind11 branch

Kept: the variant/map/PyPin structure and the binding shape.
Dropped: the private set_common string parser (superseded by hal_set_p), hal_mutex_guard, the get_info_signals stub, waitWritable (was non-functional).
Compatibility aliases (hal_comp, hal_pin<T>, hal_dir, PyPin) are provided so existing consumers of the old hal.hh keep working.

Test evidence

Full tree builds; tests/halmodule.0/.1 pass; new smoke suites cover component lifecycle, typed pins/params (incl. negative S32 and 64-bit values), signals, linking, by-name get/set with coercion, port pins, and error paths. The suites auto-skip query-API checks when built without the query API, so they run on both master and the #4247 tree.

[grandixximo]

This seems to be working, but I am not sure if it is the right shape we want, these are the direction I took on the issue raised in the meeting, subject to revision.

Problem	Solution
Handle is width-blind	Type lives in traits<T> (compile time) or the variant tag (runtime, from q->pp.type)
hal_link rewrites the handle slot	pin<T> stores a pointer to the slot and re-reads per access (C semantics, staleness impossible)
Throwing through query callbacks wedges the HAL mutex session-wide	Exceptions captured in the callback, rethrown after the library releases the mutex
Handle storage must be shmem	hal_malloc'd hal_refs_u slot per item (halmodule's own pattern)

@BsAtHome A bare hal_sint_t is width-blind, and wrapping foreign raw handles across widths is unsafe in the interim. But the C++ layer never selects from the handle alone. Component-created pins carry the type at construction via traits (int32_t vs int64_t are distinct C++ types, compile-time selection), and the runtime path uses q->pp.type from your query API to pick the variant alternative, the stored-type-tag multiplex you described in #4099. The only unsafe path is explicitly reinterpreting a raw handle, which can be guarded with a query lookup. So the layer works transitionally, it just gets simpler after the break.

[rene-dev]

oh, that was quick. I think this should be separate from the other big PR, as this can go in master fairly quickly. the only reason I stopped was that buster had no pybind11 package. this should also replace the current pyhal/halmodule instead of adding a third module. and, the same cpp api should be used in xhc, task and other places which are implemented in cpp.

[grandixximo]

Buster is basically not supported anymore with master, I will fix the review points tomorrow, thank you both ☺️
My bad on missing the include, getting late time for bed...

[BsAtHome]

There is another issue. The new hal_lib uses a reference counted init/exit and has separated out hal_lib_init() and hal_lib_exit(). The reason to take this out is that you do not need a component for pin/param/signal read/write or query interactions. You only need to have mapped shared memory.

Calling hal_init() to create a component (if you want to create pins) now calls the factored out hal_lib_init() and hal_exit()calls the factored out hal_lib_exit().

The python class must initialize using hal_lib_init() and terminate by calling hal_lib_exit(). If the user has not called hal_exit() on any created components, you will now get a proper error message (because the user forgot to terminate correctly).

[rene-dev]

can you rename the module to halmodule, so that the UIs and the tests that are already in place use the new bindings, and remove the old ones?

[grandixximo]

can you rename the module to halmodule, so that the UIs and the tests that are already in place use the new bindings, and remove the old ones?

Replacing halmodule is the goal, agreed, but renaming now would break things for two separate reasons.

First, a hard technical one: on master the new bindings cannot cover the existing Python surface. get_info_pins/signals/params, pin_has_writer and component_exists are used by qtvcp (qt_halobjects, hal_selectionbox) and hal_glib, and they need the query API (hal_list_*, hal_get_p) from #4247. Without it, the options are a strict subset module (GUIs break immediately) or private shmem access (exactly what this API change abolishes).

Second, the compatibility tail: hal.error exception type, KeyboardInterrupt on halcmd unload (documented in hal.py's docstring, user comps wrap their main loop in it), the exact get_info_* dict shapes (tests/halmodule.0 compares verbatim), the hal.stream object API (tests/halmodule.1), setitem coercion quirks, pin .dir/.type attributes, and the hal.py wrapper shims. Matching all of that is what makes the swap a user-visible event, and it deserves its own PR with a full parity pass, not a rename here.

So the proposal stands as: land this additive now (hal.hh + halpp alongside, nothing replaced), and do the rename + replacement as a follow-up once #4247 is in. Your old branch's consumer sweep (hal_glib, qtvcp core, raster, tests) is the starting checklist for that PR. Is this acceptable? Open for discussion...

[rene-dev]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

[BsAtHome]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

[rene-dev]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

What is halquery?

[BsAtHome]

I see that #4231 changes halmodule.cc, we should also add those functions via c++/pybind11

No, all of this should be in halquery.

What is halquery?

That is the new interface to all of HAL's internals nicely wrapped up from the hal query interface without someone needing to dig into HAL's internals. Hal_lib's internals are fully isolated in #4247 and nobody should be writing code that circumvents that. Inclusion of hal_priv.h is strictly forbidden in any code that is not part of hal_lib.
Halmodule deals with components and everything else was "bolted on". You do not want one big blob of code to do it all. You want modularization, which halquery provides.

[rene-dev]

ah, found it. I would much prefer if it would be a c++ api, which can be used to automatically generate the python bindings.
most of userspace code is c++ anyway. I have maintained the c bindings in the past(pyhton2->3 migration), and do not ever want to deal with them again.

[BsAtHome]

It is a pure C version, so it does not have the same problems as C++ interfacing C-based Python ;-)

[grandixximo]

This is exactly the split #4251 (this PR) implements: the query API stays pure C in the library (Bertho's layer, no bypass possible), and the C++ API sits on top of it (hal.hh), which is where Python bindings get generated (halpybind) , no hand-written CPython code. halquery.c proves the query API is sufficient; the same functions can be exposed as pybind wrappers over hal_get_p/hal_list_* once #4247 lands. So the endgame can be: C query API in hal_lib, C++ API above it, Python bindings generated from the C++ layer, and the hand-written C modules retired on the same schedule. Happy to demonstrate the halquery-equivalent functions in halpybind as a follow-up once the base is in.

[BsAtHome]

Fine by me to get it C++ wrapped. It does not yet have priority one.

There is one thing that should be fixed in both halmodule and halquery: use IntEnum types for both type and direction. It may be necessary to have a haltype module with the enums so you it can be properly shared between the other modules. Not sure how this can be handled properly in using consistent namespace names and hierarchy.

[grandixximo]

A shared pure-Python haltype.py with HALType(IntEnum) and HALDir(IntEnum) is the cleanest shape. IntEnum members are ints, so the C boundary stays plain int and everything interops both ways; hal, halquery and the pybind module import and re-export from it, keeping hal.HAL_FLOAT working unchanged. I would keep the namespace flat (sibling of hal.py) rather than a linuxcnc.* package, which breaks every existing import hal. One pybind11 note: py::enum_ does not emit real IntEnum objects, that needs py::native_enum (pybind11 >= 3.0, trixie ships 2.13), so for now the bindings take constants from haltype.py and we flip to native_enum when the distro catches up.

[BsAtHome]

A shared pure-Python haltype.py with HALType(IntEnum) and HALDir(IntEnum) is the cleanest shape. IntEnum members are ints, so the C boundary stays plain int and everything interops both ways; hal, halquery and the pybind module import and re-export from it, keeping hal.HAL_FLOAT working unchanged.

That would be the simple way. If it works, then it should be nice. The real benefit is in halquery to have the iteration result show the right name instead of a number. There it is big value. Probably also need to add the same for other constants.

I would keep the namespace flat (sibling of hal.py) rather than a linuxcnc.* package, which breaks every existing import hal.

Global namespace naming and fixing hierarchy is a problem we need to fix in a later version. This would be a version 3.0 thing, I guess ;-)

One pybind11 note: py::enum_ does not emit real IntEnum objects, that needs py::native_enum (pybind11 >= 3.0, trixie ships 2.13), so for now the bindings take constants from haltype.py and we flip to native_enum when the distro catches up.

Ha, and so much for "C++ is easy to maintain" ;-)
The version and support problem always pops up regardless of language and library. You always end in hell once you need to support over long periods of time and different distros and distro versions. At least it is warm in hell.

[grandixximo]

Going back to draft, waiting for master new API to materialize ;-)

[grandixximo]

Branch rewritten and pushed (33012db => 63c8a05), four commits on current master:

• hal.hh replaces the deprecated upstream header with the new C++ API (typed pin/param handles, runtime-typed anypin, component, streams), built strictly on the public C API and the query API
• haltype.py: stdlib-only HALType/HALDir IntEnums plus tests/haltype.0 as drift guard against the _hal constants
• streams: linuxcnc::hal::stream and halpp.stream with _hal stream semantics; tests/halpp now runs under runtests
• hal.query registration prototype: empty _hal.query submodule, hal.py publishes it in sys.modules, tests/halquery.0 checks the four import paths

Build notes: the pybind11 bindings are skipped with a note when the headers are absent, not failed, so the tree still builds without python3-pybind11; tests/halpp self-skips in that case. hal.hh is exported to include/ alongside hal.h. tests/halpp/test.sh compiles cpp_test.cc against the tree and runs it in a live HAL session, so the native C++ side has CI coverage too.

Still draft on purpose: #4247 moves the same ground (hal.hh, the type system, HAL isolation), so this waits for that to land before the query bindings follow as hal.query.

[grandixximo]

Branch reworked after feedback Bertho sent me in a private mail (63c8a05 => 7beaaf3). The haltype.py commit is gone, replaced by:

• src/hal/halenum.hh builds the two IntEnum classes from the hal.h constants through the plain Python C API: one native source of truth, nothing to drift, and any extension module can instantiate the same classes, pybind11 or not.
• _hal registers the shared classes as _hal.type/hal.dir. hal.py serves hal.type.REAL, hal.dir.IN etc. (HAL* spellings are aliases), plus the HALType/HALDir class names so pickle round-trips.
• halpp drops its private py::enum_ copies and casts hal_type_t/hal::dir arguments and results through the shared classes, so tags print with their names and compare equal to the plain constants, which stay exported for compatibility.
• Canonical members are the platform-stable spellings (BOOL, REAL, S32, U32, PORT, S64, U64 and IN..RW); SINT/UINT follow the platform width and are aliases.

pybind11 note: real enum.IntEnum via py::native_enum needs pybind11 >= 3.0 and trixie ships 2.13, so construction goes through the enum module's functional API. The classes are genuine IntEnums either way, so moving to py::native_enum later (TODO in the header) is invisible to user code.

Query results stay plain dicts as discussed; the future hal.query bindings tag type/dir with these classes, no per-function wrappers. tests/haltype.0 is now a sanity suite for the classes, tests/halpp checks halpp shares them. All HAL suites pass.

[BsAtHome]

These must be ordered differently. I found out in interactive tests that the auto-complete uses the order when duplicates are found (braindead, yes). Order: bool, real, sint, uint, port, s32, u32, s64, u64 (also for the prefixed).