Translate the serial line and modem-control ioctls by jotpalch · Pull Request #314 · sysprog21/elfuse

jotpalch · 2026-08-20T17:00:00Z

Note

Piece 2 of #310, on top of the CBAUD/CIBAUD work merged in #312.

Summary

Thirteen serial line-control and modem-control ioctls fell through to ENOTTY. They now route to the Darwin equivalents, so pyserial/esptool/picocom-style tools (and any program calling tcflush() on its tty) work on a passthrough serial port.

Important

What to look at:

The drain before TCSBRK/TCSBRKP/TIOCSBRK is interruptible: the kernel's tty_wait_until_sent aborts on a pending signal with a plain -EINTR (no restart), so the arms poll TIOCOUTQ in short slices with the io_wait_fd_or_interrupted() interruption checks instead of parking the vCPU thread in macOS tcdrain(), which guest signals cannot reach; a flow-control-stalled port would otherwise hang the guest unkillably. Only TCSBRK arg 0 actually breaks (arg != 0 is glibc/musl tcdrain()), and TIOCCBRK does not drain, matching the kernel's prep block.
TCFLSH/TCXONC selectors are renumbered explicitly: Linux 0-based (asm-generic/termbits-common.h), Darwin 1-based (sys/termios.h). Out-of-range -> EINVAL, as the kernel.
The TIOCM set family masks the guest word to the settable lines the way tty_tiocmset() does (Linux mask DTR|RTS|OUT1|OUT2|LOOP; Darwin has no OUT1/OUT2/LOOP, so the intersection is DTR|RTS), dropping read-only status bits silently as Linux does. TIOCMSET is emulated with a TIOCMBIS/TIOCMBIC pair because Linux only touches masked bits while Darwin's TIOCMSET overwrites the whole word. Bit values themselves are identical on both. A pty answers ENOTTY on both systems, and the driver check precedes the value (a status-bits-only TIOCMBIS is still ENOTTY on a pty).
Every new arm closes host_ref on all paths.

Problem

sys_ioctl default arm (io.c:2573 at bffd6bd) returns ENOTTY for: TCSBRKTCSBRKPTCFLSHTCXONCTIOCOUTQTIOCEXCLTIOCNXCLTIOCSBRKTIOCCBRKTIOCMGETTIOCMSETTIOCMBISTIOCMBIC.

Who hits it (from source): pyserial open() -> _reset_input_buffer() -> termios.tcflush (serialposix.py:344,677, fatal); setDTR/setRTS -> TIOCMBIS/TIOCMBIC (:706-715); out_waiting -> TIOCOUTQ (:755); break_condition -> TIOCSBRK/TIOCCBRK (:300-301); esptool reset sequences -> TIOCMSET/TIOCMGET (esptool/reset.py:22-25); glibc/musl tcdrain() -> TCSBRK 1; Python termios.tcflush on the controlling tty fails today for any guest.

Change

linux-wire.h: LINUX_TCSBRK/TCXONC/TCFLSH/TIOCEXCL/TIOCNXCL/TIOCOUTQ/TCSBRKP/TIOCSBRK/TIOCCBRK/TIOCMGET/TIOCMBIS/TIOCMBIC/TIOCMSET, values from asm-generic/ioctls.h (same on aarch64 and x86_64).
io.csys_ioctl: arms routing to tcdrain/tcsendbreak/tcflush/tcflow and host ioctl(TIOCOUTQ|TIOCEXCL|TIOCNXCL|TIOCSBRK|TIOCCBRK|TIOCM*).
tests/test-pty.c: tcflush/tcdrain/tcflow on the slave; TCFLSH(3) -> EINVAL; TIOCOUTQ; TIOCEXCL/TIOCNXCL; TIOCMGET on a pty is ENOTTY.

Evidence

macOS pty, glibc 2.41 guest: before vs after

before:

== 4. line / modem-control ioctls ==
TIOCMGET = -1Inappropriate ioctl for device bits=0x0
TIOCMBIS(DTR|RTS)= -1Inappropriate ioctl for device
TIOCMBIC(DTR|RTS)= -1Inappropriate ioctl for device
tcflush/TCFLSH = -1Inappropriate ioctl for device
tcdrain/TCSBRK = -1Inappropriate ioctl for device
TIOCOUTQ = -1Inappropriate ioctl for device outq=-1
TIOCEXCL = -1Inappropriate ioctl for device
host-side speed after guest tcsetattr(B115200): ispeed=9600 ospeed=9600

after:

== 4. line / modem-control ioctls ==
TIOCMGET = -1Inappropriate ioctl for device bits=0x0
TIOCMBIS(DTR|RTS)= -1Inappropriate ioctl for device
TIOCMBIC(DTR|RTS)= -1Inappropriate ioctl for device
tcflush/TCFLSH = 0
tcdrain/TCSBRK = 0
TIOCOUTQ = 0 outq=0
TIOCEXCL = 0
host-side speed after guest tcsetattr(B115200): ispeed=115200 ospeed=115200

real hardware: ESP32-S3 (/dev/cu.usbmodem101), reproducer before vs after

before:

== 4. line / modem-control ioctls ==
TIOCMGET = -1Inappropriate ioctl for device bits=0x0
TIOCMBIS(DTR|RTS)= -1Inappropriate ioctl for device
TIOCMBIC(DTR|RTS)= -1Inappropriate ioctl for device
tcflush/TCFLSH = -1Inappropriate ioctl for device
tcdrain/TCSBRK = -1Inappropriate ioctl for device
TIOCOUTQ = -1Inappropriate ioctl for device outq=-1
TIOCEXCL = -1Inappropriate ioctl for device

after:

== 4. line / modem-control ioctls ==
TIOCMGET = 0 bits=0x6
TIOCMBIS(DTR|RTS)= 0
TIOCMBIC(DTR|RTS)= 0
tcflush/TCFLSH = 0
tcdrain/TCSBRK = 0
TIOCOUTQ = 0 outq=0
TIOCEXCL = 0

Linux esptool 5.3.1 (python:3.12-slim arm64 sysroot) on the ESP32-S3: before vs after

before (dies inside Serial.open()):

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/esptool/loader.py", line 443, in __init__
self._port.open()
File "/usr/local/lib/python3.12/site-packages/serial/serialposix.py", line 344, in open
self._reset_input_buffer()
File "/usr/local/lib/python3.12/site-packages/serial/serialposix.py", line 677, in _reset_input_buffer
termios.tcflush(self.fd, termios.TCIFLUSH)
termios.error: (25, 'Inappropriate ioctl for device')

after (--before usb-reset; see note):

00:15:50 WARN src/core/sysroot.c:409: sysroot /private/tmp/claude-501/-Users-chenweicheng-claude-project/287bec26-be2d-4f79-81dd-14d93c69f673/scratchpad/sysroot-esptool is case-insensitive; workloads with colliding guest names such as Linux kernel trees may fail. Use --create-sysroot to run inside a case-sensitive APFS sparsebundle.
esptool v5.3.1
Serial port /dev/cu.usbmodem101:
Connecting...
Detecting chip type... ESP32-S3
Connected to ESP32-S3 on /dev/cu.usbmodem101:
Chip type: ESP32-S3 (QFN56) (revision v0.1)
Features: Wi-Fi, BT 5 (LE), Dual Core + LP Core, 240MHz, Embedded PSRAM 8MB (AP_3v3)
Crystal frequency: 40MHz
MAC: 34:85:18:42:6c:98
Uploading stub flasher...
Running stub flasher...
Stub flasher running.
Flash Memory Information:
=========================
Manufacturer: c8
Device: 4017
Detected flash size: 8MB
Flash type set in eFuse: quad (4 data lines)
Flash voltage set by eFuse: 3.3V
Hard resetting via RTS pin...

--before usb-reset is needed because esptool picks its reset strategy from the port's VID/PID, which pyserial on Linux reads from /sys/class/tty/<name>/device/... (serial/tools/list_ports_linux.py:30-53); that sysfs view is piece 3 in #310, not this PR.

Tests

build/elfuse build/test-pty: 70 passed, 0 failed (on top of Honor CBAUD/CIBAUD in TCGETS, TCGETS2 and TCSETS #312, rebased onto current main)
make check ...: rc=0, All 77 tests passed
clang-format --dry-run -Werror clean

Not in this PR

TIOCGSERIAL/TIOCSSERIAL stay ENOTTY (no Darwin equivalent; pyserial only uses them in the opt-in set_low_latency_mode()).
TCSBRKP duration is approximated: Darwin tcsendbreak() ignores the duration (fixed ~0.4 s) vs Linux arg*100 ms.
Device-node aliases (/dev/ttyUSB*, /dev/ttyACM*) and the /sys/class/tty walk -> piece 3 in Map Linux USB-serial device nodes (/dev/ttyUSB*, /dev/ttyACM*) onto macOS /dev/cu.* and translate the serial termios/modem ioctls #310.

Summary by cubic

Translate Linux serial line-control and modem-control ioctls to Darwin so passthrough ttys work with pyserial/esptool/picocom and any program calling tcflush()/tcdrain(). Previously these ioctls returned ENOTTY; now they route to tcdrain/tcsendbreak/tcflush/tcflow and TIOC* equivalents with Linux-matching behavior where it affects semantics.

Drain semantics before TCSBRK/TCSBRKP/TIOCSBRK match Linux: poll TIOCOUTQ in short slices and return plain -EINTR (no restart), avoiding vCPU hangs in macOS tcdrain(); documented in scripts/check-eintr-contract.py. Only TCSBRK arg 0 sends break; nonzero drains.
Selector mapping: TCFLSH (0..2 -> TCIFLUSH/TCOFLUSH/TCIOFLUSH) and TCXONC (0..3 -> TCOOFF/TCOON/TCIOFF/TCION) are renumbered to Darwin’s 1-based values; out-of-range -> EINVAL. TCSBRKP duration is accepted but approximated by Darwin.
Modem control: TIOCMGET/TIOCMBIS/TIOCMBIC forwarded; TIOCMSET emulated with BIS/BIC so only masked bits change. Mask to DTR|RTS (Darwin lacks OUT1/OUT2/LOOP). Bit values are identical; guest 32-bit ints preserved. ptys continue to return ENOTTY.
TIOCOUTQ and TIOCEXCL/TIOCNXCL forwarded. New request numbers added in src/syscall/linux-wire.h. Tests in tests/test-pty.c cover flush/drain/flow, selector validation, TIOCOUTQ, exclusivity, and pty ENOTTY.

^{Written for commit 79d3e0a. Summary will update on new commits.}

jserv · 2026-08-20T17:51:03Z

 return 0;
 }

+/* Interruptible output drain, mirroring tty_wait_until_sent(): the kernel waits


Be precise about "the kernel" while specifying macOS, Linux, or general cases.

Fair point. Reworded in 79d3e0a: every "the kernel" in this diff now names Linux explicitly ("the Linux kernel waits", "Linux returns a plain -EINTR"), and the commit message follows. The macOS side was already named at each mention.

A Linux guest driving a USB serial adapter (pyserial, esptool, picocom) issues TCFLSH, TCSBRK (tcdrain), TCXONC, TCSBRKP, TIOCSBRK/TIOCCBRK, TIOCOUTQ, TIOCEXCL/NXCL and TIOCMGET/MSET/MBIS/MBIC; all of them fell through to ENOTTY, so pyserial's open() died at termios.tcflush() and esptool never reached its DTR/RTS reset sequence. tcflush() on the controlling tty fails the same way for any guest program. The line-control arms route to the POSIX wrappers, since macOS has no ioctl form. The Linux kernel waits for pending output before TCSBRK, TCSBRKP and TIOCSBRK (drivers/tty/tty_io.c) and a pending signal aborts that wait with a plain -EINTR, so the drain is emulated by polling TIOCOUTQ in short slices with the interruption checks from io_wait_fd_or_interrupted() rather than parking the vCPU thread in macOS tcdrain(), which guest signals cannot reach -- a serial port stalled by flow control would otherwise hang the guest unkillably -- and the EINTR forbids the SVC restart, as the Linux one does. The tcflush/tcflow selectors are renumbered explicitly (Linux 0-based in asm-generic/termbits-common.h, macOS 1-based in sys/termios.h). The modem-control set family masks the guest's word to the output lines the way tty_tiocmset() does before it reaches the driver: Darwin has no OUT1/OUT2/LOOP, so the settable intersection is DTR|RTS, and read-only status bits are silently dropped as on Linux. TIOCMSET means "set = val & mask, clear = ~val & mask" on Linux while Darwin's TIOCMSET overwrites the whole word, so it is emulated with a TIOCMBIS/TIOCMBIC pair. TIOCM_* bit values themselves are identical on both (asm-generic/termios.h vs sys/ttycom.h). A pty answers ENOTTY on both systems, only a real serial driver implements them. All request numbers come from asm-generic/ioctls.h and so are the same for aarch64 and x86_64 (Rosetta) guests. tests/test-pty.c covers tcflush/tcdrain/tcflow, TCFLSH selector validation, TIOCOUTQ, TIOCEXCL/NXCL, the ENOTTY of TIOCMGET on a pty, and that a status-bits-only TIOCMBIS still reports ENOTTY on a pty (the driver check precedes the value on Linux).

jserv · 2026-08-21T03:19:22Z

Thank @jotpalch for contributing!

jotpalch mentioned this pull request Aug 20, 2026
Map Linux USB-serial device nodes (/dev/ttyUSB*, /dev/ttyACM*) onto macOS /dev/cu.* and translate the serial termios/modem ioctls #310
Open

This comment was marked as resolved.
Sign in to view

jotpalch force-pushed the serial-ioctls branch 2 times, most recently from 84a2d76 to ebdd0abCompare August 20, 2026 17:21

jserv reviewed Aug 20, 2026
View reviewed changes

jotpalch force-pushed the serial-ioctls branch from ebdd0ab to 79d3e0aCompare August 21, 2026 02:30

jserv merged commit 6ceabe2 into sysprog21:mainAug 21, 2026
14 checks passed

jotpalch mentioned this pull request Aug 21, 2026
Alias USB serial devices as ttyACM and ttyUSB jotpalch/elfuse#9
Draft

Navigation Menu

Uh oh!

Translate the serial line and modem-control ioctls - #314

Translate the serial line and modem-control ioctls#314
jserv merged 1 commit into
sysprog21:mainfrom
jotpalch:serial-ioctls

jotpalch commented Aug 20, 2026•
edited by cubic-dev-ai Bot
Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

jservAug 20, 2026

Uh oh!

jotpalchAug 21, 2026

Uh oh!

Uh oh!

jserv commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jotpalch commented Aug 20, 2026• edited by cubic-dev-ai BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Change

Evidence

Tests

Not in this PR

Summary by cubic

Uh oh!

This comment was marked as resolved.

Uh oh!

jservAug 20, 2026

Choose a reason for hiding this comment

Uh oh!

jotpalchAug 21, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jserv commented Aug 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jotpalch commented Aug 20, 2026•
edited by cubic-dev-ai Bot
Loading