read1 in cbreak mode returns 0x0D (instead of 0x0A) when <enter> is entered.

[johannesnoordanus]

Bug report

Bug description:

A read1 of 'stdin' in cbreak mode returns 0x0D (instead of 0x0A) when <enter> is entered.
That is for Python version 3.12.1, version 3.11.6 returns 0x0A.

The following code demonstrates this. When run enter <enter>.

"""
cbreak <enter> test
"""

import sys
import tty
import select
import termios
from time import sleep

# save stdin attributes
prevStdinAttributes = termios.tcgetattr(sys.stdin)

# set cbreak mode:
# "Enter cbreak mode. In cbreak mode (sometimes called “rare” mode) normal tty line buffering
# is turned off and characters are available to be read one by one. However, unlike raw mode,
# special characters (interrupt, quit, suspend, and flow control) retain their effects on the
# tty driver and calling program. Calling first raw() then cbreak() leaves the terminal in cbreak mode."
tty.setcbreak(sys.stdin, when = termios.TCSANOW)

c = b''
# wait for a byte
while True:
 if select.select([sys.stdin], [], [], 0) == ([sys.stdin], [], []):
 c = sys.stdin.buffer.read1(1)
 break
 sleep(.1)

# restore stdin attributes
termios.tcsetattr(sys.stdin, termios.TCSADRAIN, prevStdinAttributes)

# print result byte in hex
print(f"bytes: '\\x{c.hex().upper()}'")

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Linux, macOS

Linked PRs

• gh-114328: tty cbreak mode should not alter ICRNL #114335
• [3.12] gh-114328: tty cbreak mode should not alter ICRNL (GH-114335) #114410

[ronaldoussoren]

Looks like this was introduced in 486bc8e, based on issue #85984, PR #101832. This user-visible change of behaviour is not mentioned anywhere though.

[johannesnoordanus]

Looks like this was introduced in 486bc8e, based on issue #85984, PR #101832. This user-visible change of behaviour is not mentioned anywhere though.

Linux and MacOs require/define <enter> (LF) mapped to 0x0A. This cannot be correct in my opinion (user-visible or not). See https://en.wikipedia.org/wiki/Newline (Unicode is similar.)

[ronaldoussoren]

Looks like this was introduced in 486bc8e, based on issue #85984, PR #101832. This user-visible change of behaviour is not mentioned anywhere though.

Linux and MacOs require/define (LF) mapped to 0x0A. This cannot be correct in my opinion (user-visible or not). See https://en.wikipedia.org/wiki/Newline (Unicode is similar.)

The change feels accidental given that it isn't mentioned anywhere (including in the discussion on the PR and issue)

cc @gpshead (who merged #101832)

[gpshead]

I'm assuming it's caused by these opening lines in the new tty.cfmakecbreak function:

def cfmakecbreak(mode):
 """Make termios mode cbreak."""
 # Do not map CR to NL on input.
 mode[IFLAG] &= ~(ICRNL)
 ...

@8vasu - any thoughts?

[gpshead]

To restore the behavior and retain compatibility we could presumably just remove that ICRNL clearing. It feels worth looking things up to reference where in what standard (posix?) this intent is defined one way or another in a comment.

presumably: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap11.html

[gpshead]

The posix doc is a bit hard to reason about w.r.t. this as it does not mention mode names as used in stty... But is what underlies all these things.

The Linux stty man page doesn't explicitly mention icrnl in cbreak AKA -icanon mode... suggesting it should just leave the bit alone.

There appear to be other opinions on this, AIX's "understanding terminals" docs explicitly say cbreak has ICRNL cleared.

Yet the very name cbreak implies to me in this context that the c stands for Canonical in which one could reasonably expect ICRNL to be set to make the read data... canonical. Regardless of keyboard or terminal type. (Enter aka Return keys typically producing a CR aka \r aka 0x0d - which some terminals remap to a NL aka \n aka 0x0a)

So I agree we should just remove the explicit clearing of ICRNL as a bugfix for this 3.12 code (and note this in our docs). I believe that means we just inherit the current setting for that flag, which is what our previous behavior had long been.

[gpshead]

The FreeBSD and macOS stty man pages appear to agree with my reading of "icrnl shouldn't be touched for cbreak".

[johannesnoordanus]

Question: this issue should be covered by a specific test I think?

[8vasu]

@gpshead

To my knowledge, the cbreak/rare mode is defined neither in the POSIX manual for stty (https://pubs.opengroup.org/onlinepubs/9699919799/) nor in the POSIX General Terminal Interface.

I think the authoritative source in this case is X/Open Curses, Issue 7 (https://pubs.opengroup.org/onlinepubs/9699909599/toc.pdf). A quick search though the pdf reveals the following definition of cbreak mode, which posits clearing ICRNL:

Characters typed by the user are immediately available to the
application and Curses does not perform special processing on
either the erase character or the kill character. An application can
select cbreak mode to do its own line editing but to let the abort
character be used to abort the task. This mode achieves the same
effect as non-canonical-mode, Case B input processing (with
MIN set to 1 and ICRNL cleared) as specified in the XBD
specification. The state of the ISIG and IXON flags are not
changed upon entering this mode.

For an example implementation, please refer to the source of NCURSES_SP_NAME(cbreak) in ncurses, which clears ICRNL: https://github.com/mirror/ncurses/blob/87c2c84cbd2332d6d94b12a1dcaf12ad1a51a938/ncurses/tinfo/lib_raw.c#L177

In my opinion, the functions in tty.py before #101832 were making arbitrary changes to the termios, but now they comply with standards, which might unfortunately break earlier behavior of Python. I tried to keep as much of earlier behavior intact as possible while complying to the standard; for example, before #101832, tty.setcbreak() was clearing ECHO, which I have retained (again, an arbitrary change not specified in the standard; the linked ncurses implementation is not doing this).