Skip to content

[3.15] gh-111487: Fix csv.Sniffer never testing the minimum delimiter consistency#154872

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pikammmmm wants to merge 2 commits into
base:3.15
Choose a base branch
from
Open

[3.15] gh-111487: Fix csv.Sniffer never testing the minimum delimiter consistency #154872

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
b517280
gh-111487: Fix csv.Sniffer never testing the minimum delimiter consis…
pikammmmm Jul 29, 2026
7258233
Use a tolerance on the threshold comparison
pikammmmm Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Lib/csv.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -409,7 +409,11 @@ def _guess_delimiter(self, data, delimiters):
consistency = 1.0
# minimum consistency threshold
threshold = 0.9
while len(delims) == 0 and consistency >= threshold:
# Tolerance for the rounding error accumulated by stepping
# 'consistency' down by 0.01: without it the counter stops at
# 0.9099999999999999 and the pass at the threshold never runs.
epsilon = 1e-9
while len(delims) == 0 and consistency >= threshold - epsilon:
for k, v in modeList:
if v[0] > 0 and v[1] > 0:
if ((v[1]/total) >= consistency and

@serhiy-storchakaserhiy-storchakaAug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is other.

@pikammmmmpikammmmmAug 5, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That one is safe without it: every counter value is at or below the percent it stands for, and v[1]/total is correctly rounded, so a score landing exactly on a percent is never below the counter. Checked every m/n up to n=20000 — none rejected. Glad to add it there too if you prefer both to match.

(Noted on length, thanks — keeping it short.)

Expand Down
15 changes: 15 additions & 0 deletions Lib/test/test_csv.py
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1521,6 +1521,21 @@ def test_guess_delimiter_crlf_not_chosen(self):
self.assertEqual(sniffer.sniff(sample).delimiter, "|")
self.assertNotEqual(sniffer.sniff(sample).delimiter, "\r")

def test_guess_delimiter_at_minimum_consistency(self):
# gh-111487: the consistency counter was stepped down by repeatedly
# subtracting 0.01 from a float, so it stopped at 0.9099999999999999
# and the pass at the documented 0.9 threshold never ran. Here ";" is
# modal on 19 of the 20 rows, which scores (19 - 1) / 20 == 90% once
# the non-matching row is subtracted -- exactly the threshold.
# Every row uses a different letter so nothing else is consistent, and
# the odd row sits in the first chunk, which keeps that chunk below the
# threshold and defers the decision to the full 20-row chunk.
rows = [(chr(ord('a') + i) + ';') * 3 + chr(ord('a') + i)
for i in range(20)]
rows[4] = 'e;e;e'
sniffer = csv.Sniffer()
self.assertEqual(sniffer.sniff('\n'.join(rows)).delimiter, ';')

def test_zero_mode_tie_order_independence(self):
sniffer = csv.Sniffer()
# ":" appears in half the rows (1, 0, 1, 0) - a tie between
Expand Down
5 changes: 5 additions & 0 deletions Misc/NEWS.d/next/Library/2026-07-29-13-21-44.gh-issue-111487.Kw9pTz.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
Fix :meth:`csv.Sniffer.sniff` never testing the documented minimum delimiter
consistency of 90%. The consistency counter was stepped down by repeatedly
subtracting ``0.01`` from a float, so it stopped at ``0.9099999999999999`` and
the final pass at the threshold was skipped; the comparison against the
threshold now allows for that accumulated error.
Loading