csv.Sniffer().sniff() Hangs on Single-Column Data

[jackdesert]

Using single-column quoted csv file of this format:

"one-one"
"two-two"
"three-three"
...

If the file is small (say 3,000 rows), it generates an error: _csv.Error: Could not determine delimiter
But if the file is large (say 30,000 rows) the process appears to hang.

Here is a code sample that reproduces this behavior:

import csv
from uuid import uuid4

items = []

items = [f'"{uuid4()}"' for _ in range(30_000)]
print('1. Created uuids')

csv_data = '\n'.join(items)
print('2. Joined')

with open('output', 'w') as h:
 h.write(csv_data)
print('3. Wrote output')

sniffer = csv.Sniffer()

# This is the line than hangs
dialect = sniffer.sniff(csv_data, delimiters=',:|\t')
print(f'4. Sniffed delimiter "{dialiect.delimiter}"')

Linked PRs

• [3.15] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields #154867
• [3.14] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) #155117
• [3.13] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) #155118
• [3.12] gh-98820: Fix quadratic time in csv.Sniffer for quoted fields (GH-154867) #155166

[Tickloop]

Can I try and tackle this one?

[Tickloop]

Here are my thoughts after a dive into the implementation of csv module:

1. Error produced:

   The error is expected as the module can't find a delimiter as there isn't one in csv_data. The code is telling the sniffer to find if the rows in csv_data use one of ',:|\t' as a delimiter, which isn't the case (csv_data does not have a delimiter as there is only one column).

   Now if the data were correctly formatted i.e. if one of the delimiters were present on even one of the lines, then the module is able to detect it as a delimiter.

2. Hanging on sniffer.sniff():

   The sniffer's implementation to find the quotechar and delimiter in the data uses regex matching.

   If a delimiter isn't found then a probabilistic analysis happens which goes through chunks of data to determine a suitable delimiter.

   If the data isn't consistent and doesn't have a delimiter, then the regex function has n^2 run time which is causing the process to hang in case of 30_000 lines of input.

[serhiy-storchaka]

Fixed in 3.16 by the csv.Sniffer rewrite (gh-83273): a sample which is a single column of quoted fields is now recognized as such and csv.Error is raised immediately, instead of parsing quadratically slower with every added row.

The maintenance branches are still affected: 3000 rows already take 2 seconds there, and the time grows quadratically. A targeted fix for them is welcome.