fix(tables): sniff CSV delimiter and capture the real header row on import by waleedlatif1 · Pull Request #5888 · simstudioai/sim

waleedlatif1 · 2026-07-23T15:37:18Z

Summary

CSV table import derived the delimiter from the file extension only, so semicolon-delimited exports (European-locale Excel) parsed as a single column — the reported "alarms-export" bug. Now the delimiter is sniffed from the file head (,;\t|), quote-aware, with the extension only as a fallback.
Header derivation used Object.keys(rows[0]); with relax_column_count a ragged/sparse first data row dropped its trailing columns from schema inference and mapping. The true header row is now captured via the csv-parsecolumns callback on every path — fixes the "sparse csv didn't work" report.
New detectCsvDelimiter (trial-parse each candidate, pick most columns, tie-break on row-width consistency) mirrors PapaParse/csv.Sniffer, and sniffCsvDelimiterFromStream (64KB peek-then-replay) keeps the streaming paths memory-bounded on multi-GB files.
Wired into all consumers: create route, append/replace route, background runner, client dialog preview, and parseFileRows (copilot).

Type of Change

Bug fix

Testing

442 table + route tests pass (49 new/updated covering delimiter detection, ragged headers, stream replay, error/destroy/empty edge cases)
Verified against the real 61 MB alarms-export file: comma → 1 column, semicolon → 34 columns / 1249 records; sniffer detects ;
Typecheck clean, biome clean, check:api-validation passes

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

vercel · 2026-07-23T15:37:26Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jul 23, 2026 5:58pm

cursor · 2026-07-23T15:37:34Z

PR Summary

Medium Risk
Changes core import parsing used on every CSV upload (sync, async, UI preview, copilot); wrong sniffing could mis-map columns, but behavior is heavily tested and scoped to parsing—not auth or persistence logic.

Overview
Table CSV import no longer trusts the file extension for the field separator. Imports now sniff,, ;, tab, and | from the first 64KB (quote-aware trial parse), with extension/TSV only as fallback—fixing semicolon Excel exports collapsing into one column.

Headers are taken from the parser’s columns callback (with duplicate-name deduping) instead of Object.keys(firstRow), so ragged first data rows and duplicate headers no longer drop or invent columns during schema inference and mapping.

New sniffCsvDelimiterFromStream replays the peeked prefix for streaming routes (sync import, create-from-CSV, background runner) without buffering whole files. The import dialog preview uses the same detectCsvDelimiter logic so UI mapping matches server behavior; parseFileRows (copilot) is updated too.

Extensive tests cover delimiter edge cases, header derivation, and stream replay/destroy behavior.

^{Reviewed by Cursor Bugbot for commit 35156cc. Configure here.}

greptile-apps · 2026-07-23T15:40:38Z

Greptile Summary

Adds consistent, quote-aware CSV delimiter detection and authoritative header capture across table-import paths.

Sniffs comma, semicolon, tab, and pipe delimiters from a bounded file prefix.
Replays streamed input without copying oversized chunks.
Uses the parser’s header callback for schema inference and column mapping.
Aligns synchronous routes, background imports, client previews, and copilot file parsing.
Adds regression coverage for delimiter scoring, ragged headers, stream replay, cleanup, and boundary conditions.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failures remain in the fixes related to the previous review threads.

Important Files Changed

Filename	Overview
apps/sim/lib/table/import.ts	Adds shared delimiter detection, header deduplication, parser header capture, and buffered-import integration.
apps/sim/lib/table/csv-delimiter-stream.ts	Adds bounded stream sniffing with byte-exact chunk replay and source cleanup.
apps/sim/lib/table/import-runner.ts	Applies sniffed delimiters and captured headers to background table imports.
apps/sim/app/api/table/import-csv/route.ts	Uses delimiter sniffing and authoritative headers when creating tables from uploaded CSV files.
apps/sim/app/api/table/[tableId]/import/route.ts	Uses delimiter sniffing and full parsed headers for append and replace imports.
apps/sim/app/workspace/[workspaceId]/tables/components/import-csv-dialog/import-csv-dialog.tsx	Aligns client-side import previews with the server’s delimiter-sniffing window and fallback.
apps/sim/lib/table/import.test.ts	Adds regression tests covering delimiter ranking, ragged headers, bounded replay, errors, cleanup, and sniff-window boundaries.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[CSV upload or stored file] --> B[Read bounded file prefix]
B --> C[Trial-parse delimiter candidates]
C --> D[Select delimiter by width and consistency]
D --> E[Replay complete stream]
E --> F[csv-parse header callback]
F --> G[Schema inference and row import]

_{Reviews (8): Last reviewed commit: "test(tables): use sleep helper instead o..." | Re-trigger Greptile}

waleedlatif1 · 2026-07-23T15:49:29Z

@greptile

waleedlatif1 · 2026-07-23T15:49:31Z

@cursor review

…mport Table CSV import derived the delimiter purely from the file extension, so semicolon-delimited exports (European-locale Excel) parsed as a single column. Header derivation also read Object.keys(rows[0]), so with relax_column_count a ragged/sparse first data row dropped its trailing columns from schema inference. - Add detectCsvDelimiter: trial-parse the head against ,/;/tab/| and pick the candidate with the most columns (tie-break on row-width consistency), quote-aware so separators inside quoted cells don't skew the guess; extension is now only the fallback - Add sniffCsvDelimiterFromStream: memory-bounded (64KB) peek-then-replay for the streaming import paths so multi-GB files sniff without buffering - Capture the true header row via the csv-parse columns callback on every path, fixing dropped columns when the first row is short - Wire into the create route, append/replace route, background runner, client dialog preview, and parseFileRows (copilot)

… buffer Addresses review findings on the delimiter sniffer: - Rank candidates by row-width consistency first (modal column count), then column count. Ranking on column count alone let a comma that appears in a semicolon file's unquoted header win over the real separator; the wide-but-ragged split now loses to the uniform one. - Move the partial-trailing-line trim into detectCsvDelimiter so every path (stream, client preview, parseFileRows) prepares the sniff sample identically and can't disagree on the delimiter. - Cap the stream peeker's detection copy at the sniff window via Buffer.concat's length arg and replay the buffered chunks by reference, so an oversized source chunk can't break the bounded-memory contract.

waleedlatif1 · 2026-07-23T15:50:51Z

@greptile

waleedlatif1 · 2026-07-23T15:50:52Z

@cursor review

Addresses round-2 review findings: - Score each delimiter candidate by modalWidth * consistency instead of consistency alone. Consistency-only let a separator that appears uniformly inside values (e.g. a pipe once per row) beat a real delimiter whose rows are legitimately ragged; the product rewards a split that is both wide and uniform, with width breaking ties. Genuinely symmetric files (two valid columns under either separator) fall back to the global-default candidate order. - Dedupe the captured header row to match the parser's record keys. csv-parse collapses duplicate column names onto one key (last value wins), so reporting the raw duplicates made schema inference invent a phantom empty column and could fail mapping validation. dedupeHeaders keeps first-occurrence order, case-sensitively, matching the emitted keys.

waleedlatif1 · 2026-07-23T16:02:24Z

@greptile

waleedlatif1 · 2026-07-23T16:02:26Z

@cursor review

detectCsvDelimiter always trimmed to the last newline, so a complete file with no trailing newline lost its final data row from scoring — leaving the header alone and letting a header-widening separator beat the real one. It now takes a `complete` flag: the trim runs only for truncated prefixes, where a mid-record cut must be dropped. The stream sniffer passes it from `exhausted`, and the buffered callers (client preview, parseFileRows) pass it when the sample covers the whole file.

waleedlatif1 · 2026-07-23T17:38:40Z

@greptile

waleedlatif1 · 2026-07-23T17:38:41Z

@cursor review

Hoist the csv-parse options out of the multi-line parse() call so the `// double-cast-allowed` annotation sits directly above the `as unknown as` token — the strict boundary audit only matches an annotation within three lines of the cast. Also simplify the stream sniffer's completeness check to `exhausted` (the loop only ends early on end-of-stream, so it already means the whole file fit the sniff window).

The stream sniffer's read loop stopped as soon as the buffered size reached the sniff window, so a file whose size is exactly the window never triggered the extra read that observes end-of-stream — it was judged a truncated prefix and dropped its final newline-less row, disagreeing with the buffered callers (which mark that size complete). Read while size <= window so the boundary case sees EOF and `exhausted` (hence `complete`) is set correctly. Regression test added.

waleedlatif1 · 2026-07-23T17:54:38Z

@greptile

waleedlatif1 · 2026-07-23T17:54:39Z

@cursor review

waleedlatif1 · 2026-07-23T17:59:20Z

@greptile

waleedlatif1 · 2026-07-23T17:59:22Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 35156cc. Configure here.}

waleedlatif1 · 2026-07-23T18:11:48Z

@greptile

waleedlatif1 · 2026-07-23T18:11:50Z

@cursor review

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 35156cc. Configure here.}

greptile-appsBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/lib/table/import.ts Outdated
Comment threadapps/sim/lib/table/csv-delimiter-stream.ts Outdated

cursorBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/app/workspace/[workspaceId]/tables/components/import-csv-dialog/import-csv-dialog.tsx

vercelBottemporarily deployed to Preview July 23, 2026 15:49 Inactive

waleedlatif1 added 2 commits July 23, 2026 08:50

waleedlatif1 force-pushed the worktree-csv-import-sparse branch from a5d29bc to a6cc15cCompare July 23, 2026 15:50

vercelBottemporarily deployed to Preview July 23, 2026 15:50 Inactive

greptile-appsBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/lib/table/import.ts Outdated

greptile-appsBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/lib/table/import.ts Outdated

cursorBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/lib/table/import.ts

vercelBottemporarily deployed to Preview July 23, 2026 16:01 Inactive

cursorBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/lib/table/import.ts Outdated

greptile-appsBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/lib/table/import.ts

vercelBottemporarily deployed to Preview July 23, 2026 17:38 Inactive

cursorBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/lib/table/csv-delimiter-stream.ts Outdated

vercelBottemporarily deployed to Preview July 23, 2026 17:51 Inactive

vercelBottemporarily deployed to Preview July 23, 2026 17:54 Inactive

test(tables): use sleep helper instead of raw setTimeout promise
35156cc

vercelBottemporarily deployed to Preview July 23, 2026 17:58 Inactive

cursorBot reviewed Jul 23, 2026
View reviewed changes

greptile-appsBot reviewed Jul 23, 2026
View reviewed changes

Comment threadapps/sim/lib/table/import.ts

cursorBot reviewed Jul 23, 2026
View reviewed changes

waleedlatif1 merged commit efe1de3 into stagingJul 23, 2026
20 checks passed

waleedlatif1 deleted the worktree-csv-import-sparse branch July 23, 2026 18:18

waleedlatif1 mentioned this pull request Jul 23, 2026
v0.7.44: nextjs upgrade, CSV imports to tables fixes, tombstone for KB docs, mcp fixes #5903
Merged

Uh oh!

Conversation

waleedlatif1 commented Jul 23, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercelBot commented Jul 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursorBot commented Jul 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

greptile-appsBot commented Jul 23, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

cursorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

waleedlatif1 commented Jul 23, 2026

Uh oh!

cursorBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercelBot commented Jul 23, 2026•
edited
Loading

cursorBot commented Jul 23, 2026•
edited
Loading

greptile-appsBot commented Jul 23, 2026•
edited
Loading