fix: keep all fields and sanitize the CSV header in export_csv

[chiruu12]

Fixes two issues in export_csv.

Dropped columns (#53). The column list came from data[0].keys(), so a key that first appears on a later record was never written and nothing was raised. Scraper output is heterogeneous, optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exported differently depending on format. Columns now come from every record, in first-seen order.

Unsanitized header (#54).#52 added _sanitize_csv_cell for values, but writeheader() wrote the field names unchanged, and those names come from keys in the scraped payload. A key like =HYPERLINK("http://evil","x") still evaluated on open. The header now goes through the same sanitizer, including when fields= is passed explicitly. sanitize=False leaves the header byte-exact as before.

Tests: 5 new cases, 21 passing in the two export_csv files. Each new case fails with the source change reverted and the tests kept.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chiruu12]

Fixes two issues in export_csv.

Dropped columns (#53). The column list came from data[0].keys(), so a key that first appears on a later record was never written and nothing was raised. Scraper output is heterogeneous, optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exported differently depending on format. Columns now come from every record, in first-seen order.

Unsanitized header (#54).#52 added _sanitize_csv_cell for values, but writeheader() wrote the field names unchanged, and those names come from keys in the scraped payload. A key like =HYPERLINK("http://evil","x") still evaluated on open. The header now goes through the same sanitizer, including when fields= is passed explicitly. sanitize=False leaves the header byte-exact as before.

Tests: 5 new cases, 21 passing in the two export_csv files. Each new case fails with the source change reverted and the tests kept.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chiruu12]

Fixes two issues in export_csv.

Dropped columns (#53). The column list came from data[0].keys(), so a key that first appears on a later record was never written and nothing was raised. Scraper output is heterogeneous, optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exported differently depending on format. Columns now come from every record, in first-seen order.

Unsanitized header (#54).#52 added _sanitize_csv_cell for values, but writeheader() wrote the field names unchanged, and those names come from keys in the scraped payload. A key like =HYPERLINK("http://evil","x") still evaluated on open. The header now goes through the same sanitizer, including when fields= is passed explicitly. sanitize=False leaves the header byte-exact as before.

Tests: 5 new cases, 21 passing in the two export_csv files. Each new case fails with the source change reverted and the tests kept.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chiruu12]

Fixes two issues in export_csv.

Dropped columns (#53). The column list came from data[0].keys(), so a key that first appears on a later record was never written and nothing was raised. Scraper output is heterogeneous, optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exported differently depending on format. Columns now come from every record, in first-seen order.

Unsanitized header (#54).#52 added _sanitize_csv_cell for values, but writeheader() wrote the field names unchanged, and those names come from keys in the scraped payload. A key like =HYPERLINK("http://evil","x") still evaluated on open. The header now goes through the same sanitizer, including when fields= is passed explicitly. sanitize=False leaves the header byte-exact as before.

Tests: 5 new cases, 21 passing in the two export_csv files. Each new case fails with the source change reverted and the tests kept.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chiruu12]

Fixes two issues in export_csv.

Dropped columns (#53). The column list came from data[0].keys(), so a key that first appears on a later record was never written and nothing was raised. Scraper output is heterogeneous, optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exported differently depending on format. Columns now come from every record, in first-seen order.

Unsanitized header (#54).#52 added _sanitize_csv_cell for values, but writeheader() wrote the field names unchanged, and those names come from keys in the scraped payload. A key like =HYPERLINK("http://evil","x") still evaluated on open. The header now goes through the same sanitizer, including when fields= is passed explicitly. sanitize=False leaves the header byte-exact as before.

Tests: 5 new cases, 21 passing in the two export_csv files. Each new case fails with the source change reverted and the tests kept.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chiruu12]

Fixes two issues in export_csv.

Dropped columns (#53). The column list came from data[0].keys(), so a key that first appears on a later record was never written and nothing was raised. Scraper output is heterogeneous, optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exported differently depending on format. Columns now come from every record, in first-seen order.

Unsanitized header (#54).#52 added _sanitize_csv_cell for values, but writeheader() wrote the field names unchanged, and those names come from keys in the scraped payload. A key like =HYPERLINK("http://evil","x") still evaluated on open. The header now goes through the same sanitizer, including when fields= is passed explicitly. sanitize=False leaves the header byte-exact as before.

Tests: 5 new cases, 21 passing in the two export_csv files. Each new case fails with the source change reverted and the tests kept.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chiruu12]

Fixes two issues in export_csv.

Dropped columns (#53). The column list came from data[0].keys(), so a key that first appears on a later record was never written and nothing was raised. Scraper output is heterogeneous, optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exported differently depending on format. Columns now come from every record, in first-seen order.

Unsanitized header (#54).#52 added _sanitize_csv_cell for values, but writeheader() wrote the field names unchanged, and those names come from keys in the scraped payload. A key like =HYPERLINK("http://evil","x") still evaluated on open. The header now goes through the same sanitizer, including when fields= is passed explicitly. sanitize=False leaves the header byte-exact as before.

Tests: 5 new cases, 21 passing in the two export_csv files. Each new case fails with the source change reverted and the tests kept.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[chiruu12]

Fixes two issues in export_csv.

Dropped columns (#53). The column list came from data[0].keys(), so a key that first appears on a later record was never written and nothing was raised. Scraper output is heterogeneous, optional fields are absent when a page does not have them, so this loses real data on ordinary output. export_json and export_jsonl keep every key, so the same result set exported differently depending on format. Columns now come from every record, in first-seen order.

Unsanitized header (#54).#52 added _sanitize_csv_cell for values, but writeheader() wrote the field names unchanged, and those names come from keys in the scraped payload. A key like =HYPERLINK("http://evil","x") still evaluated on open. The header now goes through the same sanitizer, including when fields= is passed explicitly. sanitize=False leaves the header byte-exact as before.

Tests: 5 new cases, 21 passing in the two export_csv files. Each new case fails with the source change reverted and the tests kept.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.