Schema decision: the canonical dtype vocabulary for schema.columns

[mmcky]

Every schema.columns entry declares a dtype, and the invariant tests in PLAN Phase 5 are meant to check it. Across the 41 manifests the vocabulary today is:

value	occurrences
float64	126
float32	57
int64	49
str	43
string	21
object	9
datetime64	2
datetime	2

Three names for text (str, string, object) and two for dates, so no validator can compare a declared dtype with what pandas reads without a mapping. Business_cycle's manifest says str where manifest-schema.yml's example says string.

To decide.

1. The canonical set: pandas dtype strings (float64, int64, string, datetime64[ns], bool, category) is the obvious candidate since validate() will compare against a pandas frame; whether float32 stays as a distinct declaration or collapses to float.
2. What the check means for a CSV: a CSV has no dtypes, so the declaration is a statement about what pd.read_csv with the manifest's read options yields. Whether the manifest records those read options (index_col, parse_dates) or the check uses pandas defaults.
3. Text columns: string versus object, given the lectures read with default pandas settings that yield object.
4. Migration: a one-time normalisation pass over the manifests once the set is fixed, and a manifest-schema.yml conformance check that rejects anything outside it.

Sub-issue of #14.

[mmcky]

Decision, 2026-09-07: pandas-3 names, compared by family; str is the text dtype

Re-measured rather than reusing the table in the body. The lectures pin anaconda=2026.07, which is pandas 3; lecture-datascience.myst is still on anaconda=2024.06 (pandas 2), and Pyodide in lecture-wasm is a pandas-2 reader too.

Eight spellings in use. float64 126, float32 57, int64 49, str 43, string 21, object 9, datetime64 2, datetime 2. All 57 float32 are in the four Stata .dta manifests where float32 is the real storage type — not drift.

What breaks when a validator compares declared against read. Every CSV read with pandas defaults, each of the 212 named columns compared with str(dtype):

	pandas 3.0.5	pandas 2.3.3
exact matches	169	142
text spelling mismatches	19 (string/object vs str)	47 (str/string vs object)
date columns read as text	4	4
other	1	0
not found by a default read	19	19

The 43 manifests already on str are right under the lectures' pin; string and object are the drift. The 19 not-found columns are all countries.csv, which is semicolon-delimited and whose manifest already records delimiter: ";" — so the repo already has a read-options precedent (delimiter, and read_as on the xlsx sheets) and the validator only has to honour it. The four date columns can never match without parse_dates. The one "other" is a real declaration error: forbes-billionaires.csv's government is declared str but is a boolean column with 2606 nulls in 2935 rows, which pandas reads as object.

The rules.

1. The canonical vocabulary is the pandas-3 str(dtype) of a default read: float64, int64, str, bool, datetime64. Storage types that non-CSV formats genuinely carry (float32, int32, int8 from .dta) stay legal. string, object and datetime are retired.
2. The validator compares by family, not exact string: str/string/object are one text family, and datetime64 matches any unit (pandas 3 gives datetime64[us], pandas 2 [ns]). The check therefore stays green on pandas-2 readers without a second vocabulary.
3. Read options: the validator honours the existing delimiter and read_as keys and derives parse_dates from any column declared datetime64, so a date declaration is both the contract and the instruction for reading it. No new field.
4. One normalisation sweep: 21 string and 9 object → str; 2 datetime → datetime64; forbes government → bool with its nulls declared per Schema decision: is known_nulls an exact count or a ceiling? #121.
5. manifest-schema.yml carries the closed list and the conformance check rejects anything outside it.

Closes with the sweep PR, which can precede #119.