Add a tested data-quality workflow - #2
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:3e79eba114
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for column in REQUIRED_COLUMNS: | ||
| missing_mask = working[column].astype("string").str.strip().eq("") | ||
| _append_reason(reasons, missing_mask, f"missing_{column}") |
There was a problem hiding this comment.
When callers pass a DataFrame containing pd.NA, None, or NaN, this expression produces an indeterminate Boolean rather than True, so _append_reason skips the missing value. The later numeric and date checks similarly fail to add a reason, allowing records with missing identifiers, modules, dates, or scores into cleaned and potentially omitting them from KPIs. Include working[column].isna() in the required-field mask.
Useful? React with 👍 / 👎.
| for column in NUMERIC_COLUMNS: | ||
| working[column] = pd.to_numeric(working[column], errors="coerce") | ||
| invalid_numeric = working[column].isna() & raw[column].astype("string").str.strip().ne("") | ||
| _append_reason(reasons, invalid_numeric, f"invalid_{column}") |
There was a problem hiding this comment.
For CSV values such as inf or -inf, pd.to_numeric returns an infinity rather than NaN, so this validation does not classify them as invalid. In particular, a row whose score and maximum are both inf passes all range checks, is accepted, and derives a NaN score percentage that contaminates the cleaned output and module KPI. Explicitly require each converted numeric value to be finite.
Useful? React with 👍 / 👎.
| if not input_path.is_file(): | ||
| raise DataQualityError(f"Input file does not exist: {input_path}") | ||
| frame = pd.read_csv(input_path, dtype="string", keep_default_na=False) |
There was a problem hiding this comment.
When an input CSV contains a physically blank line, read_csv skips it by default, so the record is silently absent from both cleaned and rejected output. This also makes input_rows undercount the source and causes every subsequent source_row value to point to the wrong physical CSV line, undermining the workflow's audit trail. Disable blank-line skipping or otherwise retain the original physical line numbers.
Useful? React with 👍 / 👎.
Uh oh!
There was an error while loading. Please reload this page.
Summary
Adds a small, auditable Data/BI preparation workflow that turns an intentionally imperfect synthetic CSV into cleaned records, rejected rows with explicit reason codes, module-level KPIs and a machine-readable quality report.
Workflow
Test coverage
CI integration
The Ubuntu 24.04 and Windows 2025 matrix now compiles and lints the new package, runs the full pytest suite, executes the data-quality CLI, validates the expected 15/8/7 control totals and uploads short-lived generated outputs.
Safety and scope
Validation status
The PR will be merged only after the complete Ubuntu and Windows matrix succeeds on the final head.