Add verified reporting and notebook reconciliation

[DataTideHH]

Summary

Adds a compact reporting layer on top of the tested data-quality workflow. Persisted module KPIs are recalculated from cleaned records before any reporting output is accepted.

Reporting workflow

• loads the four generated data-quality outputs
• validates required files and columns
• recalculates module KPIs from cleaned records
• fails on persisted KPI differences
• explodes pipe-separated rejection reasons into deterministic counts
• calculates overall reporting control totals
• generates average-score and pass-rate SVG charts with Matplotlib
• exports a machine-readable reporting summary

Notebook verification

• adds a clean notebooks/reporting_verification.ipynb
• reads generated data-quality outputs rather than embedding data
• displays verified module KPIs and rejection reasons
• renders both charts
• asserts the expected 4 modules, 8 accepted results, 7 rejected rows, 70.00% overall average and 62.50% pass rate
• retains no committed output, execution count, local path or IDE timestamp

Portfolio evidence

• adds two compact SVG reference charts for the committed synthetic fixture
• documents the complete validation-to-reporting flow
• updates the README with control totals, commands, charts and scope boundaries

Automated verification

• tests expected reporting totals and output files
• tests complete rejection-reason counts
• rejects a deliberately modified KPI value
• verifies missing-input failure behaviour
• verifies generated SVG labels and values
• extends notebook hygiene tests to both notebooks

CI

The Python 3.12 matrix on Ubuntu 24.04 and Windows 2025 now runs:

1. compilation, Ruff and pytest
2. baseline entry point
3. complete data-quality workflow and 15/8/7 controls
4. reporting workflow and KPI reconciliation
5. reporting control totals and SVG checks
6. optional ML example
7. both clean notebooks
8. short-lived verified output artefacts

Scope

This remains a learning-grade Data/BI workflow. It does not claim a production semantic model, Power BI report, scheduled orchestration, interactive dashboard or statistical inference from the synthetic fixture.

The PR will be merged only after the complete final matrix succeeds.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9e6b655918

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Resolve CI paths from the repository root

In the newly added Execute reporting verification notebook step in .github/workflows/python-quality.yml, nbconvert executes the kernel with the input notebook's parent directory as its working directory. Consequently, this path resolves to notebooks/.ci-output/data-quality, while the preceding workflow creates .ci-output/data-quality at the repository root, so both CI matrix jobs stop here with a missing-input ReportingError. Resolve the paths relative to the repository root or explicitly configure the notebook execution directory.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Handle reporting runs with zero accepted rows

When a valid data-quality run rejects every input row (or processes a header-only file), the upstream workflow deliberately emits an empty cleaned dataset and stable empty KPI schema, but passed.mean() returns pd.NA and converting it to float raises TypeError; the score mean also becomes NaN, which would produce non-standard JSON. Thus reporting cannot consume a valid upstream result. Define explicit null or zero semantics for both overall metrics, or raise a controlled ReportingError for this case.

Useful? React with 👍 / 👎.