feat: add obs4REF and ESMValTool reference dataset types - #805
Conversation
Extract the obs4MIPs-style column block shared by `Obs4MIPsDataset` and `PMPClimatologyDataset` into a `ReferenceDatasetMixin`, then declare two new polymorphic subtypes on top of the `Dataset` hierarchy. `obs4REF` is REF-curated observational data that follows the obs4MIPs metadata conventions but has not been published to the obs4MIPs ESGF archive. It reuses the mixin unchanged. `ESMValToolReference` is ESMValTool's own reference data, which is not CMOR compliant, so it carries a smaller column set of its own. The mixin extraction changes no existing table. `obs4mips_dataset` and `pmp_climatology_dataset` keep exactly the columns they had. The migration is not deferrable. `executor/reingest.py` queries datasets with `with_polymorphic(Dataset, "*")`, which joins every mapped subtype table whether or not rows of that type exist, so declaring a model without its table breaks `reconstruct_execution_definition` on every database. ESMValTool reference data records `frequency` and no MIP table. `native6` data carries a frequency and never a table, while `OBS`/`OBS6` carry a table that reduces to one, so frequency is the only axis both layouts share. Adds `frequency_from_mip_table` to do that reduction (`Amon` -> `mon`). The mapping is enumerated rather than derived from the table name, because `Oclim` -> `monC`, `E1hrClimMon` -> `1hrCM` and the zonal-mean tables do not follow the suffix rule. Neither type can be ingested yet. The adapters, solver wiring and registry changes follow separately.
|
Warning Review limit reached
Next review available in: 58 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Add the adapters that turn the obs4REF and ESMValTool reference dataset types (added in Climate-REF#805) into ingestible source types. - `Obs4REFDatasetAdapter` subclasses the obs4MIPs adapter. obs4REF shares the obs4MIPs metadata conventions, so it reuses the parser and only overrides the accepted activity ids and the instance_id prefix. - `parse_obs4mips` gains an `accepted_activity_ids` hook. A file whose activity_id is outside the adapter's set but is still a known obs4MIPs/obs4REF id is ingested with a warning rather than rejected. The hard-reject cutover for that cross-contamination case is a tracked follow-up. - `ESMValToolReferenceDatasetAdapter` parses ESMValTool's own layout (`OBS`/`OBS6`, `native6` and non-compliant `obs4MIPs` subtrees). It records `frequency`, reducing the `OBS`/`OBS6` MIP table to it via `frequency_from_mip_table`, since `native6` carries a frequency and no table. - Wire obs4REF through the solver and catalog. ESMValTool reference is dispatch-only for now, matching its provenance-first use case.
Description
Adds two new source dataset types,
obs4REFandESMValToolReference, along with their tables. This is schema only. Neither can be ingested yet, so the adapters, solver wiring and registry changes follow separately.Obs4MIPsDatasetandPMPClimatologyDatasetinto aReferenceDatasetMixin. No existing table changes.obs4REFis REF-curated observational data following the obs4MIPs conventions that has not been published to ESGF. It reuses the mixin.ESMValToolReferenceis ESMValTool's own reference data. It is not CMOR compliant, so it has its own smaller column set.ESMValToolReferencerecordsfrequencyand no MIP table, becausenative6never carries one.frequency_from_mip_tablereduces theOBS/OBS6table (Amontomon). It is non-nullable because it is part of the dataset identity.Checklist
Please confirm that this pull request has done the following:
changelog/