MDIO v1 Ingestion

[tasansal]

Update segy_to_mdio for v1 with Custom SEG-Y Schemas

Overview

Enhance MDIO's segy_to_mdio function to support v1 updates, enabling flexible ingestion of SEG-Y files using custom schemas. The function converts an input SEG-Y file to an MDIO output file (local or remote).

In v0, the function used rigid parameters and assumptions. In v1, it leverages arbitrary MDIO structures and flexible data definitions. Ingestion relies on detailed SEG-Y specifications (via TGSAI/segy) mapped to pre-defined or custom MDIO Seismic Templates.

Major Changes

1. Remove explicit parameterization from the function signature; integrate it into the MDIO Seismic Template.
2. Explicitly define or fetch SEG-Y file structures from TGSAI/segy standards.

Questions / Decision Points

• What input types should segy_to_mdio accept for specs?
  • Aim for simplicity: Ideally, just registry names for standard SEG-Y and MDIO templates.
  • SEG-Y files often require customization due to variations in the wild.
  • MDIO templates are preferred for parameterization; SEG-Y specs may need high customization.
  • Support four spec/template formats: (1) registry keys, (2) JSON strings, (3) JSON files, (4) Pydantic models.
  • Registry keys (1) are common for MDIO but rare for SEG-Y; Pydantic (4) suits developers but not general users; JSON options (2/3) are redundant but useful.

Requirements

1. Ingest SEG-Y files to MDIO, parsing user-defined headers per SEG-Y spec.
2. Conform to pre-defined MDIO templates.
3. Allow custom SEG-Y specs for extensibility.
4. Support custom MDIO templates (discouraged).
5. Integrate MDIO and TGSAI/segy registries.
6. Handle read/write for cloud and local storage.

Function Signature

def segy_to_mdio(
 input_uri: str,
 output_uri: str,
 segy_spec: str, # See note (1)
 mdio_template: str, # See note (2)
 grid_overrides: dict | None = None,
 storage_options_input: dict[str, Any] | None = None,
 storage_options_output: dict[str, Any] | None = None,
 overwrite: bool = False,
):
 """A function that converts SEG-Y files to MDIO files."""

Notes:
(1) segy_spec and mdio_template must be a valid name of a registered SegySpec or MdioTemplate.
a. Must be a str that is a key in the SeismicTemplateRegistry (#563) and SegyStandard registry in TGSAI/segy.
b. Custom templates must be registered in #563 before being used here
c. If a template doesn't exist, we should raise

For CLI mdio segy import:
Both should be one of the following, prioritized in this order:
i. A str that is a key in TGSAI/segy standard registry https://github.com/TGSAI/segy/blob/main/src/segy/standards/registry.py AND MDIO template registry. For SEG-Y that this could be a float as well, so we need to check for float conversion of the string first.
ii. A str to a path pointing at a JSON file with a fully qualified spec. Please ensure to use fsspec instead of regular open to support local and remote (cloud files)
iii. If custom template registration is needed, Pydantic Model will be generated on the fly and registered before ingestion is triggered

Usage Examples

Convert a local SEG-Y file using SEG-Y rev 1 standard and a pre-defined MDIO template

from mdio import segy_to_mdio

segy_to_mdio(
 input_uri="path/to/input.segy",
 output_uri="path/to/output.mdio",
 segy_spec="1.0", # SEG-Y rev 1 from TGSAI/segy registry
 mdio_template="PostStack3D", # Pre-defined MDIO template for 3D seismic data
 overwrite=True
)

from mdio import segy_to_mdio

# Path to a JSON file with SEG-Y specification
segy_spec_path = "path/to/segy_spec_rev2.json"

# Content of segy_spec_rev2.json (for reference):
# {
# "revision": 2.0,
# "endian": "little",
# ...
# }

# Convert SEG-Y to MDIO
segy_to_mdio(
 input_uri="/data/seismic/input.segy",
 output_uri="/data/seismic/output.mdio",
 segy_spec=segy_spec_path,
 mdio_template="PreStackCdpGather2D", # Pre-defined MDIO template for 2D CDP gathers
 overwrite=True
)

[dmitriyrepin]

@tasansal , @BrianMichell ,
Here is my understanding of what segy_to_mdio_v1() is supposed to do. Please correct:

Basic Case (no customizations):

1. Parse multiple input options.
2. Validate that the specified type of MDIO template matches the SegySpec. How?
3. Given segy_spec, mdio_template parameters in different formats (see above), convert those to instances of segy.schema.SegySpec and DatasetTemplate.
4. Load the SEG-Y file according to the schema specified by SegySpec
5. Extract the dataset dimensions for the loaded SEG-Y file. Is it OK to use the current code (e.g.,get_grid_plan)?
6. Instantiate MDIO dataset using an appropriate type of the DatasetTemplate and the dimensions
7. Persist the MDIO dataset to an empty Zarr dataset
8. Write traces to the MDIO Zarr dataset
   • Create Grid instance based on MDIO Dataset instance
   • Write traces to MDIO Dataset via blocked_io.to_zarr
   • Write actual stats to the MDIO Zarr dataset (Would this be already written as a part of a dataset?)
9. zarr.consolidate_metadata

[tasansal]

@dmitriyrepin makes sense in general. Some context/comments:

• In general we should use as much v0 code as we can, we don't want to re-write everything.
• Your item 1; i would like to get some of your input on input parameters
• For item 2: the MDIO template has the required keys (dimension names, and coordinates). We need to validate if the SegySpec has these keys because MDIO will try to read from those.
• For item 3: yes, but I believe most of it is trivial because pydantic can parse JSON and we have registries in both TGSAI/segy and MDIO.
• Item 4: I assume by load you mean open? loading comes later.
• Yes the grid gets extracted in Item 5. Of course we should re-use all the code we can.
• Yes to 6 & 7.
• Item 8 uses multiprocessing and its performance critical, we need to ensure there is no regression. we could potentially utilize xarray and dask semantics (e.g. map_blocks and to_zarr). There is an example of it here using ProcessPoolExecutor (current v0 way) + xarray based workers for v1 ingestion. v0 ingestion is ProcessPool + zarr.
• Stats in Item 8 is currently calculated online by workers and then aggregated at the very end in the main process, then its written to MDIO.
• Yes to 9.

[tasansal]

Updated with some notes; please let us know if anything is unclear

[dmitriyrepin]

• old trace worker:
  

  mdio-python/src/mdio/segy/_workers.py

  Line 70 in 12c29a4

  def trace_worker(

• new trace worker:
  

  mdio-python/src/mdio/factory.py

  Line 727 in f8d7607

  def traces_to_zarr( # noqa: PLR0913

• teapot dome test:
  

  mdio-python/tests/integration/test_segy_import_export.py

  Line 269 in 12c29a4

  class TestReader:

• to_zarr
  https://github.com/BrianMichell/mdio-python/blob/v1_ingestion_YOLO/src/mdio/segy/blocked_io.py#L37
  https://github.com/BrianMichell/mdio-python/blob/v1_ingestion_YOLO/src/mdio/segy/_workers.py#L53
  This change involves:
  • src/mdio/segy/_workers.py
  • src/mdio/segy/blocked_io.py
  • src/mdio/core/indexing.py
  • src/mdio/core/grid.py

[tasansal]

resolved