Skip to content

Extend lazy MFD dask assembly to stream/hand modules and study the convergence phase #2885

Description

@brendancol

Follow-up to #2865, which made the dask output assembly lazy in flow_accumulation_mfd, flow_length_mfd, and watershed_mfd.

Remaining eager paths

The same eager-assembly pattern (.compute() per tile, then da.from_array / da.block) lives in other MFD dask paths that #2865 left untouched to keep its diff scoped and reduce conflicts with parallel hydro work:

  • xrspatial/hydro/stream_link_mfd.py, _stream_link_mfd_dask
  • xrspatial/hydro/stream_order_mfd.py
  • xrspatial/hydro/hand_mfd.py

These can take the same da.map_blocks treatment. They have extra secondary inputs (accumulation, stream rasters) that need the same chunk-alignment handling used for pour points in watershed_mfd.

Convergence phase

The iterative boundary-convergence sweep that runs before assembly still reads tile data eagerly. That is inherent to the algorithm: converging cross-tile boundaries needs real values across sweeps. Making it lazy would need a different approach (for example a fixed number of map_overlap rounds, or a delayed reduction), which is a separate design question worth its own investigation. Filing it here so it is tracked rather than lost.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestoomOut-of-memory risk with large datasets

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions