Skip to content

flow_length_dinf: no memory guard on H*W working arrays #1355

Description

@brendancol

flow_length_dinf in xrspatial/hydro/flow_length_dinf.py allocates several full-grid working arrays in its eager numpy and cupy backends with no upfront budget check. A 50000x50000 raster asks for about 72 GB of host RAM before anything errors out.

Same shape as #1332 for flow_length_d8 and #1351 for flow_length_mfd, and the broader hydro audit started with #1318/#1319 for flow_accumulation_d8. Eager backends check, dask backends skip since per-tile allocations are bounded by chunk size.

Allocations

_flow_length_dinf_downstream_cpu and _flow_length_dinf_upstream_cpu (called by both numpy and cupy backends) allocate:

Array dtype bytes/px
in_degree int32 4
valid int8 1
flow_len float64 8
order_r int64 8
order_c int64 8
Total 29

The D-inf flow direction array is 2-D (angles in radians), matching the d8 budget. The numpy public dispatch also runs fd = data.astype(np.float64) before the kernel, but that 8 B/px copy is dwarfed by the 29 B/px kernel locals.

For the cupy backend, _flow_length_dinf_cupy calls .get().astype(np.float64) then runs the CPU kernel and copies the output back via cp.asarray. Device-side residency at peak is the input float64 (8 B/px) plus the output float64 (8 B/px); host-side matches the 29 B/px CPU budget. Use 32 B/px as a conservative GPU budget covering both copies plus headroom.

Worked example

50000x50000 = 2.5e9 pixels. CPU peak working set:

2.5e9 * 29 = 72.5 GB

Allocated before any sanity check runs.

Fix

Mirror PR #1332: _check_memory and _check_gpu_memory helpers, wired into the numpy and cupy dispatch in flow_length_dinf(). Leave dask alone. Add tests for oversize rejection, valid pass-through, dask bypass, and error message content.

One fix per PR per the security-sweep policy.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghigh-priorityoomOut-of-memory risk with large datasets

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions