Skip to content

hand_dinf: no memory guard on H*W working arrays #1344

Description

@brendancol

hand_dinf in xrspatial/hydro/hand_dinf.py allocates several full-grid working arrays in its eager numpy and cupy backends with no upfront budget check. A 50000x50000 raster asks for about 95 GB of host RAM before anything errors out.

Same shape as #1323 (hand_d8, fixed in #1326) and #1338 (hand_mfd, fixed in #1340). The broader hydro audit started with #1318/#1319 for flow_accumulation_d8. Eager backends check, dask backends skip since per-tile allocations are bounded by chunk size.

Allocations

_hand_dinf_cpu (called by both numpy and cupy backends) allocates:

Array dtype bytes/px
in_degree int32 4
valid int8 1
is_stream int8 1
drain_elev float64 8
hand_out float64 8
order_r int64 8
order_c int64 8
Total 38

The D-inf flow_dir input is a single (H, W) float64 grid — 8 B/px — but it is caller-provided and lives in RAM before hand_dinf() runs, so it isn't double-counted.

_hand_dinf_cupy calls .get() on the three inputs then runs _hand_dinf_cpu, so it pays 38 B/px on the host plus the input copies. A conservative 32 B/px applies to the device-side budget for symmetry with #1326 and #1340.

Worked example

50000x50000 = 2.5e9 pixels. CPU peak working set:

2.5e9 * 38 = 95 GB

Allocated before any sanity check runs.

Fix

Mirror PR #1326 / #1340: _check_memory and _check_gpu_memory helpers, wired into the numpy and cupy dispatch in hand_dinf(). Leave dask alone. Add tests for oversize rejection, valid pass-through, dask bypass, and error message content.

One fix per PR per the security-sweep policy.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghigh-priorityoomOut-of-memory risk with large datasets

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions