Skip to content

Add memory guards to dasymetric.pycnophylactic and disaggregate(method='limiting_variable') #1261

Description

@brendancol

Describe the bug

pycnophylactic() and disaggregate(method='limiting_variable') in xrspatial/dasymetric.py allocate full-shape working arrays without checking available memory first. pycnophylactic() additionally stores one boolean mask per zone, so memory grows with N_zones * H * W. Most other modules already check _available_memory_bytes() before this kind of allocation; dasymetric does not.

Working memory footprint

_pycnophylactic_numpy (lines 464-545) keeps the following alive at peak:

  • four float64 full-shape arrays: surface, smoothed, neighbour_sum, neighbour_count
  • two bool full-shape arrays: valid, has_neighbours
  • a zone_masks dict holding one bool mask per zone in values

For 1000 zones on a 10000x10000 raster, iteration buffers come to about 3.4 GB (34 bytes per pixel) and the per-zone masks alone come to about 100 GB.

_disaggregate_limiting_numpy (lines 251-316) holds three to five full-shape masks plus the result and a pixel_class int32 array, which is roughly 4-5x the input size.

Expected behavior

Both public functions should call _available_memory_bytes() and raise MemoryError before the first allocation when projected working memory exceeds 0.5 * available. The check needs to account for zone count in pycnophylactic since masks scale with len(values).

Reproducer

import numpy as np
import xarray as xr
from xrspatial.dasymetric import pycnophylactic

zones = xr.DataArray(np.random.randint(1, 10001, (10000, 10000)), dims=['y','x'])
values = {i: 100.0 for i in range(1, 10001)}
pycnophylactic(zones, values)  # tries to allocate ~1 TB of zone masks

Additional context

Found during the dasymetric security sweep. Same pattern as the fixes in #1231, #1229, #1240, #1257, #1256, #1236, and #1241.

A separate MEDIUM finding on dtype validation is being deferred to its own PR per the security-sweep policy of one fix per PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinginput-validationInput validation and error messagesoomOut-of-memory risk with large datasets

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions