Skip to content

focal apply()/focal_stats()/hotspots() accept unbounded user kernels #1284

Description

@brendancol

What is the bug?

focal.apply(), focal.focal_stats(), and focal.hotspots() accept any 2-D NumPy array as the kernel argument. The validator (convolution.custom_kernel()) only checks ndarray-ness and odd row/column counts. There is no cap on kernel shape.

The kernel-size memory guard added in #1241 (_check_kernel_memory) only runs inside circle_kernel / annulus_kernel. A caller who builds the kernel directly with NumPy bypasses it, so a tiny raster with a giant kernel allocates many gigabytes:

import numpy as np
import xarray as xr
from xrspatial.focal import apply

raster = xr.DataArray(np.zeros((10, 10), dtype=np.float32))
kernel = np.ones((50001, 50001), dtype=np.float32)  # ~10 GB on its own
apply(raster, kernel)

Downstream paths driven by kernel.shape:

  • _apply_numpy (focal.py:385) allocates kernel_values = np.zeros_like(kernel, dtype=data.dtype) once per call.
  • _apply_numpy_boundary (focal.py:404-412) and _convolve_2d_numpy_boundary (convolution.py:374-382) call _pad_array(data, (kernel.shape[0]//2, kernel.shape[1]//2)) for any non-NaN boundary, which allocates (H + 2*pad_h, W + 2*pad_w).
  • The dask paths (_apply_dask_numpy, _apply_dask_cupy, _focal_stats_dask_cupy) set the map_overlap depth to kernel.shape // 2 per chunk.
  • hotspots() calls convolve_2d(data, kernel/kernel.sum(), boundary), which inherits the same padded allocation.

Same shape as the bilateral fix in #1236, where sigma_spatial drove an unbounded radius. That one was patched by clamping the derived radius to max(rows, cols) of the raster before dispatch.

How to reproduce

  1. Build a small DataArray, for example xr.DataArray(np.zeros((10, 10), dtype=np.float32)).
  2. Build a kernel with a half-extent larger than the raster, for example np.ones((50001, 50001), dtype=np.float32) (~10 GB on its own; padded allocation is much bigger).
  3. Call focal.apply(raster, kernel) (or focal_stats(...) / hotspots(...)). The process allocates gigabytes before doing any useful work, which can OOM the host.

Anything else we need to know?

Other categories in focal were clean during this audit:

  • All ten CUDA kernels in focal.py have if i >= rows or j >= cols: return plus if 0 <= ii < rows and 0 <= jj < cols: stencil checks before reads.
  • _validate_raster() is called on mean, apply, focal_stats, and hotspots.
  • hotspots already raises ZeroDivisionError when global_std == 0.
  • _focal_variety_cuda uses a fixed-size local buffer (MAX_UNIQ = 25) and silently truncates rather than overflowing.
  • _focal_std_cuda and _focal_var_cuda use the E[x^2] - E[x]^2 form, which can go slightly negative under cancellation, but both clamp if var < 0.0: var = 0.0 so the failure mode is precision loss, not a NaN out of sqrt.

Fix: clamp the kernel half-extent in apply(), focal_stats(), and hotspots() so that pad_h <= rows and pad_w <= cols of the raster. Either reject with a clear error, or trim the kernel to the raster extent before dispatch. Add a regression test that passes a small raster with a deliberately oversize kernel and asserts the call raises instead of allocating.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinghigh-priorityinput-validationInput validation and error messagesoomOut-of-memory risk with large datasets

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions