diff --git a/docs/src/userguide/cube_statistics.rst b/docs/src/userguide/cube_statistics.rst index 4eb016078e..d62a056f33 100644 --- a/docs/src/userguide/cube_statistics.rst +++ b/docs/src/userguide/cube_statistics.rst @@ -23,9 +23,9 @@ Collapsing Entire Data Dimensions In the :doc:`subsetting_a_cube` section we saw how to extract a subset of a cube in order to reduce either its dimensionality or its resolution. -Instead of simply extracting a sub-region of the data, -we can produce statistical functions of the data values -across a particular dimension, +Instead of simply extracting a sub-region of the data, +we can produce statistical functions of the data values +across a particular dimension, such as a 'mean over time' or 'minimum over latitude'. .. _cube-statistics_forecast_printout: @@ -57,9 +57,9 @@ For instance, suppose we have a cube: um_version: 7.3 -In this case we have a 4 dimensional cube; -to mean the vertical (z) dimension down to a single valued extent -we can pass the coordinate name and the aggregation definition to the +In this case we have a 4 dimensional cube; +to mean the vertical (z) dimension down to a single valued extent +we can pass the coordinate name and the aggregation definition to the :meth:`Cube.collapsed() ` method: >>> import iris.analysis @@ -88,8 +88,8 @@ we can pass the coordinate name and the aggregation definition to the mean: model_level_number -Similarly other analysis operators such as ``MAX``, ``MIN`` and ``STD_DEV`` -can be used instead of ``MEAN``, see :mod:`iris.analysis` for a full list +Similarly other analysis operators such as ``MAX``, ``MIN`` and ``STD_DEV`` +can be used instead of ``MEAN``, see :mod:`iris.analysis` for a full list of currently supported operators. For an example of using this functionality, the @@ -103,14 +103,14 @@ in the gallery takes a zonal mean of an ``XYT`` cube by using the Area Averaging ^^^^^^^^^^^^^^ -Some operators support additional keywords to the ``cube.collapsed`` method. -For example, :func:`iris.analysis.MEAN ` supports -a weights keyword which can be combined with +Some operators support additional keywords to the ``cube.collapsed`` method. +For example, :func:`iris.analysis.MEAN ` supports +a weights keyword which can be combined with :func:`iris.analysis.cartography.area_weights` to calculate an area average. -Let's use the same data as was loaded in the previous example. -Since ``grid_latitude`` and ``grid_longitude`` were both point coordinates -we must guess bound positions for them +Let's use the same data as was loaded in the previous example. +Since ``grid_latitude`` and ``grid_longitude`` were both point coordinates +we must guess bound positions for them in order to calculate the area of the grid boxes:: import iris.analysis.cartography @@ -155,24 +155,24 @@ including an example on taking a :ref:`global area-weighted mean Partially Reducing Data Dimensions ---------------------------------- -Instead of completely collapsing a dimension, other methods can be applied -to reduce or filter the number of data points of a particular dimension. +Instead of completely collapsing a dimension, other methods can be applied +to reduce or filter the number of data points of a particular dimension. Aggregation of Grouped Data ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The :meth:`Cube.aggregated_by ` operation -combines data for all points with the same value of a given coordinate. -To do this, you need a coordinate whose points take on only a limited set -of different values -- the *number* of these then determines the size of the +The :meth:`Cube.aggregated_by ` operation +combines data for all points with the same value of a given coordinate. +To do this, you need a coordinate whose points take on only a limited set +of different values -- the *number* of these then determines the size of the reduced dimension. -The :mod:`iris.coord_categorisation` module can be used to make such -'categorical' coordinates out of ordinary ones: The most common use is -to aggregate data over regular *time intervals*, +The :mod:`iris.coord_categorisation` module can be used to make such +'categorical' coordinates out of ordinary ones: The most common use is +to aggregate data over regular *time intervals*, such as by calendar month or day of the week. -For example, let's create two new coordinates on the cube +For example, let's create two new coordinates on the cube to represent the climatological seasons and the season year respectively:: import iris @@ -188,8 +188,8 @@ to represent the climatological seasons and the season year respectively:: .. note:: - The 'season year' is not the same as year number, because (e.g.) the months - Dec11, Jan12 + Feb12 all belong to 'DJF-12'. + The 'season year' is not the same as year number, because (e.g.) the months + Dec11, Jan12 + Feb12 all belong to 'DJF-12'. See :meth:`iris.coord_categorisation.add_season_year`. @@ -206,10 +206,10 @@ to represent the climatological seasons and the season year respectively:: iris.coord_categorisation.add_season_year(cube, 'time', name='season_year') annual_seasonal_mean = cube.aggregated_by( - ['clim_season', 'season_year'], + ['clim_season', 'season_year'], iris.analysis.MEAN) - + Printing this cube now shows that two extra coordinates exist on the cube: .. doctest:: aggregation @@ -238,20 +238,20 @@ These two coordinates can now be used to aggregate by season and climate-year: .. doctest:: aggregation >>> annual_seasonal_mean = cube.aggregated_by( - ... ['clim_season', 'season_year'], + ... ['clim_season', 'season_year'], ... iris.analysis.MEAN) >>> print(repr(annual_seasonal_mean)) - -The primary change in the cube is that the cube's data has been -reduced in the 'time' dimension by aggregation (taking means, in this case). -This has collected together all data points with the same values of season and + +The primary change in the cube is that the cube's data has been +reduced in the 'time' dimension by aggregation (taking means, in this case). +This has collected together all data points with the same values of season and season-year. The results are now indexed by the 19 different possible values of season and season-year in a new, reduced 'time' dimension. -We can see this by printing the first 10 values of season+year -from the original cube: These points are individual months, +We can see this by printing the first 10 values of season+year +from the original cube: These points are individual months, so adjacent ones are often in the same season: .. doctest:: aggregation @@ -271,7 +271,7 @@ so adjacent ones are often in the same season: djf 2007 djf 2007 -Compare this with the first 10 values of the new cube's coordinates: +Compare this with the first 10 values of the new cube's coordinates: All the points now have distinct season+year values: .. doctest:: aggregation @@ -294,7 +294,7 @@ All the points now have distinct season+year values: Because the original data started in April 2006 we have some incomplete seasons (e.g. there were only two months worth of data for 'mam-2006'). -In this case we can fix this by removing all of the resultant 'times' which +In this case we can fix this by removing all of the resultant 'times' which do not cover a three month period (note: judged here as > 3*28 days): .. doctest:: aggregation @@ -306,7 +306,7 @@ do not cover a three month period (note: judged here as > 3*28 days): >>> full_season_means -The final result now represents the seasonal mean temperature for 17 seasons +The final result now represents the seasonal mean temperature for 17 seasons from jja-2006 to jja-2010: .. doctest:: aggregation diff --git a/lib/iris/common/metadata.py b/lib/iris/common/metadata.py index 40dccf9428..e81c6b206c 100644 --- a/lib/iris/common/metadata.py +++ b/lib/iris/common/metadata.py @@ -43,6 +43,9 @@ # https://www.unidata.ucar.edu/software/netcdf/docs/netcdf_data_set_components.html#object_name + +from ..util import guess_coord_axis + _TOKEN_PARSE = re.compile(r"""^[a-zA-Z0-9][\w\.\+\-@]*$""") # Configure the logger. @@ -1339,6 +1342,136 @@ def equal(self, other, lenient=None): return super().equal(other, lenient=lenient) +def metadata_filter( + instances, + item=None, + standard_name=None, + long_name=None, + var_name=None, + attributes=None, + axis=None, +): + """ + Filter a collection of objects by their metadata to fit the given metadata + criteria. Criteria be one or both of: specific properties / other objects + carrying metadata to be matched. + + Args: + + * instances + An iterable of objects to be filtered. + + Kwargs: + + * item + Either + + (a) a :attr:`standard_name`, :attr:`long_name`, or + :attr:`var_name`. Defaults to value of `default` + (which itself defaults to `unknown`) as defined in + :class:`~iris.common.CFVariableMixin`. + + (b) a 'coordinate' instance with metadata equal to that of + the desired coordinates. Accepts either a + :class:`~iris.coords.DimCoord`, :class:`~iris.coords.AuxCoord`, + :class:`~iris.aux_factory.AuxCoordFactory`, + :class:`~iris.common.CoordMetadata` or + :class:`~iris.common.DimCoordMetadata` or + :class:`~iris.experimental.ugrid.ConnectivityMetadata`. + * standard_name + The CF standard name of the desired coordinate. If None, does not + check for standard name. + * long_name + An unconstrained description of the coordinate. If None, does not + check for long_name. + * var_name + The netCDF variable name of the desired coordinate. If None, does + not check for var_name. + * attributes + A dictionary of attributes desired on the coordinates. If None, + does not check for attributes. + * axis + The desired coordinate axis, see + :func:`~iris.util.guess_coord_axis`. If None, does not check for + axis. Accepts the values 'X', 'Y', 'Z' and 'T' (case-insensitive). + + Returns: + A list of the objects supplied in the ``instances`` argument, limited + to only those that matched the given criteria. + + """ + name = None + obj = None + + if isinstance(item, str): + name = item + else: + obj = item + + result = instances + + if name is not None: + result = [instance for instance in result if instance.name() == name] + + if standard_name is not None: + result = [ + instance + for instance in result + if instance.standard_name == standard_name + ] + + if long_name is not None: + result = [ + instance for instance in result if instance.long_name == long_name + ] + + if var_name is not None: + result = [ + instance for instance in result if instance.var_name == var_name + ] + + if attributes is not None: + if not isinstance(attributes, Mapping): + msg = ( + "The attributes keyword was expecting a dictionary " + "type, but got a %s instead." % type(attributes) + ) + raise ValueError(msg) + + def attr_filter(instance): + return all( + k in instance.attributes + and hexdigest(instance.attributes[k]) == hexdigest(v) + for k, v in attributes.items() + ) + + result = [instance for instance in result if attr_filter(instance)] + + if axis is not None: + axis = axis.upper() + result = [ + instance + for instance in result + if guess_coord_axis(instance) == axis + ] + + if obj is not None: + if hasattr(obj, "__class__") and issubclass( + obj.__class__, BaseMetadata + ): + target_metadata = obj + else: + target_metadata = obj.metadata + + result = [ + instance + for instance in result + if instance.metadata == target_metadata + ] + + return result + + def metadata_manager_factory(cls, **kwargs): """ A class instance factory function responsible for manufacturing diff --git a/lib/iris/cube.py b/lib/iris/cube.py index a15951900b..e8b6d4a692 100644 --- a/lib/iris/cube.py +++ b/lib/iris/cube.py @@ -13,7 +13,6 @@ from collections.abc import ( Iterable, Container, - Mapping, MutableMapping, Iterator, ) @@ -40,11 +39,10 @@ import iris.aux_factory from iris.common import ( CFVariableMixin, - CoordMetadata, CubeMetadata, - DimCoordMetadata, metadata_manager_factory, ) +from iris.common.metadata import metadata_filter import iris.coord_systems import iris.coords import iris.exceptions @@ -1639,14 +1637,6 @@ def coords( See also :meth:`Cube.coord()`. """ - name = None - coord = None - - if isinstance(name_or_coord, str): - name = name_or_coord - else: - coord = name_or_coord - coords_and_factories = [] if dim_coords in [True, None]: @@ -1656,62 +1646,15 @@ def coords( coords_and_factories += list(self.aux_coords) coords_and_factories += list(self.aux_factories) - if name is not None: - coords_and_factories = [ - coord_ - for coord_ in coords_and_factories - if coord_.name() == name - ] - - if standard_name is not None: - coords_and_factories = [ - coord_ - for coord_ in coords_and_factories - if coord_.standard_name == standard_name - ] - - if long_name is not None: - coords_and_factories = [ - coord_ - for coord_ in coords_and_factories - if coord_.long_name == long_name - ] - - if var_name is not None: - coords_and_factories = [ - coord_ - for coord_ in coords_and_factories - if coord_.var_name == var_name - ] - - if axis is not None: - axis = axis.upper() - guess_axis = iris.util.guess_coord_axis - coords_and_factories = [ - coord_ - for coord_ in coords_and_factories - if guess_axis(coord_) == axis - ] - - if attributes is not None: - if not isinstance(attributes, Mapping): - msg = ( - "The attributes keyword was expecting a dictionary " - "type, but got a %s instead." % type(attributes) - ) - raise ValueError(msg) - - def attr_filter(coord_): - return all( - k in coord_.attributes and coord_.attributes[k] == v - for k, v in attributes.items() - ) - - coords_and_factories = [ - coord_ - for coord_ in coords_and_factories - if attr_filter(coord_) - ] + coords_and_factories = metadata_filter( + coords_and_factories, + item=name_or_coord, + standard_name=standard_name, + long_name=long_name, + var_name=var_name, + attributes=attributes, + axis=axis, + ) if coord_system is not None: coords_and_factories = [ @@ -1720,20 +1663,6 @@ def attr_filter(coord_): if coord_.coord_system == coord_system ] - if coord is not None: - if hasattr(coord, "__class__") and coord.__class__ in ( - CoordMetadata, - DimCoordMetadata, - ): - target_metadata = coord - else: - target_metadata = coord.metadata - coords_and_factories = [ - coord_ - for coord_ in coords_and_factories - if coord_.metadata == target_metadata - ] - if contains_dimension is not None: coords_and_factories = [ coord_ diff --git a/lib/iris/exceptions.py b/lib/iris/exceptions.py index 1c05d13163..12d24ef70f 100644 --- a/lib/iris/exceptions.py +++ b/lib/iris/exceptions.py @@ -39,6 +39,12 @@ class AncillaryVariableNotFoundError(KeyError): pass +class ConnectivityNotFoundError(KeyError): + """Raised when a search yields no connectivities.""" + + pass + + class CoordinateMultiDimError(ValueError): """Raised when a routine doesn't support multi-dimensional coordinates.""" diff --git a/lib/iris/experimental/ugrid.py b/lib/iris/experimental/ugrid.py index 002c40952f..45c94dbf16 100644 --- a/lib/iris/experimental/ugrid.py +++ b/lib/iris/experimental/ugrid.py @@ -10,7 +10,8 @@ """ -from collections import Mapping, namedtuple +from abc import ABC, abstractmethod +from collections import namedtuple from functools import wraps import dask.array as da @@ -18,28 +19,28 @@ from .. import _lazy_data as _lazy from ..common.metadata import ( - _hexdigest, BaseMetadata, - CoordMetadata, - DimCoordMetadata, metadata_manager_factory, SERVICES, SERVICES_COMBINE, SERVICES_EQUAL, SERVICES_DIFFERENCE, + metadata_filter, ) from ..common.lenient import _lenient_service as lenient_service from ..common.mixin import CFVariableMixin from ..config import get_logger from ..coords import _DimensionalMetadata, AuxCoord -from ..exceptions import CoordinateNotFoundError +from ..exceptions import CoordinateNotFoundError, ConnectivityNotFoundError from ..util import guess_coord_axis __all__ = [ "Connectivity", "ConnectivityMetadata", + "Mesh1DConnectivities", "Mesh1DCoords", + "Mesh2DConnectivities", "Mesh2DCoords", "MeshEdgeCoords", "MeshFaceCoords", @@ -70,6 +71,20 @@ MeshEdgeCoords = namedtuple("MeshEdgeCoords", ["edge_x", "edge_y"]) MeshFaceCoords = namedtuple("MeshFaceCoords", ["face_x", "face_y"]) +# Mesh connectivity manager namedtuples. +Mesh1DConnectivities = namedtuple("Mesh1DConnectivities", ["edge_node"]) +Mesh2DConnectivities = namedtuple( + "Mesh2DConnectivities", + [ + "face_node", + "edge_node", + "face_edge", + "face_face", + "edge_face", + "boundary_node", + ], +) + class Connectivity(_DimensionalMetadata): """ @@ -823,7 +838,7 @@ def __init__( attributes=None, edge_coords_and_axes=None, face_coords_and_axes=None, - # connectivities=None, + connectivities=None, node_dimension=None, edge_dimension=None, face_dimension=None, @@ -883,15 +898,18 @@ def normalise(location, axis): if self.topology_dimension == 1: self._coord_manager = _Mesh1DCoordinateManager(**kwargs) + self._connectivity_manager = _Mesh1DConnectivityManager( + *connectivities + ) elif self.topology_dimension == 2: self._coord_manager = _Mesh2DCoordinateManager(**kwargs) + self._connectivity_manager = _Mesh2DConnectivityManager( + *connectivities + ) else: emsg = f"Unsupported 'topology_dimension', got {topology_dimension!r}." raise NotImplementedError(emsg) - # based on the topology_dimension, create the appropriate connectivity manager - # self._connectivity_manager = ... - def __eq__(self, other): # TBD return NotImplemented @@ -918,20 +936,54 @@ def __str__(self): args = [] return f"{self.__class__.__name__}({', '.join(args)})" + def _set_dimension_names(self, node, edge, face, reset=False): + args = (node, edge, face) + currents = ( + self.node_dimension, + self.edge_dimension, + self.face_dimension, + ) + zipped = zip(args, currents) + if reset: + node, edge, face = [ + None if arg else current for arg, current in zipped + ] + else: + node, edge, face = [arg or current for arg, current in zipped] + + self.node_dimension = node + self.edge_dimension = edge + self.face_dimension = face + + if self.topology_dimension == 1: + result = Mesh1DNames(self.node_dimension, self.edge_dimension) + elif self.topology_dimension == 2: + result = Mesh2DNames( + self.node_dimension, self.edge_dimension, self.face_dimension + ) + else: + message = ( + f"Unsupported topology_dimension: {self.topology_dimension} ." + ) + raise NotImplementedError(message) + + return result + @property def all_coords(self): return self._coord_manager.all_members @property def edge_dimension(self): - return self._edge_dimension + return self._metadata_manager.edge_dimension @edge_dimension.setter def edge_dimension(self, name): if not name or not isinstance(name, str): - self._edge_dimension = f"Mesh{self.topology_dimension}d_edge" + edge_dimension = f"Mesh{self.topology_dimension}d_edge" else: - self._edge_dimension = name + edge_dimension = name + self._metadata_manager.edge_dimension = edge_dimension @property def edge_coords(self): @@ -939,14 +991,15 @@ def edge_coords(self): @property def face_dimension(self): - return self._face_dimension + return self._metadata_manager.face_dimension @face_dimension.setter def face_dimension(self, name): if not name or not isinstance(name, str): - self._face_dimension = f"Mesh{self.topology_dimension}d_face" + face_dimension = f"Mesh{self.topology_dimension}d_face" else: - self._face_dimension = name + face_dimension = name + self._metadata_manager.face_dimension = face_dimension @property def face_coords(self): @@ -954,55 +1007,53 @@ def face_coords(self): @property def node_dimension(self): - return self._node_dimension + return self._metadata_manager.node_dimension @node_dimension.setter def node_dimension(self, name): if not name or not isinstance(name, str): - self._node_dimension = f"Mesh{self.topology_dimension}d_node" + node_dimension = f"Mesh{self.topology_dimension}d_node" else: - self._node_dimension = name + node_dimension = name + self._metadata_manager.node_dimension = node_dimension @property def node_coords(self): return self._coord_manager.node_coords - # @property - # def all_connectivities(self): - # # return a namedtuple - # # conns = mesh.all_connectivities - # # conns.edge_node, conns.boundary_node - # pass - # - # @property - # def face_node_connectivity(self): - # # required - # return self._connectivity_manager.face_node - # - # @property - # def edge_node_connectivity(self): - # # optionally required - # return self._connectivity_manager.edge_node - # - # @property - # def face_edge_connectivity(self): - # # optional - # return self._connectivity_manager.face_edge - # - # @property - # def face_face_connectivity(self): - # # optional - # return self._connectivity_manager.face_face - # - # @property - # def edge_face_connectivity(self): - # # optional - # return self._connectivity_manager.edge_face - # - # @property - # def boundary_node_connectivity(self): - # # optional - # return self._connectivity_manager.boundary_node + @property + def all_connectivities(self): + return self._connectivity_manager.all_members + + @property + def face_node_connectivity(self): + # required + return self._connectivity_manager.face_node + + @property + def edge_node_connectivity(self): + # optionally required + return self._connectivity_manager.edge_node + + @property + def face_edge_connectivity(self): + # optional + return self._connectivity_manager.face_edge + + @property + def face_face_connectivity(self): + # optional + return self._connectivity_manager.face_face + + @property + def edge_face_connectivity(self): + # optional + return self._connectivity_manager.edge_face + + @property + def boundary_node_connectivity(self): + # optional + return self._connectivity_manager.boundary_node def add_coords( self, @@ -1022,25 +1073,56 @@ def add_coords( face_y=face_y, ) - # def add_connectivities(self, *args): - # # this supports adding a new connectivity to the manager, but also replacing an existing connectivity - # self._connectivity_manager.add(*args) - - # def connectivities( - # self, - # name_or_coord=None, - # standard_name=None, - # long_name=None, - # var_name=None, - # attributes=None, - # node=False, - # edge=False, - # face=False, - # ): - # pass - - # def connectivity(self, ...): - # pass + def add_connectivities(self, *connectivities): + self._connectivity_manager.add(*connectivities) + + def connectivities( + self, + item=None, + standard_name=None, + long_name=None, + var_name=None, + attributes=None, + cf_role=None, + node=None, + edge=None, + face=None, + ): + return self._connectivity_manager.filters( + item=item, + standard_name=standard_name, + long_name=long_name, + var_name=var_name, + attributes=attributes, + cf_role=cf_role, + node=node, + edge=edge, + face=face, + ) + + def connectivity( + self, + item=None, + standard_name=None, + long_name=None, + var_name=None, + attributes=None, + cf_role=None, + node=None, + edge=None, + face=None, + ): + return self._connectivity_manager.filter( + item=item, + standard_name=standard_name, + long_name=long_name, + var_name=var_name, + attributes=attributes, + cf_role=cf_role, + node=node, + edge=edge, + face=face, + ) def coord( self, @@ -1090,9 +1172,29 @@ def coords( face=face, ) - # def remove_connectivities(self, ...): - # # needs to respect the minimum UGRID contract - # self._connectivity_manager.remove(...) + def remove_connectivities( + self, + item=None, + standard_name=None, + long_name=None, + var_name=None, + attributes=None, + cf_role=None, + node=None, + edge=None, + face=None, + ): + return self._connectivity_manager.remove( + item=item, + standard_name=standard_name, + long_name=long_name, + var_name=var_name, + attributes=attributes, + cf_role=cf_role, + node=node, + edge=edge, + face=face, + ) def remove_coords( self, @@ -1106,7 +1208,7 @@ def remove_coords( edge=None, face=None, ): - self._coord_manager.remove( + return self._coord_manager.remove( item=item, standard_name=standard_name, long_name=long_name, @@ -1138,42 +1240,18 @@ def xml_element(self): # def to_MeshCoord(self, location, axis): # # factory method # # return MeshCoord(..., location=location, axis=axis) - # # use Connectivity.indices_by_src() for fetching indices. + # # use Connectivity.indices_by_src() for fetching indices, passing in the lazy_indices() result as an argument. # # def to_MeshCoords(self, location): # # factory method # # return MeshCoord(..., location=location, axis="x"), MeshCoord(..., location=location, axis="y") - # # use Connectivity.indices_by_src() for fetching indices. + # # use Connectivity.indices_by_src for fetching indices, passing in the lazy_indices() result as an argument. def dimension_names_reset(self, node=False, edge=False, face=False): - if node: - self.node_dimension = None - if edge: - self.edge_dimension = None - if face: - self.face_dimension = None - if self.topology_dimension == 1: - result = Mesh1DNames(self.node_dimension, self.edge_dimension) - else: - result = Mesh2DNames( - self.node_dimension, self.edge_dimension, self.face_dimension - ) - return result + return self._set_dimension_names(node, edge, face, reset=True) def dimension_names(self, node=None, edge=None, face=None): - if node: - self.node_dimension = node - if edge: - self.edge_dimension = edge - if face: - self.face_dimension = face - if self.topology_dimension == 1: - result = Mesh1DNames(self.node_dimension, self.edge_dimension) - else: - result = Mesh2DNames( - self.node_dimension, self.edge_dimension, self.node_dimension - ) - return result + return self._set_dimension_names(node, edge, face, reset=False) @property def cf_role(self): @@ -1248,88 +1326,6 @@ def __str__(self): ] return f"{self.__class__.__name__}({', '.join(args)})" - @staticmethod - def _filters( - members, - item=None, - standard_name=None, - long_name=None, - var_name=None, - attributes=None, - axis=None, - ): - """ - TDB: support coord_systems? - - """ - name = None - coord = None - - if isinstance(item, str): - name = item - else: - coord = item - - if name is not None: - members = {k: v for k, v in members.items() if v.name() == name} - - if standard_name is not None: - members = { - k: v - for k, v in members.items() - if v.standard_name == standard_name - } - - if long_name is not None: - members = { - k: v for k, v in members.items() if v.long_name == long_name - } - - if var_name is not None: - members = { - k: v for k, v in members.items() if v.var_name == var_name - } - - if axis is not None: - axis = axis.upper() - members = { - k: v for k, v in members.items() if guess_coord_axis(v) == axis - } - - if attributes is not None: - if not isinstance(attributes, Mapping): - emsg = ( - "The attributes keyword was expecting a dictionary " - f"type, but got a {type(attributes)} instead." - ) - raise ValueError(emsg) - - def _filter(coord): - return all( - k in coord.attributes - and _hexdigest(coord.attributes[k]) == _hexdigest(v) - for k, v in attributes.items() - ) - - members = {k: v for k, v in members.items() if _filter(v)} - - if coord is not None: - if hasattr(coord, "__class__") and coord.__class__ in ( - CoordMetadata, - DimCoordMetadata, - ): - target_metadata = coord - else: - target_metadata = coord.metadata - - members = { - k: v - for k, v in members.items() - if v.metadata == target_metadata - } - - return members - def _remove(self, **kwargs): result = {} members = self.filters(**kwargs) @@ -1481,6 +1477,7 @@ def add(self, node_x=None, node_y=None, edge_x=None, edge_y=None): self._add(MeshEdgeCoords(edge_x, edge_y)) def filter(self, **kwargs): + # TODO: rationalise commonality with MeshConnectivityManager.filter and Cube.coord. result = self.filters(**kwargs) if len(result) > 1: @@ -1525,6 +1522,8 @@ def filters( edge=None, face=None, ): + # TBD: support coord_systems? + # rationalise the tri-state behaviour args = [node, edge, face] state = not any(set(filter(lambda arg: arg is not None, args))) @@ -1532,28 +1531,22 @@ def filters( lambda arg: arg if arg is not None else state, args ) - def func(args): - return args[1] is not None + def populated_coords(coords_tuple): + return list(filter(None, list(coords_tuple))) - members = {} + members = [] if node: - members.update( - dict(filter(func, self.node_coords._asdict().items())) - ) + members += populated_coords(self.node_coords) if edge: - members.update( - dict(filter(func, self.edge_coords._asdict().items())) - ) + members += populated_coords(self.edge_coords) if hasattr(self, "face_coords"): if face: - members.update( - dict(filter(func, self.face_coords._asdict().items())) - ) + members += populated_coords(self.face_coords) else: dmsg = "Ignoring request to filter non-existent 'face_coords'" logger.debug(dmsg, extra=dict(cls=self.__class__.__name__)) - result = self._filters( + result = metadata_filter( members, item=item, standard_name=standard_name, @@ -1563,7 +1556,12 @@ def func(args): axis=axis, ) - return result + # Use the results to filter the _members dict for returning. + result_ids = [id(r) for r in result] + result_dict = { + k: v for k, v in self._members.items() if id(v) in result_ids + } + return result_dict def remove( self, @@ -1680,102 +1678,308 @@ def remove( ) -# # keep an eye on the __init__ inheritance -# class _Mesh1DConnectivityManager: -# REQUIRED = ( -# "edge_node", -# ) -# OPTIONAL = () -# def __init__(self, edge_node): -# # required -# self.edge_node = edge_node -# -# # WOO-GA - this can easily get out of sync with the self attributes. -# # choose the container wisely e.g., could be an dict..., also the self -# # attributes may need to be @property's that access the chosen _members container -# -# # is this a list? as dict? a namedtuple? use case is self.add() -# self._members = [] -# -# if self.edge_node is not None: -# self._members.append(self.edge_node) -# -# def __iter__(self): -# for member in self._members: -# yield member -# -# def __getstate__(self): -# pass -# -# def __setstate__(self, state): -# pass -# -# def connectivity(self, **kwargs): -# # see Cube.coord for pattern, checking for a single result -# return self.connectivities(**kwargs)[0] -# -# def connectivities(self, ...): -# # see Cube.coords for relevant patterns -# # return [ ... ] -# pass -# -# def add(self, *args): -# # loop thru args and add (clobber) -# # adopt same philosophy as remove for adding connectivites with unsupported cf-role -# pass -# -# def remove(self, ...): -# # needs to respect the minimum UGRID contract -# # use logging/warning to flag items not removed - highlight in doc-string -# # don't raise an exception -# -# def __str__(self): -# pass -# -# def __repr__(self): -# pass -# -# def __eq__(self, other): -# # Full equality could be MASSIVE, so we want to avoid that. -# # Ideally we want a mesh signature from LFRic for comparison, although this would -# # limit Iris' relevance outside MO. -# # TL;DR: unknown quantity. -# raise NotImplemented -# -# def __ne__(self, other): -# # See __eq__ -# raise NotImplemented -# -# -# class _Mesh2DConnectivityManager(_Mesh1DConnectivityManager): -# REQUIRED = ( -# "face_node", -# ) -# OPTIONAL = ( -# "edge_node", -# "face_edge", -# "face_face", -# "edge_face", -# "boundary_node", -# ) -# def __init__(self, face_node, edge_node=None, face_edge=None, face_face=None, edge_face=None, boundary_node=None): -# # required -# self.face_node = face_node -# self._members = [self.face_node] -# -# # optionally required -# self.edge_node = edge_node -# # optional -# self.face_edge = face_edge -# self.face_face = face_face -# self.edge_face = edge_face -# self.boundary_node = boundary_node -# -# # edge_node could be None here. are we okay with this pattern? -# super().__init__(edge_node) -# -# # does order matter? -# self._members.extend([member for member in self.OPTIONAL if member is not None and member != "edge_node"]) +class _MeshConnectivityManagerBase(ABC): + # Override these in subclasses. + REQUIRED: tuple = NotImplemented + OPTIONAL: tuple = NotImplemented + + def __init__(self, *connectivities): + cf_roles = [c.cf_role for c in connectivities] + for requisite in self.REQUIRED: + if requisite not in cf_roles: + message = ( + f"{self.__name__} requires a {requisite} Connectivity." + ) + raise ValueError(message) + + self.ALL = self.REQUIRED + self.OPTIONAL + self._members = {member: None for member in self.ALL} + self.add(*connectivities) + + def __eq__(self, other): + # TBD + return NotImplemented + + def __getstate__(self): + # TBD + pass + + def __iter__(self): + for item in self._members.items(): + yield item + + def __ne__(self, other): + # TBD + return NotImplemented + + def __repr__(self): + args = [ + f"{member}={connectivity!r}" + for member, connectivity in self + if connectivity is not None + ] + return f"{self.__class__.__name__}({', '.join(args)})" + + def __setstate__(self, state): + # TBD + pass + + def __str__(self): + args = [ + f"{member}=True" + for member, connectivity in self + if connectivity is not None + ] + return f"{self.__class__.__name__}({', '.join(args)})" + + @property + @abstractmethod + def all_members(self): + return NotImplemented + + def add(self, *connectivities): + # Since Connectivity classes include their cf_role, no setters will be + # provided, just a means to add one or more connectivities to the + # manager. + # No warning is raised for duplicate cf_roles - user is trusted to + # validate their outputs. + add_dict = {} + for connectivity in connectivities: + if not isinstance(connectivity, Connectivity): + message = f"Expected Connectivity, got: {type(connectivity)} ." + raise ValueError(message) + cf_role = connectivity.cf_role + if cf_role not in self.ALL: + message = ( + f"Not adding connectivity ({cf_role}: " + f"{connectivity!r}) - cf_role must be one of: {self.ALL} ." + ) + logger.debug(message, extra=dict(cls=self.__class__.__name__)) + else: + add_dict[cf_role] = connectivity + + # Validate shapes. + proposed_members = {**self._members, **add_dict} + locations = set( + [ + c.src_location + for c in proposed_members.values() + if c is not None + ] + ) + for location in locations: + counts = [ + len(c.indices_by_src(c.lazy_indices())) + for c in proposed_members.values() + if c is not None and c.src_location == location + ] + # Check is list values are identical. + if not counts.count(counts[0]) == len(counts): + message = ( + f"Invalid Connectivities provided - inconsistent " + f"{location} counts." + ) + raise ValueError(message) + + self._members = proposed_members + + def filter(self, **kwargs): + # TODO: rationalise commonality with MeshCoordManager.filter and Cube.coord. + result = self.filters(**kwargs) + if len(result) > 1: + names = ", ".join( + f"{member}={connectivity!r}" + for member, connectivity in result.items() + ) + message = ( + f"Expected to find exactly 1 connectivity, but found " + f"{len(result)}. They were: {names}." + ) + raise ConnectivityNotFoundError(message) + elif len(result) == 0: + item = kwargs["item"] + _name = item + if item is not None: + if not isinstance(item, str): + _name = item.name() + bad_name = ( + _name or kwargs["standard_name"] or kwargs["long_name"] or "" + ) + message = ( + f"Expected to find exactly 1 {bad_name} connectivity, " + f"but found none." + ) + raise ConnectivityNotFoundError(message) + + return result + + def filters( + self, + item=None, + standard_name=None, + long_name=None, + var_name=None, + attributes=None, + cf_role=None, + node=None, + edge=None, + face=None, + ): + members = [c for c in self._members.values() if c is not None] + + if cf_role is not None: + members = [ + instance for instance in members if instance.cf_role == cf_role + ] + + def location_filter(instances, loc_arg, loc_name): + if loc_arg is False: + filtered = [ + instance + for instance in instances + if loc_name + not in (instance.src_location, instance.tgt_location) + ] + elif loc_arg is None: + filtered = instances + else: + # Interpret any other value as =True. + filtered = [ + instance + for instance in instances + if loc_name + in (instance.src_location, instance.tgt_location) + ] + + return filtered + + for arg, loc in ( + (node, "node"), + (edge, "edge"), + (face, "face"), + ): + members = location_filter(members, arg, loc) + + # No need to actually modify filtering behaviour - already won't return + # any face cf-roles if none are present. + supports_faces = any(["face" in role for role in self.ALL]) + if face and not supports_faces: + message = ( + "Ignoring request to filter for non-existent 'face' cf-roles." + ) + logger.debug(message, extra=dict(cls=self.__class__.__name__)) + + result = metadata_filter( + members, + item=item, + standard_name=standard_name, + long_name=long_name, + var_name=var_name, + attributes=attributes, + ) + + # Use the results to filter the _members dict for returning. + result_ids = [id(r) for r in result] + result_dict = { + k: v for k, v in self._members.items() if id(v) in result_ids + } + return result_dict + + def remove( + self, + item=None, + standard_name=None, + long_name=None, + var_name=None, + attributes=None, + cf_role=None, + node=None, + edge=None, + face=None, + ): + removal_dict = self.filters( + item=item, + standard_name=standard_name, + long_name=long_name, + var_name=var_name, + attributes=attributes, + cf_role=cf_role, + node=node, + edge=edge, + face=face, + ) + for cf_role in self.REQUIRED: + excluded = removal_dict.pop(cf_role, None) + if excluded: + message = ( + f"Ignoring request to remove required connectivity " + f"({cf_role}: {excluded!r})" + ) + logger.debug(message, extra=dict(cls=self.__class__.__name__)) + + for cf_role in removal_dict.keys(): + self._members[cf_role] = None + + return removal_dict + + +class _Mesh1DConnectivityManager(_MeshConnectivityManagerBase): + REQUIRED = ("edge_node_connectivity",) + OPTIONAL = () + + @property + def all_members(self): + return Mesh1DConnectivities(edge_node=self.edge_node) + + @property + def edge_node(self): + return self._members["edge_node_connectivity"] + + +class _Mesh2DConnectivityManager(_MeshConnectivityManagerBase): + REQUIRED = ("face_node_connectivity",) + OPTIONAL = ( + "edge_node_connectivity", + "face_edge_connectivity", + "face_face_connectivity", + "edge_face_connectivity", + "boundary_node_connectivity", + ) + + @property + def all_members(self): + return Mesh2DConnectivities( + face_node=self.face_node, + edge_node=self.edge_node, + face_edge=self.face_edge, + face_face=self.face_face, + edge_face=self.edge_face, + boundary_node=self.boundary_node, + ) + + @property + def boundary_node(self): + return self._members["boundary_node_connectivity"] + + @property + def edge_face(self): + return self._members["edge_face_connectivity"] + + @property + def edge_node(self): + return self._members["edge_node_connectivity"] + + @property + def face_edge(self): + return self._members["face_edge_connectivity"] + + @property + def face_face(self): + return self._members["face_face_connectivity"] + + @property + def face_node(self): + return self._members["face_node_connectivity"] #: Convenience collection of lenient metadata combine services. diff --git a/lib/iris/tests/unit/common/metadata/test_metadata_filter.py b/lib/iris/tests/unit/common/metadata/test_metadata_filter.py new file mode 100644 index 0000000000..dafb50554b --- /dev/null +++ b/lib/iris/tests/unit/common/metadata/test_metadata_filter.py @@ -0,0 +1,118 @@ +# Copyright Iris contributors +# +# This file is part of Iris and is released under the LGPL license. +# See COPYING and COPYING.LESSER in the root of the repository for full +# licensing details. +""" +Unit tests for the :func:`iris.common.metadata_filter`. + +""" + +# Import iris.tests first so that some things can be initialised before +# importing anything else. +import iris.tests as tests + +import numpy as np + +from iris.common.metadata import ( + CoordMetadata, + DimCoordMetadata, + metadata_filter, +) +from iris.coords import AuxCoord + +Mock = tests.mock.Mock + + +class Test_standard(tests.IrisTest): + def test_name(self): + name_one = Mock() + name_one.name.return_value = "one" + name_two = Mock() + name_two.name.return_value = "two" + input_list = [name_one, name_two] + result = metadata_filter(input_list, item="one") + self.assertIn(name_one, result) + self.assertNotIn(name_two, result) + + def test_item(self): + coord = Mock(__class__=AuxCoord) + mock = Mock() + input_list = [coord, mock] + result = metadata_filter(input_list, item=coord) + self.assertIn(coord, result) + self.assertNotIn(mock, result) + + def test_item_metadata(self): + coord = Mock(metadata=CoordMetadata) + dim_coord = Mock(metadata=DimCoordMetadata) + input_list = [coord, dim_coord] + result = metadata_filter(input_list, item=coord) + self.assertIn(coord, result) + self.assertNotIn(dim_coord, result) + + def test_standard_name(self): + name_one = Mock(standard_name="one") + name_two = Mock(standard_name="two") + input_list = [name_one, name_two] + result = metadata_filter(input_list, standard_name="one") + self.assertIn(name_one, result) + self.assertNotIn(name_two, result) + + def test_long_name(self): + name_one = Mock(long_name="one") + name_two = Mock(long_name="two") + input_list = [name_one, name_two] + result = metadata_filter(input_list, long_name="one") + self.assertIn(name_one, result) + self.assertNotIn(name_two, result) + + def test_var_name(self): + name_one = Mock(var_name="one") + name_two = Mock(var_name="two") + input_list = [name_one, name_two] + result = metadata_filter(input_list, var_name="one") + self.assertIn(name_one, result) + self.assertNotIn(name_two, result) + + def test_attributes(self): + # Confirm that this can handle attrib dicts including np arrays. + attrib_one_two = Mock( + attributes={"one": np.arange(1), "two": np.arange(2)} + ) + attrib_three_four = Mock( + attributes={"three": np.arange(3), "four": np.arange(4)} + ) + input_list = [attrib_one_two, attrib_three_four] + result = metadata_filter( + input_list, attributes=attrib_one_two.attributes + ) + self.assertIn(attrib_one_two, result) + self.assertNotIn(attrib_three_four, result) + + def test_invalid_attributes(self): + attrib_one = Mock(attributes={"one": 1}) + input_list = [attrib_one] + self.assertRaisesRegex( + ValueError, + ".*expecting a dictionary.*", + metadata_filter, + input_list, + attributes="one", + ) + + def test_axis(self): + axis_lon = Mock(standard_name="longitude") + axis_lat = Mock(standard_name="latitude") + input_list = [axis_lon, axis_lat] + result = metadata_filter(input_list, axis="x") + self.assertIn(axis_lon, result) + self.assertNotIn(axis_lat, result) + + def test_multiple_args(self): + coord_one = Mock(__class__=AuxCoord, long_name="one") + coord_two = Mock(__class__=AuxCoord, long_name="two") + input_list = [coord_one, coord_two] + result = metadata_filter(input_list, item=coord_one, long_name="one") + self.assertIn(coord_one, result) + self.assertNotIn(coord_two, result) diff --git a/lib/iris/tests/unit/experimental/ugrid/test_MeshMetadata.py b/lib/iris/tests/unit/experimental/ugrid/test_MeshMetadata.py index cfc668fb88..105365c908 100644 --- a/lib/iris/tests/unit/experimental/ugrid/test_MeshMetadata.py +++ b/lib/iris/tests/unit/experimental/ugrid/test_MeshMetadata.py @@ -398,10 +398,10 @@ def test_op_lenient_same_members_none(self): with mock.patch( "iris.common.metadata._LENIENT", return_value=True ): - self.assertTrue( + self.assertEqual( expected, lmetadata.combine(rmetadata)._asdict() ) - self.assertTrue( + self.assertEqual( expected, rmetadata.combine(lmetadata)._asdict() )