Skip opening data sets when climatologies (and time series) are already cached

[xylar]

Presently, we create an xarray data set before caching climatologies in many MPAS-Analysis tasks. If the climatology is already fully cached, opening the xarray data set is a potentially time consuming, unnecessary step.

Presumably, the only information needed to determine whether the cached data is complete is the start and end time of the cached data and the start and end time of the data set to be cached. Reading the full data set (even in xarray, where only a minimum of information is read in from each file in the data set) is potentially a major waste of time.

It would be good to have a method for determining which data is already cached and to ready only those data files that are needed to compute uncached data. If all data is cached, none of the original output files would be read at all.

[xylar]

This came up in discussion of #181. @milenaveneziani feels that it will likely be a high priority, particularly at higher resolution.

[xylar]

I have given this some thought. I think it will be a lot easier to address after we have converted analysis tasks to classes in #128. Once analysis tasks are classes, we can make a base class for climatology tasks. That base class will have the functionality for creating and caching climatologies that is currently in the climatology module. "Modelvsobs" and similar tasks can derive from this base class (meaning, for those not familiar with object-oriented programming that they "inherit" all its capabilities) and they can implement a function for loading the necessary data set, given a start and end year. When caching climatologies, we could then call this function to load only the data needed to compute each cached climatology, not the full data set. If all climatologies are already cached, the function would never be called and only the cached data would be loaded, not the original data.

The climatology task class would still need a way to find out what the first and last date available would be. This would presumably require having a list of available files that can be used for computing climatologies (which is something we already create), loading the Time variable from the first (or first 2) and last (or last 2) of those files, and finding the first and last Time values that are within the start and end data of the climatology. A little tricky but definitely doable and way cheaper than loading header information from 500-1000 files for no good reason...

[xylar]

Just a quick update on this. I'm working on a branch that addresses this issue. It involves:

• Caching times from multifile data sets once (and adding additional times later if the range of times changes or new data becomes available)
• Each analysis task that creates a cached climatology having a special function for opening (and maybe computing) a portion of the data set from a given subset of files. This allows only the parts of the data set that haven't already been cached to be read.

With this strategy, all data files will need to be opened only once to read the times in each and once to add their data to a climatology, not every time MPAS-Analysis is called.

[xylar]

I have a branch that takes care of this issue for climatologies (but not yet for time series):
https://github.com/xylar/MPAS-Analysis/tree/climatology_caching_skip_opening_ds
It also makes some other improvements by making the climatology module into classes instead of functions. This lets the code for computing climatologies know more about tasks and thus lets it handle more of the process behind the scenes so there's less redundancy between different tasks.

I'm not going to make a PR for this until we've got #128 off our plates because I'm worried the backlog is too large as it is.

[xylar]

This issue will likely be obsolete if we move to ncclimo in #222. If so, this issue should be closed after that PR is merged.