All "scripts" must work without user modification

[pwolfram]

All scripts work without user input (e.g., on example data) or for specific user configurations, reporting errors if they occur. This issue requires at least the following

• some small output, e.g., from a GMPAS-240 run, in order to have appropriate data sets for this to work "out of the box" (to resolve Analysis documentation required (with examples) #3)
• a configuration scheme that allows scripts to be run in batch mode (Single batch processing script to automate analysis across cores #6) following modification of a configuration file (namelist?)

Following this issue only "working" code should be committed to the repository.

[xylar]

I'm running an ACME G-case with the QU240km grid right now to hopefully help address the first checkbox above. I will try to figure out what the minimum sized data set is that we can use as output for testing and also where we can store it if it's too big to live in this repo.

[xylar]

@pwolfram and @milenaveneziani, so the problem with coming up with a "small" test problem to use for validation and CI is that the observational data sets that we currently use as part of the analysis computation is 77 GB in size. It would almost certainly be possible to pull out only a minimum subset of the observations to compare with 2 years of ACME data (the minimum where we can still get a meaningful plot of the time-average ocean data, it seems). But we're still likely talking about multiple GB of data between the ACME output (~1.5-2 GB) and the minimum set of observations.

Any thoughts on how to handle this?

[pwolfram]

@xylar and @milenaveneziani, can we provide some type of mechanism where we pre-cache the results from the observations, e.g., just the data in the line that we want to plot? This may be the way to go in terms of storage requirements. Alternatively, could we store a subsampled time-series or pcolor plot that would be "good enough" and use that dataset? The point is that by the time we get to plotting it is quite likely that the observational dataset used for plotting is quite small because the information contained in a plot is also fairly small unless the image is at very high resolution.

[milenaveneziani]

@xylar, @pwolfram: yes, I think we need to strip down the actual observational data that we use right now. Most of the data files we have now in the obs directories are the original data (monthly or even higher frequency) that we downloaded from the online source. The processed/climatological data is much smaller. I can take a look at the size of what we actually use.
As far as model data goes, we just need 5 years of monthly files at most. For the tutorial, I took 3 years, for example, but maybe 5 could be affordable for the Q240 grid.

[milenaveneziani]

OK, the actual data we use right now is ~500Mb, of which:

1. the majority is taken by the SST, which is a monthly product over years 1870-2012. We could save the climatologies that are now computed by ocean_modelvsobs and reduce that size even further.
2. the sea-ice data only takes <20Mb, but that is because it's the interpolated product onto the 0.5deg regular lon,lat grid. The size would go up if we interpolated onto a finer grid, but that is not the case for the purposes we are discussing here.

[xylar]

@pwolfram wrote:

can we provide some type of mechanism where we pre-cache the results from the observations, e.g., just the data in the line that we want to plot?

I don't understand how that would be useful for testing that the scripts work as expected. It would be skipping the vast majority of their functionality, thus proving nothing.

[xylar]

@milenaveneziani wrote:

As far as model data goes, we just need 5 years of monthly files at most. For the tutorial, I took 3 years, for example, but maybe 5 could be affordable for the Q240 grid.

That's still several GB of data unless we strip out only those fields that we currently use for analysis. I do think that's something we want to try to do -- pare down to a minimum data set that we can use for relatively quick testing -- but even the pared down data is far, far too large for the kind of unit testing @pwolfram has in mind.

[milenaveneziani]

what would be a reasonable size for the testing we are thinking about?

[xylar]

Anything over 100 MB would be pushing our luck, I think.

[pwolfram]

I would agree with @xylar. Ideally we are talking ~< 10 mb.

[milenaveneziani]

that means we'd need to put much more effort in downsizing the data.. Ocean obs on the 1deg grid would have to be interpolated onto a 2deg grid, at least; sea-ice data would also have to be re-interpolated onto a 1deg or 2deg grid.
And we'd have to estimate the size of the stripped out model data as well.

[xylar]

For that amount of work, it would actually be better to just create fake data sets to test with, it seems to me. More importantly, I'm not sure it makes sense to think of the scripts as "units" for unit testing. Instead, we should focus on modularizing the scripts and unit testing the modules.

A parallel goal would be to have a regression test that's light enough weight that it could be run manually (or better yet automatically) on one of the machines we want to make sure we're supporting. This is where I think the GMPAS-240km test I just ran would fit in. It would still be nice to pare down the data to a few hundred MB or so, so we could also perform this regression test on laptops and such. But the goal woudl not be something so small it could be done as part of our CI.