MPAS-Analysis does not run on OLCF

[vanroekel]

Following the switch to ncremap, MPAS-Analysis does not work on OLCF. @pwolfram's conda install does not have esmf installed preventing any regrinding from working.

Also, the MPAS-Analysis config file for OLCF is broken. It is missing mappingSubdirectory. I will fix and make a small PR.

[milenaveneziani]

Thanks for reporting this, @vanroekel. Indeed we need to update the conda packages on OLCF, NERSC and aims. I can do aims. @pwolfram, @xylar: could you please take care of the other two? Our LANL package is set. thanks.

[xylar]

@milenaveneziani and @pwolfram, I am taking care of edison right now and will check on cori shortly after.

Here are the commands:

conda config --add channels conda-forge
conda install numpy scipy matplotlib netCDF4 xarray dask bottleneck basemap nco fasteners
conda update --all

Note the fasteners package is not needed now but will be required for parallel tasks.

Also, note that the latest nco depends on esmf so that package should not need to be listed explicitly.

[xylar]

One more note, after the upgrade of xarray, it is possible that the current version of MPAS-Analysis in ACMEPreAndPostProcessingScripts will no longer work. @milenaveneziani, can you test this? If not, we will need to put together a patch quickly!

[xylar]

@milenaveneziani, I don't have an OLCF account, so either @pwolfram or @vanroekel will need to do this (depending on who set up the original anaconda installation and who has permission to update it).

[vanroekel]

@xylar, @milenaveneziani, I tried, but don't have write permission, so @pwolfram will have to update.

[milenaveneziani]

@xylar: that is a good point. I just tested ACMEPreAndPostProcessing using beta1 output and indeed I got this error:

Traceback (most recent call last):
  File "python/MPAS-Analysis/run_analysis.py", line 182, in <module>
    analysis(config)
  File "python/MPAS-Analysis/run_analysis.py", line 99, in analysis
    variableMap=oceanVariableMap)
  File "/global/u2/m/milena/ACME-PreAndPostProcessing/coupled_diags/python/MPAS-Analysis/mpas_analysis/ocean/ohc_timeseries.py", line 132, in ohc_timeseries
    preprocess=lambda x: preprocess_mpas(x,
  File "/global/project/projectdirs/acme/software/anaconda_envs/edison/lib/python2.7/site-packages/xarray/backends/api.py", line 509, in open_mfdataset
    datasets = [preprocess(ds) for ds in datasets]
  File "/global/u2/m/milena/ACME-PreAndPostProcessing/coupled_diags/python/MPAS-Analysis/mpas_analysis/ocean/ohc_timeseries.py", line 136, in <lambda>
    varmap=variableMap))
  File "/global/u2/m/milena/ACME-PreAndPostProcessing/coupled_diags/python/MPAS-Analysis/mpas_analysis/shared/mpas_xarray/mpas_xarray.py", line 315, in preprocess_mpas
    datetimes = get_datetimes(ds, timestr, yearoffset)
  File "/global/u2/m/milena/ACME-PreAndPostProcessing/coupled_diags/python/MPAS-Analysis/mpas_analysis/shared/mpas_xarray/mpas_xarray.py", line 164, in get_datetimes
    ends = get_datetimes(ds, timestr[1], yearoffset)
  File "/global/u2/m/milena/ACME-PreAndPostProcessing/coupled_diags/python/MPAS-Analysis/mpas_analysis/shared/mpas_xarray/mpas_xarray.py", line 177, in get_datetimes
    for x in time]
ValueError: invalid literal for int() with base 10: ''

which resembles the restart problem we were seeing before, but I made sure to exclude year 1 from the time series because I knew that the very first file was corrupted (0001-01-01). So, do you think this is due to the newer xarray?

Besides this, people will still have problems running the current ACME analysis because we don't have remapping files for the EC60to30v3 grid..

[vanroekel]

@xylar and @milenaveneziani I was able to get OLCF working again with my own conda install. I've also fixed the titan config file (see #135). @milenaveneziani, do you expect a need for remapping files for the EC6030v3? I've run the mpas-analysis on v3 with no apparent issues. Is this because develop hasn't been pulled to the ACME side?

Example output for 18to6v3 below (this output is year 4 only, not 4-20)

[milenaveneziani]

@vanroekel: yes, it's because develop hasn't been pulled in ACME analysis yet (it's not that immediate, especially now that we have advanced MPAS-Analysis quite a bit with respect to master).
But I am thinking/seeing two things:

1. if I run MPAS-Analysis on beta1, I can create a mapping file for v3, so that will solve this problem;
2. I just tried to do just that, but I am getting the same error as the one I reported above, so this has to have nothing to do with the newer xarray. I am not sure what could be wrong though, since all the beta1 mpas files look OK if we exclude the very first year. I will look into it now..

[xylar]

@milenaveneziani, could you post a separate issue with the problems you're having with ACMEPreAndPostProcessingScripts on Edison? I think the issue should be here, rather than in that repo. Then, we can discuss that issue in that issue.

[milenaveneziani]

I know what is the cause of 2.: the OHC relies on the first year of data. So I will have to skip that for the time being (or have it compute the anomaly wrt the second year of beta1, just for today..).
And regarding 1.: I take it back. Even if we have a mapping file, we still don't have the regridded sea-ice observations.
@xylar: I will rerun the ACME analysis skipping OHC time series. Things may just work.

[xylar]

@milenaveneziani, glad you figured out the error. I'm almost completely sure it's caused by the same issue we saw before (#126) where there is more than one xtime in at least one file.

[milenaveneziani]

@xylar: indeed. I turned off the OHC trend calculation in ACMEPreAndPostProcessing, and things worked: so at least we are sure that the new conda environment does not break it. Phew.
Just for today's coupled telecon, I computed OHC starting from year 2, and things seemed to have worked.
We'll still have the problem of the sea-ice climos in ACME-Analysis, before I pull in MPAS-Analysis into ACMEPreAndPostProcessing. I have to decide what to prioritize: that or the MOC postprocessing. It's hard to decide, since MOC is almost ready..

[pwolfram]

@xylar, I've updated the OLCF conda installation as requested.

[xylar]

@pwolfram, great, thanks!