18to6 performance

[vanroekel]

At the request of @milenaveneziani I have been doing timing/testing of MPAS-Analysis. I want to log some information here

1. The MOC on develop does not work for the 18to6 as expected. It hangs in the "load data" phase of the computation. Note this test was for a single year of data.

2. I have run the diagnostics for all_seaIce for the 18to6 on 4 years of data. It took 7.5 hours to complete the seaIce diagnostics alone. This seems concerning to me. If we assume the ocean diagnostics are roughly equivalent, it would be 15 hours for the timeSeries and regridded plots alone for 4 years of data. I will test on the individual tasks to break timings down further.

[xylar]

So the first thing we can do to speed things up is task parallelism (#129). This would be further helped by breaking analysis tasks like sea_ice.modelvsobs and sea_ice.timeseries into subtasks (#133).

From there, we would need to know what the bottlenecks are. If it is loading data, we may need to discuss storing data in a more analysis-friendly way with fewer variables and more time slices per file. Or we may need an intermediate (and optimized) step for pulling to data required by analysis out of the data files and caching it.

If it is computing climatologies, we should explore using nco instead of python or adding threading or MPI parallelism to MPAS-Analysis (a considerable challenge).

@vanroekel, thanks for getting us started. I don't know what is feasible but it might be helpful to move 1 year worth of data (including namelists, streams files, timeSeriesStatsMonthly history files and one restart file each for ocean and sea ice) to a location on turquoise or NERSC. Or I just need to get an account on Titan.

[pwolfram]

@vanroekel, #129 as developed by @xylar should help out a ton with speedup.

Note, we need to be cognizant of the cost to load in memory from disk, which is related to on-node bus speed as well as the total amount of data that needs to be read into memory. This could be the ultimate bottleneck with very large datasets. At that point, we have no real recourse but to use the dask.distributed backend to xarray that allows us to use multiple nodes to read in memory simultaneously, which would allow us to effectively cut the run time by just adding nodes because reading from disk would be parallel. In that scenario, having more time slices in fewer files actually hurts performance. Hence, I would be careful about changing file structure in the short term. It could be a lot of work that may not be quite the performance gain we expect in the long term.

I think the lowest hanging fruit past #129 is probably caching strategies, e.g., compute climatologies etc incrementally, similar to what @xylar has already been employing.

[vanroekel]

@xylar I am now moving a year of 18to6 (ocean timeSeriesStatsMonthly) data to NERSC (/global/cscratch1/sd/lvroekel/18to6Titan) I'll fix up permissions once the transfer is done. Note, that surface area averages is not on for this data, thus the SST time series analysis will not work. Once the ocean data transfers, I'll move seaIce.

A bit of an update on timing, computing ice timeseries alone took 4 hours for the 4 years of data.

[xylar]

Thanks, @vanroekel.

A bit of an update on timing, computing ice timeseries alone took 4 hours for the 4 years of data.

This is almost certainly all about file read time, since the data itself isn't likely to be that big and averaging it should be fast.

@pwolfram, it would be hard to argue against the idea of having fewer variables per file so you didn't have to seek through a file to find what you need (except, of course that you then have a ton of different files). I get what you're saying about multiple time slices in a single file.

For now, I don't think we have the expertise to solve these issues. I guess we're going to have to develop it.

[pwolfram]

@pwolfram, it would be hard to argue against the idea of having fewer variables per file so you didn't have to seek through a file to find what you need (except, of course that you then have a ton of different files). I get what you're saying about multiple time slices in a single file.

I agree in general, although care is needed here because some file systems have more trouble when there are more smaller files as opposed to fewer larger files, so this may not necessarily be a clear-cut win.

For now, I don't think we have the expertise to solve these issues. I guess we're going to have to develop it.

@xylar, can you please specify what expertise we don't have that you think we need? If we use dask.distributed and can effectively load files in parallel this would largely solve the problem because we can just use more nodes to get close to ideal speed-up. This seems like a long-hanging fruit win to me, assuming there aren't too many problems with the admittedly nascent dask.distributed. I'm happy to tap my xarray and dask networks to bring in the correct ideas. These individuals certainly have the necessary expertise. Is there any reason we shouldn't go this route planning on using dask.distributed on top of xarray?

[xylar]

@xylar, can you please specify what expertise we don't have that you think we need? If we use dask.distributed and can effectively load files in parallel this would largely solve the problem because we can just use more nodes to get close to ideal speed-up. This seems like a long-hanging fruit win to me, assuming there aren't too many problems with the admittedly nascent dask.distributed. I'm happy to tap my xarray and dask networks to bring in the correct ideas. These individuals certainly have the necessary expertise. Is there any reason we shouldn't go this route planning on using dask.distributed on top of xarray?

This is the area where we don't have the necessary expertise ourselves and may need to develop it. I'm not as optimistic as you seem to be that using dask.distributed will a) be easy or b) have the performance we need (reasonable total execution time). Maybe once Luke has some files at NERSC for us to play with, we can see what the options are.

[vanroekel]

one other possibility is to utilize the timeSeriesStatsClimatology capability in MPAS (suggested in discussion with @milenaveneziani). This would require far fewer files to be read.

[pwolfram]

@xylar, how about this:

I think the simplest thing we do is a mean, e.g., "climatology", so I would propose a compute off "contest" to compute the mean where we have a set of netcdf files and essentially the fastest and cleanest code implementation for a mean, including and excluding writing results to disk, wins. The metrics:

1. loading time of data and compute time (together because this is really the time to solution)
2. writing time of solution to disk (we need this because we probably should be caching data results)
3. number of code lines written (I would argue this is secondary to 1 and 2 to a limit: e.g., as long as we are within a factor of 2-3 I would take a 10 line solution over a 100-1000 line solution that is marginally faster.)
4. programmer time (This is less of an issue for easy to implement operations, but more of an issue for complex operations. We should keep track of the implementation time for this solution as part of our metrics, however.)

Does this seem reasonable to you? We would use the same dataset on NERSC and the computation and its metrics would be computed via a submitted job so that dedicated nodes are used (and to also avoid spurious performance due to burst-buffer caching).

[pwolfram]

@vanroekel, you are proposing to compute more analytics online, is that correct? That would most certainly alleviate many of the challenges we are facing.

[xylar]

So part of the problem in terms of plotting time series is that we're storing the regionally averaged (therefore tiny) data sets in the same files as the 3D time-averaged data sets. If we put regionally averaged data in their own files, these would be small, manageable files and time series could be produced in about the same amount of time regardless of the size of the MPAS mesh. I think this should be added to the agenda of tomorrow's ACME ice-ocean meeting, and maybe spawn a special topic meeting later this month.

[xylar]

@pwolfram, your compute-off sounds reasonable if there is time to develop several competing solutions. I really need to be developing land-ice related metrics instead of improving what we have. So I won't be able to put time into this particular area of optimization. My main concern is that performance is going to drain a lot of our already very limited resources. I don't know what to do about that, other than try to have as much of the heavy lifting done on the MPAS side, rather than on the analysis side.

[vanroekel]

@xylar I think a special meeting regarding the analysis is on @milenaveneziani's to-do list. And @pwolfram yes, online climatology is indeed what is proposed.

[pwolfram]

@xylar, I think having more work done on the MPAS analysis side is going to always be the way to go because the data is in memory and easy to compute in the AM. Writing a small, reduced dataset out to disk is cheap.

My initial vision of MPAS-Analysis was just that, essentially a 2D and timeseries plotting routine that takes output from MPAS AMs to produce plots. It has obviously evolved into more than that but fundamentally if we can keep that as its bread and butter we should be very successful.

[pwolfram]

@vanroekel, @xylar and @milenaveneziani I agree that we should meet.

[xylar]

My initial vision of MPAS-Analysis was just that, essentially a 2D and timeseries plotting routine that takes output from MPAS AMs to produce plots. It has obviously evolved into more than that but fundamentally if we can keep that as its bread and butter we should be very successful.

So @vanroekel's tests basically prove that this only really works if the resulting data files are also of a reasonable size. If you want to pick out a reasonable amount of data from an unreasonably large file, you apparently spend 1 hour per simulation year doing so. This is where I think we need to work with the ocean team to rethink how timeSeriesStatsMonthly is being written out. Thus, I want to make clear that what I have in mind is not a meeting of the people currently working on MPAS-Analysis but rather anyone working on any aspect of MPAS AMs, so their output can be better designed.