Ptool

A tool to manage pools across different sites. With Ptool it is possible to take snap-shot of a pool at regular intervals and compare them to monitor the changes to the pool overtime. These snap-shots can be from the same machine or from different machines. It also provides tools to sync the state between pool on different machines.

Installation

The current approach is to clone the repository on the machine where Ptool needs to installed

git clone https://github.com/esm-tools/pool_tool.git
cd pool_tool

Create a virtual environment, either using conda or pyvenv to install the package.

Using Conda

conda env create -f environment.yml
conda activate ptool
pip install .

Usage

Ptool provides 4 commands to manage the pool.

• checksums to create snap-shot of the pool
• summary to get an overview by comparing 2 snap-shots
• compare to write concrete results of comparing 2 snap-shots
• prepare-rsync produces script to transfer files from one machine to another

checksums (snap-shot of pool)

Getting the help text to see the all the options

$ ptool checksums --help
Usage: ptool checksums [OPTIONS] PATH
Calculates imohash checksum of file(s) at the given path. Results are
presented as csv.
`--ignore` and `--ignore-dirs` support *wildcards*in filtering down the
matches. If no *wildcards* are provided, then it performs a literal match.
For multiple patterns, use comma separation.
Options:
--drop-hidden-files / --no-drop-hidden-files
ignore hidden files [default: drop-hidden-
files]
--ignore TEXT ignore files
--ignore-dirs TEXT ignore directories
-o, --outfile FILENAME output filename
--help Show this message and exit.

Lets say, Ptool is installed on Levante and the pool to take snap-shot is fesom2 project, then invoke checksum as follows:

$ ptool checksums --ignore-dirs dist_* -o levante_fesom2.csv /pool/data/AWICM/FESOM2
Gathering files...
skipping.. /pool/data/AWICM/FESOM2/FORCING/ERA5 -> /mnt/lustre01/work/ba1138/a270099/era5/forcing/inverted
getting files Elapsed 0.22s
nfiles: 3060
Calculating hashes...
100%|██████████| 3060/3060 [00:01<00:00, 1610.56files/s]
calculating hashes Elapsed 3.21s
Writing results to levante_fesom2.csv

Remote checksums

It is also possible to get the checksums of pool on the remote site. Lets say we are on Albedo machine and want to compare snap-shot for the project fesom2 from both Albedo and Levante then we can also directly invoke checksums command on Levante from Albedo using ssh command as follows:

$ ssh a270243@levante.dkrz.de "~/miniforge3/envs/ptool/bin/ptool checksums /pool/data/AWICM/FESOM2 --ignore-dirs dist_*"> levante_fesom2.csv
Gathering files...
skipping.. /pool/data/AWICM/FESOM2/FORCING/ERA5 -> /mnt/lustre01/work/ba1138/a270099/era5/forcing/inverted
getting files Elapsed 0.22s
nfiles: 3060
Calculating hashes...
100%|██████████| 3060/3060 [00:01<00:00, 1610.56files/s]
calculating hashes Elapsed 3.21s
Writing results to <stdout>

In the above command, since there is no -o/--outfile option provided, the results are written to <stdout> which is piped to a local file on Albedo. It is also certainly possible to write the snap-shot results to a file on Levante and then copy it over to Albedo. It is up-to the user, how they want to trigger the computation but the main point here is the user is supposed to gather the snap-shots on to the machine where further analysis is carried out.

NOTE: User is free to choose any meaningful name for the csv file as they see fit. The same name is used in displaying the results in the analysis part.

Summary

Lets say we have computed the snap-shot for the project pool fesom2 on both Levante and Albedo, then invoke summary as follows to get a quick overview of the states these pool are in

$ ptool summary --compact levante_fesom2.csv albedo_fesom2.csv
Table 1: Summary with respect to LEVANTE_FESOM2
levante_fesom2 albedo_fesom2
--------------- ------------------------ -------------------
pool FESOM2 FESOM2
checksum file levante_fesom2.csv albedo_fesom2.csv
prefix /pool/data/AWICM/FESOM2/ /albedo/pool/FESOM2
files 3031 (29.8 TB) 1577 (1.1 TB)
duplicate files 56 (615.2 MB) 167 (5.6 GB)
identical files 547 (44.8 GB) 547 (44.8 GB)
unique files 2458 (29.8 TB) nan
modified files nan 9 (17.5 MB)
----------------------------------------------------------------------
Table 2: Common directory mapping
rparent_levante_fesom2 rparent_albedo_fesom2
-- ------------------------ ----------------------------
0 FORCING/CORE2 /forcing/CORE2
1 INITIAL/phc3.0 /hydrography_dsidoren/phc3.0
2 MESHES/CORE2 /core2_meanz_broken
3 MESHES/CORE2/figures /core2_meanz_broken/figures
4 MESHES_FESOM2.1/core2 /core2
5 MESHES_FESOM2.1/hr /HR
6 MESHES_FESOM2.1/mr /mr
----------------------------------------------------------------------
Table 3: LEVANTE_FESOM2 perspective, per directory associations
modified identical unique total
--------------------- ---------- ----------- -------- -------
MESHES_FESOM2.1/hr 1 14 1 16
MESHES_FESOM2.1/mr 1 14 1 16
MESHES_FESOM2.1/core2 7 18 11 36
FORCING/CORE2 - 479 - 479
INITIAL/phc3.0 - 5 - 5
MESHES/CORE2/figures - 8 - 8
MESHES/CORE2 - 9 - 9
----------------------------------------------------------------------

There are more options available for summary command to alter the results. Use the ptool summary --help to see and investigate other options.

comapre

To get the specifics of the per-files associations, use the compare command as follows

$ ptool compare -o lev_alb_fesom2_cmp.csv levante_fesom2.csv albedo_fesom2.csv Writing results as csv to file lev_alb_fesom2_cmp.csv
rpath_left rpath_right
flag identical FORCING/CORE2/ncar_precip.1948.nc /forcing/CORE2/ncar_precip.1948.nc
identical FORCING/CORE2/ncar_precip.1952.nc /forcing/CORE2/ncar_precip.1952.nc
identical FORCING/CORE2/ncar_precip.1955.nc /forcing/CORE2/ncar_precip.1955.nc
identical FORCING/CORE2/ncar_precip.1953.nc /forcing/CORE2/ncar_precip.1953.nc
identical FORCING/CORE2/ncar_precip.1956.nc /forcing/CORE2/ncar_precip.1956.nc
... ... ...
unique FORCING/era5/forcing/inverted/t2m.1972.nc NaN
unique FORCING/era5/forcing/inverted/t2m.1940.nc NaN
unique FORCING/era5/forcing/inverted/t2m.1948.nc NaN
unique FORCING/era5/forcing/inverted/t2m.1960.nc NaN
unique FORCING/era5/forcing/inverted/t2m.1956.nc NaN
[3014 rows x 2 columns]

The comparison results are written to lev_alb_fesom2_cmp.csv file. Explore the contents this file using your favorite editor or search for specific entries using the grep command. For instance, using grep to finding the occurrences of folder MESHES_FESOM2.1/hr

$ grep MESHES_FESOM2.1/hr lev_alb_fesom2_cmp.csv
identical,MESHES_FESOM2.1/hr/edgenum.out,/HR/edgenum.out
identical,MESHES_FESOM2.1/hr/elvls.out,/HR/elvls.out
identical,MESHES_FESOM2.1/hr/hr_griddes_elements.nc,/HR/hr_griddes_elements.nc
identical,MESHES_FESOM2.1/hr/hr_zaxis.txt,/HR/hr_zaxis.txt
identical,MESHES_FESOM2.1/hr/nlvls.out,/HR/nlvls.out
identical,MESHES_FESOM2.1/hr/elem2d.out,/HR/elem2d.out
identical,MESHES_FESOM2.1/hr/hr_griddes_elements_IFS.nc,/HR/hr_griddes_elements_IFS.nc
identical,MESHES_FESOM2.1/hr/nod2d.out,/HR/nod2d.out
identical,MESHES_FESOM2.1/hr/fesom.mesh.diag.nc,/HR/fesom.mesh.diag.nc
identical,MESHES_FESOM2.1/hr/aux3d.out,/HR/aux3d.out
identical,MESHES_FESOM2.1/hr/hr_griddes_nodes.nc,/HR/hr_griddes_nodes.nc
identical,MESHES_FESOM2.1/hr/hr_griddes_nodes_IFS.nc,/HR/hr_griddes_nodes_IFS.nc
identical,MESHES_FESOM2.1/hr/edge_tri.out,/HR/edge_tri.out
identical,MESHES_FESOM2.1/hr/edges.out,/HR/edges.out
modified_latest_right,MESHES_FESOM2.1/hr/README.md,/HR/README.md
unique,MESHES_FESOM2.1/hr/README,

prepare-rsync

This command produces a shell script which contains a list of rsync commands to be executed. Before running the shell script, it is recommended to check the contents of this file to see if prepare-rsync has produced the desired result. User can directly manipulate the shell script to adjust for minor artifacts in-case the options offered by the command does not yield the exact result the user is expecting. Please check out the --help command for details on the options along with few examples of invoking this command.

$ ptool prepare-rsync --help
Usage: ptool prepare-rsync [OPTIONS] LEFT RIGHT
Prepares rsync commands for the transfer.
Denpending on where data needs to pushed or pulled, provide either
`--lefthost` or `--righthost` information to prefix that path.
Note: when Albedo system is invloved, run this command on Albedo and provide
the other host information as Albedo can not be reached from other machines.
Examples that WORK:
# commands executed on Albedo (i.e., we are on Albedo)
1. sync data: Levante -> Albedo
ptool prepare-rsync --lefthost user@levante.dkrz.de
checksum_levante_fesom2.csv checksum_albedo_fesom2.csv
2. sync data: Albedo -> Levante
ptool prepare-rsync --righthost user@levante.dkrz.de
checksum_albedo_fesom2.csv checksum_levante_fesom2.csv
Examples that FAIL:
# commands executed on Levante (i.e., we are on Levante)
1. sync data: Levante -> Albedo
ptool prepare-rsync --righthost user@albedo0.dmawi.de
checksum_levante_fesom2.csv checksum_albedo_fesom2.csv
will produce rsync commands as follows:
rsync /some/path/on/levante user@albedo0.dmawi.de:/some/path/on/albedo
Although syntactically correct command, it fails as Albedo is not
reachable from other machines
2. sync data: Albedo -> Levante
ptool prepare-rsync --lefthost user@albedo0.dmawi.de
checksum_albedo_fesom2.csv checksum_levante_fesom2.csv
Options:
--ignore TEXT ignores directory and files
--flags [unique|modified|both] association type to include
-t, --threshold FLOAT minumin value to satisfy valid association
[default: 0.1]
-l, --lefthost TEXT username@host prefix to the path for left
file
-r, --righthost TEXT username@host prefix to the path for right
file
--help Show this message and exit.

Assuming we are on Albedo and want to transfer the files from Levante to Albedo, invoke the prepare-rsync command as follows:

$ ptool prepare-rsync --lefthost a270243@levante.dkrz.de levante_fesom2.csv albedo_fesom2.csv Created sync_cmd.sh

Verify the contents of sync_cmd.sh before executing the script. It is also possible to directly edit this file to remove selected files from the transaction.

To give a sneak-peak into the sync_cmd.sh file, looking for MESHES_FESOM2.1/hr entry as follows:

$ grep MESHES_FESOM2.1/hr sync_cmd.sh
# MESHES_FESOM2.1/hr
rsync -av --files-from=flist/95087e3b a270243@levante.dkrz.de:/pool/data/AWICM/FESOM2/MESHES_FESOM2.1/hr/ /albedo/pool/FESOM2/HR/