About

This repository contains code that requests all metadata from all HAPI servers daily and creates additional metadata and plots.

• python run.py abouts updates abouts.json, which contains the master list of HAPI servers; see also the README of its repository.

• python run.py catalogs reads /catalog responses (which contain a list of datasets) from each server in abouts.json. The /info response for each dataset from each server is then requested. The catalog response for each server is stored in a subdirectory of hapi-server.org/meta/catalogs. The /info responses for each dataset are stored in a subdirectory of hapi-server.org/meta/infos. The file catalogs-all.json contains all /catalog and /info responses in a single file.

• python run.py availabilities creates dataset availability plots based on the {start,stop}Date found in the dataset /info responses. Plots are stored at hapi-server.org/meta/availabilities/, and they are visible at hapi-server.org/servers when selecting a server and clicking "View SERVER Time Range Coverage."

• python run.py spase creates partial SPASE records for all datasets of all servers. It uses spase.json for configuration information. The output is stored in https://hapi-server.org/meta/spase/.

Development

See etc/crontab.py for commands that are executed nightly.

For testing changes to the table configuration using the nightly generated metadata, use

rsync -avz weigel@rweigel.dynu.net:git/hapi/server-metadata/data .
pip install -e .cd ..; pip install git+https://github.com/rweigel/table-ui
cd server-metadata
python ../table-ui/serve.py --config table/tableui.json --port 8052

To regenerate table metadata, use

python run.py table [server]

Installation

conda create -y -n hapimeta-python-3.11.9 python=3.11.9 && conda activate hapimeta-python-3.11.9
git clone https://github.com/hapi-server/server-metadata &&cd server-metadata
pip install -e .
python run.py spase --servers TestData2.0 --use-remote-catalog
python run.py spase --servers TestData2.0,TestData2.1,TestData3.0,TestData3.1,TestData3.2,TestData3.3 --use-remote-catalog