GitHub metadata importer

This tool imports GitHub metadata from repositories into the Software Observatory database. It identifies the GitHub repositories listed in the database entries, retrieves metadata for each repository using the GitHub metadata API, and stores the retrieved metadata back in the database.

If you are looking for a tool to import metadata from a single GitHub repository directly, you can use the GitHub metadata importer itself. In particular, this importer relies on this endpoint.

Features

The importer includes several safeguards to make long runs more robust:

• --resume support to continue interrupted runs
• local cache of successfully imported repositories
• local cache of failed repositories
• repository listing cache to avoid rebuilding the input list on every run
• retry support for previously failed repositories
• delays with jitter between requests
• exponential backoff for rate limiting and transient server errors
• JSONL run log for debugging and auditing

These mechanisms are especially useful to reduce the impact of 429 Too Many Requests errors and avoid repeating work after interruptions.

Installation

The tool is written in Python 3.12 and requires the packages listed in requirements.txt.

Install dependencies with:

pip install -r requirements.txt

Configuration

The tool requires the following environment variables to be set:

• MONGO_HOST: the hostname of the MongoDB server.
• MONGO_PORT: the port of the MongoDB server.
• MONGO_USER: the username for the MongoDB server.
• MONGO_PWD: the password for the MongoDB server.
• MONGO_AUTH_SRC: the authentication source for the MongoDB server.
• MONGO_DB: the name of the MongoDB database.
• ALAMBIQUE: the name of the database where the gathered metadata will be stored.
• PRETOOLS: the name of the Pretools database. The tool will read the list of repositories from this database.
• GITHUB_TOKEN: the user GitHub token to use for the GitHub metadata API. The token must have read:packages enabled.

Put these environment variables in a .env file in the root directory of the project.

Usage

To run the tool, execute the following command:

python3 main.py

To import a single repository:

python3 main.py --repo https://github.com/inab/github-metadata-api
python3 main.py --repo inab/github-metadata-api

This skips the PRETOOLS listing entirely and imports only the repository given, whether or not it appears in the database. The repository can be passed as a full GitHub URL or as an OWNER/REPO shorthand. In this mode --resume, --retry-failed and --limit are ignored, so a repository already marked as completed is imported again.

To resume an interrupted run:

python3 main.py --resume

This skips repositories already marked as completed in the local import cache.

To resume and retry previous failures:

python3 main.py --resume --retry-failed

This retries repositories that failed in earlier runs while still skipping successful ones.

To refresh the cached repository listing:

python3 main.py --refresh-listing-cache

This rebuilds the list of repositories from PRETOOLS instead of using the local listing cache.

To import only recently updated entries:

python3 main.py --updated-within-days 10

This only imports repositories whose PRETOOLS entry was updated within the last 10 days (the last_updated_at field of the entry, not the activity of the GitHub repository itself). Omitting the option, or passing 0 or a negative value, imports the whole listing as before.

The windowed listing is kept in its own cache file (repos_to_import.updated-10d.json for the example above), so it never overwrites the full repos_to_import.json. Like the full listing, it is reused until you pass --refresh-listing-cache; the importer logs how old it is on reuse, since a window slides as time passes.

Note that --resume skips every repository already marked as completed, so a repository imported once is never refreshed even when its PRETOOLS entry changes. For a recurring incremental run, use --updated-within-days without --resume.

To Limit the number of repositories processed:

python3 main.py --limit 20

For a slower, safer execution:

python3 main.py --resume --delay 3 --max-retries 8

Command-line options

The importer supports the following options:

• --repo: import a single repository (https://github.com/OWNER/REPO or OWNER/REPO) instead of the PRETOOLS listing.
• --resume: skip repositories already completed in the import cache.
• --retry-failed: when used with --resume, include repositories that failed in previous runs.
• --cache-file: path to the import cache file. Default: github_import_cache.json.
• --listing-cache-file: path to the repository listing cache file. Default: repos_to_import.json.
• --refresh-listing-cache: rebuild the repository list from PRETOOLS
• --updated-within-days: only import repositories whose PRETOOLS entry was updated within the last N days. Uses a separate listing cache file (repos_to_import.updated-<N>d.json) and is ignored with --repo. Default: no filter.
• --run-log-file: path to the JSONL run log file. Default: import_run.jsonl.
• --delay: base delay in seconds between requests. Default: 1.5.
• --max-retries: maximum number of attempts per repository request. Default: 1, meaning a failing repository is recorded and skipped immediately, with no backoff wait. Raise it to trade run time for completeness.
• --limit: maximum number of repositories to process in the current run.

Local cache files

During execution, the importer creates and updates a few local files:

* `repos_to_import.json`: cached list of repository URLs to process.
* `repos_to_import.updated-<N>d.json`: same, restricted to the `--updated-within-days N` window.
* `github_import_cache.json`: cache of completed and failed repositories.
* `import_run.jsonl`: append-only run log with one JSON record per processed repository.

These files allow the importer to resume work safely and avoid repeating already completed imports.

Running with Docker

Images are published to ghcr.io/inab/github-importer. The image contains the source only: no .env, no cache files. Configuration comes in as environment variables at run time, and the state files live on a volume mounted at /data, which is also the working directory.

docker run --rm \
  --env-file .env \
  -v gh-importer-state:/data \
  ghcr.io/inab/github-importer:latest

The entrypoint is main.py, so command-line options are appended directly:

docker run --rm --env-file .env -v gh-importer-state:/data \
  ghcr.io/inab/github-importer:latest --resume --updated-within-days 10

MONGO_HOST needs a value the container can reach. Inside a container localhost is the container itself, so a host-local MongoDB or SSH tunnel has to be reached another way: -e MONGO_HOST=host.docker.internal on Docker Desktop, --network host on Linux, or the server's real hostname.

Mounting /data is what makes a full import restartable. A full pass is around 11,000 repositories and takes hours, and without the volume the import cache dies with the container, leaving --resume nothing to resume from. A host directory works in place of the named volume, but it has to be writable by uid 1000, the unprivileged user the importer runs as.

--updated-within-days compares against a local naive timestamp written by the upstream ETL, so the image sets TZ=Europe/Madrid to match. Pass -e TZ=... if that ETL moves to another host.