it-claws

it-claws

Automated concurrent scraper for staging PC driver deployment environments
Report Bug · Request Feature

About The Project

it-claws is a Python-based tool that automatically downloads the latest PC hardware drivers, diagnostic tools, and common software from official vendor websites. Using Selenium for browser automation and httpx for static requests, it navigates vendor sites to retrieve up-to-date installation packages — suitable for staging driver packs for enterprise deployment.

Unlike simple download utilities, it-claws runs concurrently with configurable retry logic, supports dynamic and static page scraping, and can pack everything into a compressed ZIP archive.

The tool serves as a companion to install-it. See the Usage section for more information.

it-claws also supports Docker deployment with tmpfs RAM disk and automated cloud upload via rclone. See scripts for the automation pipeline.

(back to top)

Built With

(back to top)

Usage

Basic flow

1. Select targets

   • Use -t to specify space-separated target names
   • Use -i for interactive selection
   • Use --all to select all targets
   • Use --target-from <file> to load a preset text file

2. Download

   • The tool resolves download URLs (static or dynamic)
   • Files are downloaded concurrently with configurable retries

3. Archive & output

   • Results are staged in the output directory
   • Optionally pack into a compressed ZIP archive

Selecting targets

it-claws -t "AMD Chipset Drivers" "Realtek HD Universal Audio"
it-claws -i
it-claws --all

Use a preset file (one target per line, # for comments):

it-claws --target-from presets/default.txt

Combine with interactive to tweak the preset before running:

it-claws --target-from presets/default.txt -i

Output options

it-claws -o ./my-drivers --max-concurrent 10

Resilience options

it-claws --max-concurrent 10 --retries 2

• --max-concurrent: parallel downloads (default: 3, 1 = sequential)
• --retries: retry attempts per failed download (default: 1)

Archiving options

it-claws -o ./downloads -z ./driver-pack.zip --zip-include install-it/conf --compress-level 9

• -z / --zip PATH: destination for the output ZIP archive
• --zip-include SOURCE[=LAYOUT]: additional files or directories to include in the archive. The <source>[=<layout>] syntax lets you map a source path to a custom entry name inside the ZIP. For example, install-it/conf=settings adds the contents of install-it/conf under a settings/ prefix inside the archive. Can be specified multiple times.
• --zip-prefix PREFIX: control how the output directory is represented in the ZIP. By default, the output directory name is stripped from archive paths. Specify a name to prefix all entries (e.g. --zip-prefix pkg places entries under pkg/).
• -l / --compress-level: compression level 0–9 (default: 5)
• --manifest: generate a manifest.json file at the root of the ZIP archive

How zip entries are determined

By default, the output directory name is stripped from archive paths. Entries are placed directly under the archive root:

it-claws -o ./downloads -z ./driver-pack.zip
# → zip contains: network/..., display/...  (downloads/ prefix stripped)

Use --zip-prefix to add a custom prefix to all entries:

it-claws -o ./downloads -z ./driver-pack.zip --zip-prefix drivers
# → zip contains: drivers/network/..., drivers/display/...

Additional files or directories can be included with --zip-include. Each flag accepts a single source path, optionally followed by =<layout> to remap the entry name inside the archive:

it-claws -o ./downloads -z ./driver-pack.zip --zip-include install-it/conf
# → zip contains: network/..., display/... + install-it/conf/...

it-claws -o ./downloads -z ./driver-pack.zip --zip-include install-it/conf=settings
# → zip contains: network/..., display/... + settings/...  (remapped from install-it/conf)

You can specify --zip-include multiple times to include several paths:

it-claws -o ./downloads -z ./driver-pack.zip \
  --zip-include install-it/conf \
  --zip-include scripts/startup
# → zip contains: network/..., display/... + install-it/conf/... + scripts/startup/...

When the same directory is included via both -o and --zip-include, or when two --zip-include paths overlap, the entries are merged without duplication.

(back to top)

Deployment

This project can be deployed via Docker with tmpfs RAM disk and automated rclone upload. See scripts/ for the automation pipeline and environment variable reference.

docker run --rm --privileged \
  -v /my/host/config:/config \
  -e RC_REMOTE_PATH="onedrive:PC_Deployments" \
  -e RETRIES="2" \
  -e COMPRESS_LEVEL="9" \
  it-claws

Override the default command:

docker run --name=it-claws \
  -v /path/to/config:/config \
  ghcr.io/install-it/it-claws:latest \
  it-claws -t "AMD Chipset Drivers" "Realtek HD Universal Audio"

(back to top)

Getting Started

Prerequisites

• uv
• A supported web browser (Chrome, Edge, or Firefox)

Install dependencies

uv sync

uv sync --group dev # install dev dependencies

Common commands

Run the scraper

it-claws

Display help

it-claws -h

(back to top)