Skip to content
BranchesTags

Latest commit

History

4 Commits

Folders and files

NameName
Last commit message
Last commit date
offline-demo
offline-demo
src
src
.gitignore
.gitignore
README.md
README.md
package-lock.json
package-lock.json
package.json
package.json

Repository files navigation

Spiderweb

Spiderweb is a small interactive web scraper CLI built with Node.js and Playwright. It keeps a browser page open for the whole session, so you can navigate a site, inspect its HTML, click elements, and capture fragments from the current DOM.

Setup

  1. Install the package dependency:

    npm install playwright

  2. Install the Chromium browser binary Playwright uses:

    npx playwright install chromium --no-shell

    This project launches Playwright with Chromium's newer headless mode, so you do not need the separate chromium-headless-shell download.

  3. Start the CLI:

    npm start

You can also open an initial URL right away:

npm start -- example.com

Or run a visible browser window while you experiment:

npm start -- --headed

You can also try the included offline demo page:

npm start -- [path]/spiderweb/offline-demo/demo.html

Commands

help
navigate <url>
goto <url>
show code
capture <css-selector>
click on <css-selector>
text <css-selector>
attr <css-selector> <attribute-name>
images [limit]
links [limit]
title
state
whereami
wait for <css-selector>
screenshot [file-path]
back
reload
history
export json [file-path]
export all [file-path]
download images [directory]
quit
exit

Example sessions

The examples below cover every command. Real-site outputs were captured from nodejs.org on April 4, 2026, so titles and page markup may change as the docs are updated.

Session 1: real website

$ npm start
Spiderweb is ready.
Browser mode: headless
Type "help" to see the available commands.
scraper> navigate https://nodejs.org/api/readline.html
Opened: https://nodejs.org/api/readline.html
Status: 200
Title: Readline | Node.js v25.9.0 Documentation
scraper> title
Readline | Node.js v25.9.0 Documentation
scraper> state
URL: https://nodejs.org/api/readline.html
Title: Readline | Node.js v25.9.0 Documentation
scraper> whereami
URL: https://nodejs.org/api/readline.html
Title: Readline | Node.js v25.9.0 Documentation
scraper> capture h1
Matches: 1
<h1>Node.js v25.9.0 documentation</h1>
scraper> text h1
Matches: 1
Node.js v25.9.0 documentation
scraper> attr link[rel="canonical"] href
Matches: 1
https://nodejs.org/api/readline.html
scraper> links 3
1. Skip to content
https://nodejs.org/api/readline.html#apicontent
2. Node.js
https://nodejs.org/
3. About this documentation
https://nodejs.org/api/documentation.html
scraper> wait for h1
Element appeared: h1
scraper> goto https://nodejs.org/api/path.html
Opened: https://nodejs.org/api/path.html
Status: 200
Title: Path | Node.js v25.9.0 Documentation
scraper> reload
Reloaded: https://nodejs.org/api/path.html
Status: 200
scraper> exit

Session 2: local demo page

$ npm start -- [path]/spiderweb/offline-demo/demo.html
Spiderweb is ready.
Browser mode: headless
Type "help" to see the available commands.
Opened: [path]/spiderweb/offline-demo/demo.html
Title: Spiderweb Demo
scraper> help
Spiderweb commands
help
Show this help menu.
navigate <url>
goto <url>
Open a page. If you omit the protocol, https:// is assumed.
...
images [limit]
List page images, including CSS background images.
...
export json [file-path]
Save the last successful command result as JSON. Defaults to exports/.
export all [file-path]
Save all successful command results from this session as JSON. Defaults to exports/.
download images [directory]
Download all discovered page images. Defaults to downloads/.
quit
exit
Close the browser and leave the CLI.
scraper> show code
<!DOCTYPE html><html lang="en"><head>
<meta charset="utf-8">
<title>Spiderweb Demo</title>
<style>
...
</style>
</head>
<body>
<main>
<h1>Spiderweb Demo</h1>
<p class="summary">This page exists so the CLI can be tested offline.</p>
<section class="gallery">
<img src="./assets/spider.svg" alt="Spider illustration">
<img src="./assets/web.svg" alt="Web illustration">
</section>
<div class="hero-panel" aria-label="Panel background"></div>
<a class="more-link" href="./details.html">Open details</a>
<button class="action-button is-hidden" type="button">
Hidden action
</button>
<button id="hello-button" class="action-button" type="button" onclick="document.querySelector('.summary').textContent = 'The visible action button changed this text.';">
Change summary
</button>
</main>
...
scraper> images
1. [img] Spider illustration
file:///Users/daniel/Desktop/spiderweb/offline-demo/assets/spider.svg
200x150
2. [img] Web illustration
file:///Users/daniel/Desktop/spiderweb/offline-demo/assets/web.svg
200x150
3. [background] Panel background
file:///Users/daniel/Desktop/spiderweb/offline-demo/assets/panel.svg
280x140
scraper> click on .action-button
Clicked first visible match out of 2 total matches.
Visible matches: 1
scraper> text .summary
Matches: 1
The visible action button changed this text.
scraper> click on a.more-link
Clicked first visible match out of 1 total matches.
Visible matches: 1
Current URL: file:///Users/daniel/Desktop/spiderweb/offline-demo/details.html
scraper> back
Navigated back to: file:///Users/daniel/Desktop/spiderweb/offline-demo/demo.html
Title: Spiderweb Demo
scraper> history
1. about:blank
2. file:///Users/daniel/Desktop/spiderweb/offline-demo/demo.html
3. file:///Users/daniel/Desktop/spiderweb/offline-demo/details.html
4. file:///Users/daniel/Desktop/spiderweb/offline-demo/demo.html
scraper> screenshot
Saved screenshot to /Users/daniel/Desktop/spiderweb/screenshots/screenshot-2026-04-04T18-59-11-123Z.png
scraper> export json
Saved JSON export to /Users/daniel/Desktop/spiderweb/exports/result-2026-04-04T18-59-14-456Z.json
scraper> export all
Saved full session export to /Users/daniel/Desktop/spiderweb/exports/session-2026-04-04T18-59-16-789Z.json
scraper> download images
Downloaded 3 of 3 images to /Users/daniel/Desktop/spiderweb/downloads/images-2026-04-04T19-31-54-748Z
scraper> quit

JSON export

export json [file-path] writes only the last successful command result to disk. That means you can:

  • run links 20, then export the link list as JSON
  • run text .headline, then export the extracted text
  • run attr a.cta href, then export the selected attribute with page metadata

Each single-result export includes:

  • the command you ran
  • the current page URL and title
  • the scraped result
  • an exportedAt timestamp

If you omit the file path, Spiderweb writes the file into exports/ automatically with a timestamped filename.

export all [file-path] writes the full session result history so far. It includes:

  • sessionStartedAt
  • exportedAt
  • entryCount
  • entries, where each entry contains recordedAt, command, page, and result

If you omit the file path, Spiderweb writes the file into exports/ automatically with a timestamped filename.

Screenshots

screenshot [file-path] saves a full-page PNG. If you omit the file path, Spiderweb writes the image into screenshots/ automatically with a timestamped filename.

Images

images [limit] lists images on the current page. It includes normal <img> elements and CSS background images, along with a label, source URL, and dimensions.

download images [directory] downloads all discovered page images, including CSS background images. If you omit the directory, Spiderweb writes them into downloads/images-<timestamp>/.

About

Web Crawler CLI

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages