RUSTJobSpy

A high-performance job board scraper written in Rust — port of Python JobSpy. Concurrently scrapes job postings from multiple platforms and outputs unified CSV or JSON data.

Supported Job Boards

Site	Method	Status
Indeed	TLS fingerprinting + HTML parsing	Works out of the box
LinkedIn	Public guest API	Works out of the box
Bayt	TLS fingerprinting + HTML parsing	Works out of the box
Glassdoor	Headless Chrome (auto-fallback)	Works out of the box*
Naukri	Headless Chrome (auto-fallback)	Works out of the box*
Google Jobs	Headless Chrome	Requires proxy
ZipRecruiter	Headless Chrome	Requires proxy
BDJobs	Headless Chrome	Requires proxy

* Requires Google Chrome installed on the system.

Installation

Prerequisites

• Rust 1.75+ (install via rustup)
• Google Chrome (for Glassdoor, Naukri, and other browser-rendered sites)
• CMake, NASM, LLVM/libclang (for building BoringSSL TLS dependency)

Build from source

git clone https://github.com/Liohtml/RUSTJobSpy.git
cd RUSTJobSpy
cargo build --release

The binary will be at target/release/rustjobspy.

Quick Start

# Search Indeed for software engineer jobs
rustjobspy --search-term "software engineer" --sites indeed --results 10
# Multi-site concurrent search with location filter
rustjobspy --search-term "data scientist" --sites indeed,linkedin,glassdoor --location "New York" --results 5
# Output as JSON to a file
rustjobspy --search-term "rust developer" --sites indeed,linkedin,bayt --results 20 --output json --output-file jobs.json
# Full-featured search with all filters
rustjobspy \
--search-term "backend engineer" \
--sites indeed,linkedin,glassdoor,naukri,bayt \
--location "San Francisco" \
--country usa \
--results 15 \
--remote \
--job-type fulltime \
--hours-old 72 \
--annual-salary \
--linkedin-descriptions \
--output csv \
--output-file results.csv

CLI Reference

Usage: rustjobspy [OPTIONS] --search-term <SEARCH_TERM>
Options:
-s, --search-term <SEARCH_TERM> Search query (required)
-S, --sites <SITES> Comma-separated sites [default: indeed]
-l, --location <LOCATION> Location filter
-c, --country <COUNTRY> Country code [default: usa]
-d, --distance <DISTANCE> Search radius in miles [default: 50]
--remote Filter remote jobs only
-j, --job-type <JOB_TYPE> fulltime, parttime, contract, internship, temporary
--easy-apply Easy apply only
-r, --results <RESULTS> Results per site [default: 15]
--hours-old <HOURS_OLD> Jobs posted within X hours
--proxies <PROXIES> Comma-separated proxy URLs
--annual-salary Normalize salaries to annual
--linkedin-descriptions Fetch full LinkedIn descriptions
--format <FORMAT> Description format: markdown, html, plain [default: markdown]
-o, --output <OUTPUT> Output format: csv or json [default: csv]
--output-file <OUTPUT_FILE> Output file path (stdout if not set)
-v, --verbose <VERBOSE> Verbosity: 0=warn, 1=info, 2=debug [default: 1]
-h, --help Print help
-V, --version Print version

Site Names

Use these values with --sites (comma-separated):

Value	Site
indeed	Indeed
linkedin	LinkedIn
glassdoor	Glassdoor
google	Google Jobs
zip_recruiter	ZipRecruiter
bayt	Bayt.com
naukri	Naukri.com
bd_jobs	BDJobs

Using Proxies

Google Jobs, ZipRecruiter, and BDJobs employ advanced anti-bot protections (Cloudflare JS challenges, CAPTCHAs) that block automated requests even with TLS fingerprinting and headless browsers. To scrape these sites, you need residential proxies.

Recommended Proxy Providers

Provider	Best For	Pricing
Bright Data	All sites, highest success rate	Pay-per-GB
Oxylabs	Enterprise-grade, reliable	Pay-per-GB
ScraperAPI	Simple integration, auto-retry	Pay-per-request
SmartProxy	Good balance of cost/quality	Pay-per-GB
IPRoyal	Budget-friendly residential	Pay-per-GB

Proxy Usage

# Single proxy
rustjobspy --search-term "developer" --sites google,zip_recruiter --proxies "http://user:pass@proxy-host:port"# Multiple proxies (rotated automatically)
rustjobspy --search-term "developer" --sites google,zip_recruiter,bd_jobs \
--proxies "http://user:pass@us1.proxy.com:8080,http://user:pass@us2.proxy.com:8080"# With ScraperAPI (use as HTTP proxy)
rustjobspy --search-term "developer" --sites google \
--proxies "http://scraperapi:YOUR_API_KEY@proxy-server.scraperapi.com:8001"

Proxy Format

http://username:password@host:port
https://username:password@host:port
socks5://username:password@host:port

Why Proxies Are Needed

Site	Protection	Why It Blocks
Google Jobs	reCAPTCHA + IP reputation	Detects datacenter IPs, requires human verification
ZipRecruiter	Cloudflare JS Challenge	Full browser fingerprinting + JavaScript challenge execution
BDJobs	Cloudflare + Angular SPA	Bot detection on initial page load

Residential proxies work because they route traffic through real ISP IP addresses, making requests appear to come from normal users.

Output Format

CSV Columns

site, title, company_name, location, job_url, job_url_direct, date_posted,
job_type, salary_min, salary_max, salary_interval, salary_currency, is_remote,
description, emails, company_url, company_industry, job_level, job_function, skills

JSON Structure

[
{
"site": "indeed",
"title": "Software Engineer",
"company_name": "Acme Corp",
"location": {
"city": "San Francisco",
"state": "CA",
"country": "usa"
},
"job_url": "https://www.indeed.com/viewjob?jk=abc123",
"compensation": {
"interval": "yearly",
"min_amount": 120000,
"max_amount": 180000,
"currency": "USD"
},
"date_posted": "2025-01-15",
"is_remote": false,
"skills": ["Rust", "Python", "AWS"]
}
]

Supported Countries

65+ countries with Indeed/Glassdoor domain routing:

USA, UK, Canada, Australia, India, Germany, France, Brazil, Mexico, Japan, Singapore, China, South Korea, Italy, Spain, Netherlands, Sweden, Switzerland, Norway, Denmark, Finland, Austria, Belgium, Ireland, Portugal, Poland, Czech Republic, Romania, Hungary, Greece, Turkey, Russia, Ukraine, Israel, UAE, Saudi Arabia, Qatar, Kuwait, Bahrain, Oman, Egypt, South Africa, Nigeria, Kenya, Morocco, Argentina, Chile, Colombia, Peru, Venezuela, New Zealand, Indonesia, Malaysia, Philippines, Thailand, Vietnam, Taiwan, Hong Kong, Pakistan, Bangladesh, and more.

Architecture

src/
├── main.rs # CLI (clap)
├── lib.rs # scrape_jobs() orchestrator
├── model.rs # Data models (JobPost, Location, Compensation, enums)
├── error.rs # Site-specific error types
├── util.rs # Salary parsing, email extraction, HTML conversion
├── scraper_trait.rs # Scraper trait + ScraperInput/JobResponse
├── browser.rs # Headless Chrome utility (BrowserFetcher)
└── scrapers/
├── mod.rs # Registry + create_scraper() factory
├── indeed.rs # Indeed HTML scraper
├── linkedin.rs # LinkedIn guest API scraper
├── glassdoor.rs # Glassdoor GraphQL + browser fallback
├── google.rs # Google Jobs browser scraper
├── ziprecruiter.rs # ZipRecruiter browser scraper
├── bayt.rs # Bayt.com HTML scraper
├── naukri.rs # Naukri API + browser fallback
└── bdjobs.rs # BDJobs browser scraper

Key Design Decisions

• Async Rust with Tokio — concurrent scraping across multiple sites
• TLS Fingerprinting (rquest) — impersonates real browsers at the TLS layer to bypass Cloudflare
• Headless Chrome Fallback — for JavaScript-rendered SPAs (Glassdoor, Naukri, Google)
• Scraper Trait Pattern — each site implements a common Scraper trait for extensibility
• Graceful Degradation — HTTP first, browser fallback only when needed

Library Usage

RUSTJobSpy can also be used as a Rust library:

use rustjobspy::{scrape_jobs,ScrapeConfig};use rustjobspy::model::{Site,Country};#[tokio::main]asyncfnmain(){let config = ScrapeConfig{site_names:vec![Site::Indeed,Site::LinkedIn],search_term:"rust developer".to_string(),location:Some("San Francisco".to_string()),country:Country::USA,results_wanted:20,
..Default::default()};matchscrape_jobs(config).await{Ok(jobs) => {println!("Found {} jobs", jobs.len());for job in&jobs {println!(" {} @ {}", job.title.as_deref().unwrap_or("?"),
job.company_name.as_deref().unwrap_or("?"));}}Err(e) => eprintln!("Error: {e}"),}}

Performance

• Concurrent execution — all sites scraped simultaneously via Tokio tasks
• Connection pooling �� HTTP/2 multiplexing with keep-alive
• Deduplication — HashSet-based duplicate detection per scraper
• Minimal memory — streaming HTML parsing, no full DOM retention

Typical timing (5 sites, 3 results each):

• Without browser fallback: ~2-3 seconds
• With Glassdoor/Naukri browser fallback: ~10-15 seconds

Comparison with Python JobSpy

Feature	Python JobSpy	RUSTJobSpy
Language	Python 3.10+	Rust 1.75+
Async	ThreadPoolExecutor	Tokio async/await
TLS Fingerprinting	tls-client	rquest (BoringSSL)
Browser Automation	None	headless_chrome
Output	pandas DataFrame	CSV/JSON
Memory Safety	Runtime	Compile-time
Binary Size	~50MB (with deps)	~15MB
Startup Time	~2s (Python init)	~10ms

License

MIT

Credits

• Original JobSpy by SpeedyApply
• rquest for TLS fingerprinting
• headless_chrome for browser automation
• RUSTScrapling for stealth header patterns