housing scraper

scrapes dutch rental websites and notifies you via discord whenever there's a new listing zz

setup

pip install -r requirements.txt
python run_scrapers.py

also set up a discord channel webhook and enter the url & your discord user id in .env (otherwise lowkey pointless)

oh - you also need to set up a postgres db, preferably before you run the script for the first time:D

config

edit core/config.py or .env or use command line args:

# basic usage
python run_scrapers.py --city amsterdam --max-price 1200
# specific scrapers only
python run_scrapers.py --scrapers kamernet pararius
# see options
python run_scrapers.py --list

supported sites (so far)

• kamernet
• pararius
• huurwoningen

adding new scrapers (see other scrapers implementation in scrapers/ dir)

1. create scrapers/newsite.py:

fromcore.scraper_utilsimportscrape_genericdefextract_listings(soup):
returnsoup.find_all("div", class_="listing")
defparse_listing(listing):
return {
"title": listing.find("h2").text.strip(),
"price": listing.find(".price").text.strip(),
"url": listing.find("a")["href"]
}
defscrape(city=None, max_price=None):
returnscrape_generic("newsite", extract_listings, parse_listing, city, max_price)

2. add to core/config.py:

SCRAPERS= {
"newsite": True,
}
SCRAPER_URLS= {
"newsite": {
"base": "https://newsite.com",
"search": "/search/{city}?max_price={max_price}"
}
}

that's it - the dynamic import system handles the rest