python-selenium

💻 (30-Mar-2021) See your Python code do web browsing on your screen with GUI.

• Important note

• Requirements

• Installation

  • Configure This Project

  • Run Your Selenium Server Jar File

• Usage

• Using Docker?

  • Existing Admin User When Using Docker

  • Using Docker's Mail Server

• iPython Django Shell

• API

• Cache View Templates

• Contributing

• License

Important note:

Before you try to scrape any website, go through its robots.txt file. You can access it via domainname/robots.txt. There, you will see a list of pages allowed and disallowed for scraping. You should not violate any terms of service of any website you scrape.

Requirements

• Tested using Python 3.13
• Java
• Chromedriver (optional)
• Selenium Server JAR file

Installation

Configure This Project

cp .env.example .env
python -m venv env && \
source env/bin/activate
pip install -r requirements.txt
python manage.py makemigrations python manage.py migrate

Run Your Selenium Server JAR File

Locate where you downloaded your Selenium Server JAR file in the requirements step and run the following.

java -jar selenium-server-[version].jar standalone --override-max-sessions true --max-sessions 10

CLI options in the Selenium Grid.

Usage

Update the command at crawl.py to perform your instructions in web scraping.

python manage.py crawl

XPath element selector cheat sheet.

Using Docker?

alias compose='docker-compose -f local.yml'
compose build
compose up
# Automated runs with Docker:# compose up --build -d && python manage.py crawl

Existing Admin User When Using Docker

The admin user details are set in ./compose/local/django/start.

export DJANGO_SUPERUSER_PASSWORD="${DJANGO_SUPERUSER_PASSWORD:-secret}"
python manage.py createsuperuser \
--no-input \
--username admin_user \
--email admin@django-app.com

Using Docker's Mail Server

Mail environment credentials are at .env.

The Mailhog Docker mail client runs at http://localhost:8025. This is running in the above image that is receiving emails from your Django app.

iPython Django Shell

py manage.py shell -i ipython

API

py manage.py show_urls

Cache View Templates

py manage.py collectstatic

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

BSD