iharshit009 · TarunTecholution · Dec 11, 2025 · Jan 16, 2026 · Feb 5, 2026
diff --git a/.appmodconfig b/.appmodconfig
diff --git a/README.md b/README.md
@@ -1,9 +1,358 @@
-# Techolution-scraper
-https://techolution.app.param.ai/jobs
+# Techolution Job Scraper
 
+A comprehensive web scraping solution for extracting job postings from dynamic websites. This project provides two scraper implementations: a Playwright-based scraper (`scraper.py`) for flexible URL targeting and a PyQt4-based scraper (`techc.py`) optimized for the Techolution job portal.
 
-Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves programmers hours or days of work.
+## Table of Contents
 
-These instructions illustrate all major features of Beautiful Soup 4, with examples. I show you what the library is good for, how it works, how to use it, how to make it do what you want, and what to do when it violates your expectations.
+- [Installation](#installation)
+- [Usage](#usage)
+ - [Running the Playwright Scraper (scraper.py)](#running-the-playwright-scraper-scraperpy)
+ - [Running the PyQt4 Scraper (techc.py)](#running-the-pyqt4-scraper-techcpy)
+- [Important Considerations](#important-considerations)
+- [Testing](#testing)
+- [Troubleshooting](#troubleshooting)
+- [Project Structure](#project-structure)
 
-Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree.
+---
+
+## Installation + Requirements
+
+### Prerequisites
+
+- Python 3.7 or higher
+- pip (Python package manager)
+
+### Virtual Environment Setup
+
+It is recommended to use a virtual environment to isolate project dependencies.
+
+**Create a virtual environment:**
+
+```bash
+python -m venv venv
+```
+
+**Activate the virtual environment:**
+
+- **Windows:**
+ ```bash
+ venv\Scripts\activate
+ ```
+
+- **macOS/Linux:**
+ ```bash
+ source venv/bin/activate
+ ```
+
+### Install Python Dependencies
+
+Once the virtual environment is activated, install all required Python packages:
+
+```bash
+pip install -r requirements.txt
+```
+
+This will install:
+- **Playwright** (1.40.0): Browser automation for dynamic content scraping
+- **BeautifulSoup4** (4.12.0): HTML parsing and data extraction
+- **Pandas** (2.0.0): Data manipulation and CSV export
+- **PyQt4**: GUI framework for the PyQt4-based scraper
+
+### PyQt4 Installation
+
+**Important:** PyQt4 is not available on PyPI and must be installed separately using system package managers.
+
+#### Ubuntu/Debian:
+
+```bash
+sudo apt-get update
+sudo apt-get install python-qt4
+```
+
+#### macOS (using Homebrew):
+
+```bash
+brew install pyqt4
+```
+
+#### Windows:
+
+For Windows, you have two options:
+
+1. **Using pre-built installers:** Download from the [official PyQt4 website](https://www.riverbankcomputing.com/software/pyqt/download)
+2. **Using conda (if you have Anaconda installed):**
+ ```bash
+ conda install pyqt=4
+ ```
+
+**Note:** If PyQt4 installation fails, the `scraper.py` (Playwright-based) can still be used independently without PyQt4.
+
+---
+
+## Usage
+
+### Running the Playwright Scraper (scraper.py)
+
+The Playwright-based scraper is a flexible, command-line driven tool that can target any website with job postings.
+
+**Basic Command:**
+
+```bash
+python scraper.py --url <URL>
+```
+
+**Arguments:**
+
+- `--url <URL>` (optional): Specify the target website URL to scrape for job postings. Default: `http://example.com/jobs`
+- `--headed` (optional): Run the browser in headed mode (visible window) for debugging. By default, the browser runs in headless mode (invisible).
+
+**Example Commands:**
+
+```bash
+# Scrape with default URL in headless mode
+python scraper.py
+
+# Scrape a specific URL in headless mode
+python scraper.py --url https://example.com/jobs
+
+# Scrape with visible browser window for debugging
+python scraper.py --url https://example.com/jobs --headed
+```
+
+**Output:**
+
+The scraper generates a CSV file named `techolution_jobs.csv` with the following columns:
+- **category**: Job category or department
+- **position**: Job title or position name
+- **type**: Employment type (e.g., Full-time, Part-time, Contract)
+- **location**: Job location or work location
+- **experience level**: Required experience level
+- **posting date**: Date when the job was posted
+
+**Features:**
+
+- Automatic retry logic (3 attempts) for network timeouts
+- Headless and headed browser modes for flexibility
+- Comprehensive error logging and reporting
+- Graceful handling of network failures
+
+### Running the PyQt4 Scraper (techc.py)
+
+The PyQt4-based scraper is optimized for the Techolution job portal and includes resource blocking for improved performance.
+
+**Basic Command:**
+
+```bash
+python techc.py
+```
+
+**Arguments:**
+
+This scraper does not require command-line arguments. It targets the Techolution job portal by default: `https://techolution.app.param.ai/jobs/`
+
+**Output:**
+
+The scraper generates a CSV file named `techolution_jobs_pyqt.csv` with the following columns:
+- **category**: Job category or department
+- **position**: Job title or position name
+- **type**: Employment type (e.g., Full-time, Part-time, Contract)
+- **location**: Job location or work location
+- **experience level**: Required experience level
+- **posting date**: Date when the job was posted
+
+**Features:**
+
+- Resource blocking to optimize page loading (blocks images, stylesheets, and fonts)
+- PyQt4-based browser automation for reliable page rendering
+- Comprehensive error handling and logging
+- Dedicated output file to differentiate from Playwright scraper results
+
+---
+
+## Important Considerations
+
+### Website Structure Dependency
+
+Both scrapers depend on the HTML structure of target websites. If a website updates its layout or CSS selectors, the scrapers may fail to extract data correctly. Regular maintenance and testing are required to ensure continued functionality.
+
+### Performance Differences
+
+- **Playwright Scraper (scraper.py)**: Generally faster due to optimized browser automation
+- **PyQt4 Scraper (techc.py)**: May be slower due to full page rendering overhead, but includes resource blocking to mitigate performance issues
+
+### Resource Blocking
+
+The PyQt4 scraper implements resource blocking to prevent loading of:
+- Images (image/*)
+- Stylesheets (text/css)
+- Fonts (font/*)
+
+This optimization reduces bandwidth usage and improves page loading speed.
+
+### Testing with Live URLs
+
+Always test both scrapers with live URLs to verify functionality before deploying to production. Website changes may break existing selectors and require updates.
+
+### CSV File Encoding
+
+Both scrapers generate CSV files with UTF-8 encoding. Ensure your CSV reader supports UTF-8 encoding to avoid character encoding issues.
+
+---
+
+## Testing
+
+### Manual Testing Scenarios
+
+Test the scrapers with the following scenarios to ensure reliability:
+
+1. **Valid URLs with Job Postings:**
+ - Verify that the scraper correctly extracts all job fields
+ - Check that the CSV file is created with proper formatting
+
+2. **Invalid URLs:**
+ - Test with non-existent URLs to verify error handling
+ - Confirm that appropriate error messages are logged
+
+3. **Pages with No Job Postings:**
+ - Test with pages that don't contain job listings
+ - Verify that the scraper handles empty results gracefully
+
+4. **Network Timeouts:**
+ - Test retry logic by simulating slow network conditions
+ - Verify that the scraper retries and logs timeout errors appropriately
+
+5. **Headed Mode Debugging:**
+ - Run `scraper.py --headed` to visually inspect page loading
+ - Verify that the browser window opens and closes correctly
+
+### Running Tests
+
+If test files are available in the project, run them using:
+
+```bash
+python -m pytest tests/
+```
+
+Or for specific test files:
+
+```bash
+python -m pytest tests/test_scraper.py
+python -m pytest tests/test_techc.py
+```
+
+---
+
+## Troubleshooting
+
+### PyQt4 Installation Issues
+
+**Problem:** PyQt4 installation fails with "No module named PyQt4"
+
+**Solution:**
+- Ensure you are using the correct system package manager for your OS
+- For Ubuntu/Debian: `sudo apt-get install python-qt4`
+- For macOS: `brew install pyqt4`
+- For Windows: Use conda or download pre-built installers
+- If issues persist, the Playwright scraper (`scraper.py`) can be used independently
+
+### Network Timeout Errors
+
+**Problem:** Scraper fails with timeout errors
+
+**Solution:**
+- The scraper includes automatic retry logic (3 attempts) for timeout errors
+- Check your internet connection and network stability
+- For `scraper.py`, use the `--headed` flag to visually inspect page loading:
+ ```bash
+ python scraper.py --url <URL> --headed
+ ```
+- Increase timeout values in the scraper code if needed (advanced users)
+
+### CSV File Encoding Issues
+
+**Problem:** CSV file contains garbled characters or encoding errors
+
+**Solution:**
+- Both scrapers use UTF-8 encoding by default
+- When opening CSV files in Excel, specify UTF-8 encoding during import
+- Use a text editor that supports UTF-8 (e.g., VS Code, Sublime Text)
+- Verify that your system locale supports UTF-8
+
+### Website Structure Changes
+
+**Problem:** Scraper fails to extract data after website updates
+
+**Solution:**
+- Website HTML structure changes require selector updates
+- Inspect the website using browser developer tools (F12)
+- Identify the new CSS selectors for job elements
+- Update the selectors in the scraper code:
+ - For `scraper.py`: Update selectors in the `parse_job_data()` function
+ - For `techc.py`: Update selectors in the `parse_html()` function
+- Test the updated scraper with the `--headed` flag to verify changes
+
+### Browser Launch Failures
+
+**Problem:** Playwright fails to launch browser
+
+**Solution:**
+- Install Playwright browsers:
+ ```bash
+ playwright install
+ ```
+- Ensure you have sufficient disk space for browser binaries
+- Check system permissions and firewall settings
+
+---
+
+## Project Structure
+
+```
+.
+├── README.md # Project documentation (this file)
+├── requirements.txt # Python package dependencies
+├── scraper.py # Playwright-based web scraper
+├── techc.py # PyQt4-based web scraper
+├── techolution_jobs.csv # Output from scraper.py
+└── techolution_jobs_pyqt.csv # Output from techc.py
+```
+
+### File Descriptions
+
+- **scraper.py**: Playwright-based scraper for flexible URL targeting
+ - Uses Playwright for browser automation
+ - Supports command-line arguments for URL and headed mode
+ - Implements retry logic for network resilience
+ - Outputs to `techolution_jobs.csv`
+
+- **techc.py**: PyQt4-based scraper optimized for Techolution portal
+ - Uses PyQt4 for browser automation
+ - Implements resource blocking for performance optimization
+ - Targets Techolution job portal by default
+ - Outputs to `techolution_jobs_pyqt.csv`
+
+- **requirements.txt**: Lists all Python package dependencies
+ - Playwright: Browser automation
+ - BeautifulSoup4: HTML parsing
+ - Pandas: Data manipulation and CSV export
+ - PyQt4: GUI framework
+
+- **Output CSV Files**: Generated by the scrapers
+ - `techolution_jobs.csv`: Output from Playwright scraper
+ - `techolution_jobs_pyqt.csv`: Output from PyQt4 scraper
+ - Both contain columns: category, position, type, location, experience level, posting date
+
+---
+
+## Additional Resources
+
+- [Playwright Documentation](https://playwright.dev/python/)
+- [BeautifulSoup Documentation](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)
+- [PyQt4 Documentation](https://www.riverbankcomputing.com/static/Docs/PyQt4/)
+- [Pandas Documentation](https://pandas.pydata.org/docs/)
+
+---
+
+**Last Updated:** 2024
+
+**Note:** This project is designed for educational and authorized web scraping purposes only. Always respect website terms of service and robots.txt guidelines when scraping.