Web Crawler + Search Engine

Web Crawling

• Multithreading: Utilizes multiple threads to crawl multiple domains in parallel.
• Stores Web Pages: Saves web page content into json files to be indexed later.
• Politeness: Adheres to robots.txt and enforces a minimum delay between requests to the same domain.
• Pauseable: Allows crawling to be stopped and resumed at any time without loss of progress.
• Customizeable: Uses a config file to configure seed url, allowable domains, and minimum politeness.

Indexing

• Multiprocessing: Utilizes multiple processes to index multiple web pages at once.
• Partial Indexing: Indexed content is regularly written to files and combined at the end, minimizing memory usage.
• N-gram Indexing: Breaks text into n-grams, enabling more flexible and precise query matching.
• Pauseable: Allows indexing to be stopped and resumed at any time without loss of progress.

Searching

• Ranking: Results are ranked by a combination of keyword frequency and placement in the webpage.
• AI Summary: Generates concise summaries of search results using OpenAI's GPT-3.5.
• Fast Results: Results take only a few milliseconds for an index of 55,000 web pages.

Setup Instructions

Prerequisites

Before you begin, ensure you have the following installed:

• Python 3.6 or higher: You can download the latest version from python.org.
• pip: Python's package installer, typically bundled with Python installations. You can verify its installation by running pip --version in your terminal. If not installed, follow the pip installation guide.

Installing Dependencies

Once you have Python and pip set up, you can install all necessary dependencies by following these steps:

1. Clone this repository to your local machine.
2. Open a terminal or command prompt and navigate to the project directory.
3. Run the following command to install all required packages:

python -m pip install -r requirements.txt

This will download and install all the libraries listed in the requirements.txt file, ensuring you have everything you need to run the project.

Running the Web Crawler

Configuration

Before running the crawler, you can adjust the configuration by modifying the config.ini file. There are three key settings:

• SEEDURL: A comma-separated string of URLs where the crawler will start. Update this with the URLs you want to begin crawling from.
• ROOTDOMAINS: A comma-separated string specifying the root domains the crawler is allowed to explore. This limits the scope of the crawling process.
• POLITENESS: The default minimum time (in seconds) the crawler will wait between making requests to the same domain to avoid overloading servers.

Starting the Crawler

To run the web crawler, navigate to the project directory and use the following command:

python start_crawler.py

Here are additional options you can use:

• --restart: Use this flag to start crawling from scratch, beginning with the seed URLs.
• -n [integer]: Use this option to specify the number of threads for the crawling process. For example, -n 4 will run the crawler with 4 threads, utilizing Python’s multithreading module for parallel crawling.

Stopping and Resuming the Crawler

You can stop the crawling process at any time by pressing Ctrl+C. If you need to resume crawling, simply run the script again, and it will continue from where it left off.

Running the Indexer

Starting the Indexer

To run the indexer, use the following command:

python start_indexer.py

Unlike the crawler, the indexer does not use a configuration file. However, the command-line arguments are similar:

• --restart: Use this flag to start indexing from scratch.
• -n [integer]: Use this option to specify the number of processes for the indexing. For example, -n 4 will run the indexer with 4 processes, utilizing Python’s multiprocessing module for parallel indexing.

Stopping and Resuming the Indexer

As with the crawler, you can stop the indexer at any time by pressing Ctrl+C. You can resume indexing by rerunning the script, and it will pick up where it left off.

Running the Search Engine

To start the search engine interface, use the following command:

streamlit run start_search_engine.py

This will launch a web-based interface where you can type your query into the search box and press the "Search" button to retrieve results.

Usage Demos

Below are some demonstrations of the web crawling, indexing, and search capabilities.

Crawling