A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.
rate-limitinghttp-requestserror-handlinghtml-parsingdata-collectiontext-processingweb-crawlingcontent-extractionyaml-configurationdata-scrapingpython-crawlermodular-designmetadata-storageurl-normalizationpdf-text-extractionstructured-data-storageconcurrent-crawlingdata-extraction-pipelinedata-preservation-and-recovery
-
Updated
Nov 18, 2024 - Python