Distributed web scraping/crawling platform including indexing for searching later
distributed web crawler/indexer