Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
-
Updated
Aug 15, 2026 - Python
#
url-normalization
Here are 8 public repositories matching this topic...
Remove clutter from URLs and return a canonicalized version
-
Updated
Jun 3, 2024 - Python
Canonical URL normalization and deterministic SEO payload generation for content platforms.
pythonstatic-site-generatordjangoweb-developmentseoopen-graphpython-librarypypitwitter-cardsschema-orgjson-ldmeta-tagsstructured-datahtml-headsearch-engine-optimizationmetadata-generatorfastapiurl-normalizationseo-metadatacanonical-url
-
Updated
Jul 20, 2026 - Python
A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.
rate-limitinghttp-requestserror-handlinghtml-parsingdata-collectiontext-processingweb-crawlingcontent-extractionyaml-configurationdata-scrapingpython-crawlermodular-designmetadata-storageurl-normalizationpdf-text-extractionstructured-data-storageconcurrent-crawlingdata-extraction-pipelinedata-preservation-and-recovery
-
Updated
Nov 18, 2024 - Python
Explainable String Intelligence Engine for Splunk. Detect suspicious domains, emails, usernames and DGA-like strings with transparent scoring and explainable detection.
rule-enginesplunkpython3siemspldgadga-detectiondomain-detectionrisk-scoringurl-normalizationemail-analysisexplainable-scoringreputation-enginesplunk-streaming-commandstring-type-detection
-
Updated
Jul 20, 2026 - Python
This project contains a Python script to extract all unique absolute URLs from a webpage and write them into a text file. This can be useful for indexing purposes.
clihttppython3requestsweb-scrapingretrydata-extractionhtml-parsingargparsebeautifulsoupbs4url-extractorlink-extractorurl-normalizationrobust-error-handlingsingle-page-scraper
-
Updated
Aug 17, 2025 - Python
web crawler from scratch as it possible by using python
-
Updated
Aug 2, 2025 - Python
Synthetic-data pipeline for URL normalization, source extraction, checkpointing, and GEO citation analysis.
-
Updated
Aug 17, 2026 - Python
Improve this page
Add a description, image, and links to the url-normalization topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the url-normalization topic, visit your repo's landing page and select "manage topics."