url-normalization

Normalize a URL

Extract and decompose (fuzzy) URLs (including emails, which are conceptually a part of URLs) in texts with Area-Pattern-based modularity

Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters

Canonicalize Varnish cache keys by sorting URL query parameters

Remove clutter from URLs and return a canonicalized version

Get a stable, canonical version of any URL, with DNS and HTTPS checks, redirects, tracker stripping, and canonical link extraction!

Turn any messy URL into a clean canonical https:// string — or null if invalid. Handles bare domains, www/port stripping, query sorting, punycode, domain extraction, and display-friendly humanization. 980B brotli. TypeScript. ESM + CJS.

Allows you to remove ad/tracking query params from a given URL in Scala

🔗 Lightweight Ruby library for parsing URLs and extracting domain components with accurate multi-part TLD support. Handles nested subdomains, query parameters, and URL normalization. Perfect for web scraping, analytics, and URL manipulation. Built on URI and public_suffix gem.

URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical

Canonical URL normalization and deterministic SEO payload generation for content platforms.

A robust, modular web crawler built in Python for extracting and saving content from websites. This crawler is specifically designed to extract text content from both HTML and PDF files, saving them in a structured format with metadata.

Explainable String Intelligence Engine for Splunk. Detect suspicious domains, emails, usernames and DGA-like strings with transparent scoring and explainable detection.

🔗 Pathor is a PHP library for normalizing, analyzing, and comparing URLs.

This project contains a Python script to extract all unique absolute URLs from a webpage and write them into a text file. This can be useful for indexing purposes.

web crawler from scratch as it possible by using python

http url normalization for web crawlers

Natural Language Precessing related notebooks (Machine Learning)

GTM variable template that returns a normalized page path with tracking parameters removed.

A simple plugin to rewrite multiple domains or subdomains to a single domain in the site's HTML output.