feat(quality): measure and gate near-duplicate knowledge pages

[drewstone]

The missing quality signal

A corpus can have perfect source registration and citation syntax while still becoming a swamp of copied pages. Exact source-hash checks do not detect two Markdown pages that restate the same result, and an opaque embedding service would make the quality verdict non-reproducible.

Change

• Add a deterministic, model-free near-duplicate detector using normalized word shingles, Unicode character shingles for short-token text, an inverted candidate index, and Jaccard similarity.
• Propose exact normalized copies before ordinary shingle candidates.
• Ignore high-document-frequency shingles so boilerplate cannot make candidate generation quadratic.
• Bind candidate-pair and retained-report limits explicitly and report truncation.
• Sort every input-derived collection so input order cannot change the result.
• Export the detector and its complete report from the RAG-eval surface.
• Add knowledge-base quality metrics for duplicate page rate, page count, and pair count.
• Add an optional maxNearDuplicatePageRate gate. When a configured gate rests on truncated analysis, fail closed rather than treating an undercount as success.

The detector reports by exact page path and id. Duplicate-page rate counts physical eligible pages, not ids, so a pre-existing duplicate id cannot hide a duplicate page.

Tests

The suite covers exact copies, near copies, unrelated text, input-order determinism, Unicode/URL normalization, short-page exclusion, candidate truncation, malformed options, default metrics, a failing duplicate-rate gate, and fail-closed truncated analysis.

No model call, network access, randomness, or wall clock enters the result.

[tangletools]

✅ Auto-approved drewstone PR — e5655f9f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:16:11Z}

[tangletools]

✅ Auto-approved drewstone PR — c7007d77

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:30:13Z}

[drewstone]

The missing quality signal

A corpus can have perfect source registration and citation syntax while still becoming a swamp of copied pages. Exact source-hash checks do not detect two Markdown pages that restate the same result, and an opaque embedding service would make the quality verdict non-reproducible.

Change

• Add a deterministic, model-free near-duplicate detector using normalized word shingles, Unicode character shingles for short-token text, an inverted candidate index, and Jaccard similarity.
• Propose exact normalized copies before ordinary shingle candidates.
• Ignore high-document-frequency shingles so boilerplate cannot make candidate generation quadratic.
• Bind candidate-pair and retained-report limits explicitly and report truncation.
• Sort every input-derived collection so input order cannot change the result.
• Export the detector and its complete report from the RAG-eval surface.
• Add knowledge-base quality metrics for duplicate page rate, page count, and pair count.
• Add an optional maxNearDuplicatePageRate gate. When a configured gate rests on truncated analysis, fail closed rather than treating an undercount as success.

The detector reports by exact page path and id. Duplicate-page rate counts physical eligible pages, not ids, so a pre-existing duplicate id cannot hide a duplicate page.

Tests

The suite covers exact copies, near copies, unrelated text, input-order determinism, Unicode/URL normalization, short-page exclusion, candidate truncation, malformed options, default metrics, a failing duplicate-rate gate, and fail-closed truncated analysis.

No model call, network access, randomness, or wall clock enters the result.

[tangletools]

✅ Auto-approved drewstone PR — e5655f9f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:16:11Z}

[tangletools]

✅ Auto-approved drewstone PR — c7007d77

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:30:13Z}

[drewstone]

The missing quality signal

A corpus can have perfect source registration and citation syntax while still becoming a swamp of copied pages. Exact source-hash checks do not detect two Markdown pages that restate the same result, and an opaque embedding service would make the quality verdict non-reproducible.

Change

• Add a deterministic, model-free near-duplicate detector using normalized word shingles, Unicode character shingles for short-token text, an inverted candidate index, and Jaccard similarity.
• Propose exact normalized copies before ordinary shingle candidates.
• Ignore high-document-frequency shingles so boilerplate cannot make candidate generation quadratic.
• Bind candidate-pair and retained-report limits explicitly and report truncation.
• Sort every input-derived collection so input order cannot change the result.
• Export the detector and its complete report from the RAG-eval surface.
• Add knowledge-base quality metrics for duplicate page rate, page count, and pair count.
• Add an optional maxNearDuplicatePageRate gate. When a configured gate rests on truncated analysis, fail closed rather than treating an undercount as success.

The detector reports by exact page path and id. Duplicate-page rate counts physical eligible pages, not ids, so a pre-existing duplicate id cannot hide a duplicate page.

Tests

The suite covers exact copies, near copies, unrelated text, input-order determinism, Unicode/URL normalization, short-page exclusion, candidate truncation, malformed options, default metrics, a failing duplicate-rate gate, and fail-closed truncated analysis.

No model call, network access, randomness, or wall clock enters the result.

[tangletools]

✅ Auto-approved drewstone PR — e5655f9f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:16:11Z}

[tangletools]

✅ Auto-approved drewstone PR — c7007d77

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:30:13Z}

[drewstone]

The missing quality signal

A corpus can have perfect source registration and citation syntax while still becoming a swamp of copied pages. Exact source-hash checks do not detect two Markdown pages that restate the same result, and an opaque embedding service would make the quality verdict non-reproducible.

Change

• Add a deterministic, model-free near-duplicate detector using normalized word shingles, Unicode character shingles for short-token text, an inverted candidate index, and Jaccard similarity.
• Propose exact normalized copies before ordinary shingle candidates.
• Ignore high-document-frequency shingles so boilerplate cannot make candidate generation quadratic.
• Bind candidate-pair and retained-report limits explicitly and report truncation.
• Sort every input-derived collection so input order cannot change the result.
• Export the detector and its complete report from the RAG-eval surface.
• Add knowledge-base quality metrics for duplicate page rate, page count, and pair count.
• Add an optional maxNearDuplicatePageRate gate. When a configured gate rests on truncated analysis, fail closed rather than treating an undercount as success.

The detector reports by exact page path and id. Duplicate-page rate counts physical eligible pages, not ids, so a pre-existing duplicate id cannot hide a duplicate page.

Tests

The suite covers exact copies, near copies, unrelated text, input-order determinism, Unicode/URL normalization, short-page exclusion, candidate truncation, malformed options, default metrics, a failing duplicate-rate gate, and fail-closed truncated analysis.

No model call, network access, randomness, or wall clock enters the result.

[tangletools]

✅ Auto-approved drewstone PR — e5655f9f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:16:11Z}

[tangletools]

✅ Auto-approved drewstone PR — c7007d77

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:30:13Z}

[drewstone]

The missing quality signal

A corpus can have perfect source registration and citation syntax while still becoming a swamp of copied pages. Exact source-hash checks do not detect two Markdown pages that restate the same result, and an opaque embedding service would make the quality verdict non-reproducible.

Change

• Add a deterministic, model-free near-duplicate detector using normalized word shingles, Unicode character shingles for short-token text, an inverted candidate index, and Jaccard similarity.
• Propose exact normalized copies before ordinary shingle candidates.
• Ignore high-document-frequency shingles so boilerplate cannot make candidate generation quadratic.
• Bind candidate-pair and retained-report limits explicitly and report truncation.
• Sort every input-derived collection so input order cannot change the result.
• Export the detector and its complete report from the RAG-eval surface.
• Add knowledge-base quality metrics for duplicate page rate, page count, and pair count.
• Add an optional maxNearDuplicatePageRate gate. When a configured gate rests on truncated analysis, fail closed rather than treating an undercount as success.

The detector reports by exact page path and id. Duplicate-page rate counts physical eligible pages, not ids, so a pre-existing duplicate id cannot hide a duplicate page.

Tests

The suite covers exact copies, near copies, unrelated text, input-order determinism, Unicode/URL normalization, short-page exclusion, candidate truncation, malformed options, default metrics, a failing duplicate-rate gate, and fail-closed truncated analysis.

No model call, network access, randomness, or wall clock enters the result.

[tangletools]

✅ Auto-approved drewstone PR — e5655f9f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:16:11Z}

[tangletools]

✅ Auto-approved drewstone PR — c7007d77

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:30:13Z}

[drewstone]

The missing quality signal

A corpus can have perfect source registration and citation syntax while still becoming a swamp of copied pages. Exact source-hash checks do not detect two Markdown pages that restate the same result, and an opaque embedding service would make the quality verdict non-reproducible.

Change

• Add a deterministic, model-free near-duplicate detector using normalized word shingles, Unicode character shingles for short-token text, an inverted candidate index, and Jaccard similarity.
• Propose exact normalized copies before ordinary shingle candidates.
• Ignore high-document-frequency shingles so boilerplate cannot make candidate generation quadratic.
• Bind candidate-pair and retained-report limits explicitly and report truncation.
• Sort every input-derived collection so input order cannot change the result.
• Export the detector and its complete report from the RAG-eval surface.
• Add knowledge-base quality metrics for duplicate page rate, page count, and pair count.
• Add an optional maxNearDuplicatePageRate gate. When a configured gate rests on truncated analysis, fail closed rather than treating an undercount as success.

The detector reports by exact page path and id. Duplicate-page rate counts physical eligible pages, not ids, so a pre-existing duplicate id cannot hide a duplicate page.

Tests

The suite covers exact copies, near copies, unrelated text, input-order determinism, Unicode/URL normalization, short-page exclusion, candidate truncation, malformed options, default metrics, a failing duplicate-rate gate, and fail-closed truncated analysis.

No model call, network access, randomness, or wall clock enters the result.

[tangletools]

✅ Auto-approved drewstone PR — e5655f9f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:16:11Z}

[tangletools]

✅ Auto-approved drewstone PR — c7007d77

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:30:13Z}

[drewstone]

The missing quality signal

A corpus can have perfect source registration and citation syntax while still becoming a swamp of copied pages. Exact source-hash checks do not detect two Markdown pages that restate the same result, and an opaque embedding service would make the quality verdict non-reproducible.

Change

• Add a deterministic, model-free near-duplicate detector using normalized word shingles, Unicode character shingles for short-token text, an inverted candidate index, and Jaccard similarity.
• Propose exact normalized copies before ordinary shingle candidates.
• Ignore high-document-frequency shingles so boilerplate cannot make candidate generation quadratic.
• Bind candidate-pair and retained-report limits explicitly and report truncation.
• Sort every input-derived collection so input order cannot change the result.
• Export the detector and its complete report from the RAG-eval surface.
• Add knowledge-base quality metrics for duplicate page rate, page count, and pair count.
• Add an optional maxNearDuplicatePageRate gate. When a configured gate rests on truncated analysis, fail closed rather than treating an undercount as success.

The detector reports by exact page path and id. Duplicate-page rate counts physical eligible pages, not ids, so a pre-existing duplicate id cannot hide a duplicate page.

Tests

The suite covers exact copies, near copies, unrelated text, input-order determinism, Unicode/URL normalization, short-page exclusion, candidate truncation, malformed options, default metrics, a failing duplicate-rate gate, and fail-closed truncated analysis.

No model call, network access, randomness, or wall clock enters the result.

[tangletools]

✅ Auto-approved drewstone PR — e5655f9f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:16:11Z}

[tangletools]

✅ Auto-approved drewstone PR — c7007d77

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:30:13Z}

[drewstone]

The missing quality signal

A corpus can have perfect source registration and citation syntax while still becoming a swamp of copied pages. Exact source-hash checks do not detect two Markdown pages that restate the same result, and an opaque embedding service would make the quality verdict non-reproducible.

Change

• Add a deterministic, model-free near-duplicate detector using normalized word shingles, Unicode character shingles for short-token text, an inverted candidate index, and Jaccard similarity.
• Propose exact normalized copies before ordinary shingle candidates.
• Ignore high-document-frequency shingles so boilerplate cannot make candidate generation quadratic.
• Bind candidate-pair and retained-report limits explicitly and report truncation.
• Sort every input-derived collection so input order cannot change the result.
• Export the detector and its complete report from the RAG-eval surface.
• Add knowledge-base quality metrics for duplicate page rate, page count, and pair count.
• Add an optional maxNearDuplicatePageRate gate. When a configured gate rests on truncated analysis, fail closed rather than treating an undercount as success.

The detector reports by exact page path and id. Duplicate-page rate counts physical eligible pages, not ids, so a pre-existing duplicate id cannot hide a duplicate page.

Tests

The suite covers exact copies, near copies, unrelated text, input-order determinism, Unicode/URL normalization, short-page exclusion, candidate truncation, malformed options, default metrics, a failing duplicate-rate gate, and fail-closed truncated analysis.

No model call, network access, randomness, or wall clock enters the result.

[tangletools]

✅ Auto-approved drewstone PR — e5655f9f

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:16:11Z}

[tangletools]

✅ Auto-approved drewstone PR — c7007d77

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-17T02:30:13Z}