fix(display): polish extracted source snippets — strip PSPF banners, repair truncation, normalize bullets - #193
Merged
Merged
Conversation
…repair truncation, normalize bullets Source cards showed raw PDF-extraction artifacts: duplicated OFFICIAL classification banners glued to titles and body text, inline bullet and sub-bullet glyphs, snippets starting mid-clause or mid-numbered-list, and stored synopses cut mid-word before an ellipsis. - stripClassificationBanner: ALL-CAPS, line-anchored PSPF marking removal, wired into the low-yield noise strippers so every compact/snippet/synopsis surface inherits it; verbatim quotes keep banners by design (pinned by test) - sourceTextForCompactDisplay: inline bullets become '; ' separators; the PDF sub-bullet 'o' glyph is normalized conservatively (digit and single-capital guards); repairTruncatedCompactTail repairs stored mid-word cuts and never leaves meaning-inverting stubs (do not… -> …) - compactSourceSnippet: ellipsis-aware fragment pipeline, drops the card's own glued title (dropTitle), sheds orphaned closers and mid-list ordinals, starts at a real sentence when a partial first fragment has substance - cleanDisplayTitle/cleanCitationTitle: banner strip + missing space before acronym parentheticals (Guideline(EMHS) -> Guideline (EMHS)) - chunking: truncateAtWordBoundary replaces raw slices in synopsis/caption builders; banner-only lines dropped and per-sentence banner strip so new chunks store clean synopses (vectors untouched, no re-embed) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… repair) Extends the in-place text-normalization backfill so retrieval_synopsis rows stored before the snippet-polish fix get the same repairs at rest: polishStoredSynopsis applies glyph repair, per-segment protective-marking banner removal (the synopsis prefix format puts the banner mid-string), and truncated-tail repair. content/section_heading keep glyph-only normalization — they feed verbatim quotes and span matching and must not be rewritten. Dry-run default, mandatory backup, and no-re-embed guarantees unchanged; synopsis-only changes now surface in the dry-run samples. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@-