fix(issue-pr-context): floor token estimate at chars/4 for compressible inputs - #2035
Merged
Merged
Conversation
…t on compressible inputs Wave 2 Fix #7 (PR #2026) merged with `tests/scripts/test_issue_formatter.py::test_format_issue_body_caps_oversized_input` failing — a 60K-char body of repeating x's was passing through unchanged, leaving the formatter's preamble and "[truncated: …]" suffix to push the wrapped output to 64K (greater than the input). Root cause: `estimate_tokens()` returns tiktoken's BPE token count when tiktoken is available, but BPE aggressively compresses repeated patterns. 60K identical x's encodes to a handful of tokens, well under the 4000 default budget — so `_cap_block` decides "fits in budget" and returns the body unchanged. Wrappers then push total length above the input. Fix: floor the estimate at chars/4 (the existing fallback when tiktoken is missing), so compressible inputs are still capped against the budget. tiktoken's accurate count still wins for normal prose; only highly-repetitive inputs hit the floor. The consumer-template path `templates/consumer-repo/scripts/langchain/` is gitignored — sync-manifest pulls from this canonical path, so no template lockstep needed. 32 issue_pr_context/issue_formatter tests now pass on Python 3.12. Issue #862 regression test (60K x body → must shrink) now passes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
Automated Status SummaryHead SHA: 8598fb2
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeNo scope information available Tasks
Acceptance criteria
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adjusts token estimation in issue_pr_context to prevent highly-compressible inputs (e.g., long runs of a single character) from bypassing token-budget truncation when tiktoken is installed, which was causing CI failures in the issue formatter truncation test.
Changes:
- Add a chars/
TOKEN_CHARS(rounded up) floor toestimate_tokens()when usingtiktoken. - Expand the
estimate_tokens()docstring to explain why the floor is needed for compressible inputs.
Comment on lines
+42
to
+46
| Returns ``max(tiktoken_count, chars/4)``: tiktoken's BPE encoding aggressively | ||
| compresses repeated patterns (e.g. 60K of the same character collapses to a | ||
| handful of tokens), which would let oversized inputs slip past token-budget | ||
| truncation. The chars/4 floor keeps the cap honest against compressible | ||
| inputs while preserving accurate counts on normal prose. |
This was referenced Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7
Automated Status Summary
Scope
tests/workflows/are excluded from CI and local runs because they import modules from Trend_Model_Project that don't exist in this repository (e.g.,scripts.mypy_return_autofix,scripts.fix_cosmetic_aggregate,scripts.update_autofix_expectations). This represents ~50% of the Python test suite being skipped.Tasks
[tool.ruff] excludein pyproject.toml.--ignoreflags from selftest-ci.yml Python test step.Acceptance criteria
- All 10 previously excluded test files are now included in CI runs.
-
python -m pytest tests/workflows/ -vruns without collection errors.- Test count increases from ~196 to include the previously skipped tests.
- CI workflow passes with expanded test coverage.
Head SHA: 976b4e8
Latest Runs: ✅ success — Gate
Required: gate: ✅ success
| Workflow / Job | Result | Logs |
|----------------|--------|------|
| Agents PR meta manager | ❔ in progress | View run |
| CI Autofix Loop | ✅ success | View run |
| Gate | ✅ success | View run |
| Health 40 Sweep | ✅ success | View run |
| Health 44 Gate Branch Protection | ✅ success | View run |
| Health 45 Agents Guard | ✅ success | View run |
| Health 50 Security Scan | ✅ success | View run |
| Maint 52 Validate Workflows | ✅ success | View run |
| PR 11 - Minimal invariant CI | ✅ success | View run |
| Selftest CI | ✅ success | View run |
Head SHA: d7822a3
Latest Runs: ✅ success — Gate
Required: gate: ✅ success