Uh oh!
There was an error while loading. Please reload this page.
fix: harden resume extractor fallbacks - #194
Conversation
Caution Review failedPull request was closed or merged during review 📝 Walkthrough🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR hardens the resume LLM extraction path by attempting to repair common “near-JSON” model outputs before falling back to heuristics, and by expanding finish_reason='length' retry behavior with an increased token budget and lower-verbosity request parameters.
Changes:
- Add JSON “repair” helpers (strip code fences, extract JSON object substring, remove comments, remove trailing commas) and apply them as a fallback when JSON decoding fails.
- Increase extraction attempts and add additional retry handling for
finish_reason='length', including new completion request kwargs (reasoning_effort,verbosity). - Add unit tests covering malformed JSON recovery and multiple
lengthretries with token escalation.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| tests/unit/test_resume_extractor.py | Adds regression tests for repaired JSON parsing and multi-step length retries, including assertion of new request kwargs. |
| packages/shared/src/five08/resume_extractor.py | Implements JSON repair pipeline and extends LLM retry logic with shared request kwargs and expanded length retry behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if finish_reason == "length" and attempt_index < 2: | ||
| attempt_max_tokens = self.max_tokens * 2 | ||
| if attempt_index == 1: | ||
| attempt_max_tokens = self.max_tokens * 4 | ||
| retry_reason = "length" |
There was a problem hiding this comment.
The max_tokens escalation for finish_reason == 'length' is keyed off attempt_index == 1, which is the second overall attempt—not necessarily the second length response. If attempt 0 fails for another reason (e.g., invalid JSON) and attempt 1 is the first length, this will jump straight to 4× self.max_tokens and skip the intended 2× step. Track a separate length-retry counter (or just double attempt_max_tokens each time you see finish_reason == 'length') so escalation reflects consecutive length truncations.
| if finish_reason == "length" and attempt_index < 2: | ||
| attempt_max_tokens = self.max_tokens * 2 | ||
| if attempt_index == 1: | ||
| attempt_max_tokens = self.max_tokens * 4 | ||
| retry_reason = "length" |
There was a problem hiding this comment.
Same issue as above: this finish_reason == 'length' retry path uses attempt_index == 1 to decide 4× tokens, but attempt_index doesn’t indicate how many length truncations have happened. Consider basing the token increase on a dedicated length_retry_count or on the current attempt_max_tokens value to avoid over-escalating when earlier attempts failed for non-length reasons.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| parts = [f"{i}. {label} {display_name}"] | ||
| if discord_username: | ||
| candidate_line = f"{candidate_line} @{discord_username}" | ||
| parts = [candidate_line] | ||
| parts.append(f"`@{discord_username}`") | ||
There was a problem hiding this comment.
discord_username is now appended as a separate entry in parts, which later gets joined with "\n". This changes output formatting from the previous single-line ... @username to putting the username on its own line. If the intent is to avoid pings while keeping the username inline, append to the first line (or keep a candidate_line variable) instead of adding a new line element.
Uh oh!
There was an error while loading. Please reload this page.
Description
Tightens resume extraction fallbacks by repairing common near-JSON LLM output before giving up, and by retrying
finish_reason='length'responses with a larger token budget and lower verbosity. Adds regression tests for malformed JSON recovery, repeated length retries, and the new request parameters.Related Issue
N/A
How Has This Been Tested?
uv run pytest tests/unit/test_resume_extractor.pygit commitSummary by CodeRabbit
Bug Fixes
Tests