feat: add LLM-based topic splitting - #608
Conversation
Replace regex parsing with intelligent LLM-based topic splitting when apply_langchain_formatting is enabled: - Add scripts/langchain/topic_splitter.py: Uses GitHub Models API to intelligently split multi-issue text into individual topics - Update Parse topics step in agents-63-issue-intake.yml to use LLM splitter when apply_langchain_formatting=true - Falls back to regex parser if LLM fails or flag is false This allows handling arbitrary input formats without adding regex patterns - the LLM understands various numbering schemes like 'Issue 1 —', '1.', 'A)', etc.
|
Status | ✅ no new diagnostics |
Automated Status SummaryHead SHA: 7d1ea44
Coverage Overview
Coverage Trend
Top Coverage Hotspots (lowest coverage)
Updated automatically; will refresh on subsequent CI/Docker completions. Keepalive checklistScopeNo scope information available Tasks
Acceptance criteria
|
🤖 Keepalive Loop StatusPR #608 | Agent: Codex | Iteration 0/5 Current State
🔍 Failure Classification| Error type | infrastructure | |
There was a problem hiding this comment.
Pull request overview
This PR adds intelligent LLM-based topic splitting to replace regex-based parsing when apply_langchain_formatting is enabled. The new approach uses the GitHub Models API to parse multi-issue text in various formats without requiring specific patterns like "1.", "A)", etc.
Key Changes:
- Introduces
scripts/langchain/topic_splitter.pyfor LLM-based topic splitting - Updates the Parse topics step in
agents-63-issue-intake.ymlto conditionally use LLM or regex parsing - Implements graceful fallback to regex parser if LLM splitting fails
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 11 comments.
| File | Description |
|---|---|
scripts/langchain/topic_splitter.py |
New LLM-based topic splitter using GitHub Models API to intelligently parse multi-issue text into individual topics |
.github/workflows/agents-63-issue-intake.yml |
Adds LangChain dependency installation and conditional logic to use LLM splitter when apply_langchain_formatting=true, with fallback to existing regex parser |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| """Get LangChain LLM client.""" | ||
| try: | ||
| from langchain_openai import ChatOpenAI | ||
| except ImportError: | ||
| print("langchain_openai not installed", file=sys.stderr) | ||
| return None | ||
|
|
||
| github_token = os.environ.get("GITHUB_TOKEN") | ||
| openai_token = os.environ.get("OPENAI_API_KEY") | ||
| if not github_token and not openai_token: | ||
| return None | ||
|
|
||
| if github_token: | ||
| return ( | ||
| ChatOpenAI( | ||
| model=DEFAULT_MODEL, | ||
| base_url=GITHUB_MODELS_BASE_URL, | ||
| api_key=github_token, | ||
| temperature=0.1, | ||
| ), | ||
| "github-models", | ||
| ) | ||
| return ( | ||
| ChatOpenAI( | ||
| model=DEFAULT_MODEL, | ||
| api_key=openai_token, | ||
| temperature=0.1, | ||
| ), | ||
| "openai", | ||
| ) | ||
|
|
||
|
|
There was a problem hiding this comment.
The _get_llm_client function duplicates LLM client initialization logic that exists in tools/llm_provider.py and other scripts like scripts/langchain/issue_formatter.py. This creates maintenance overhead and potential inconsistencies. Consider creating a shared utility function or importing from tools.llm_provider to ensure consistent LLM client configuration across all scripts.
| """Get LangChain LLM client.""" | |
| try: | |
| from langchain_openai import ChatOpenAI | |
| except ImportError: | |
| print("langchain_openai not installed", file=sys.stderr) | |
| return None | |
| github_token = os.environ.get("GITHUB_TOKEN") | |
| openai_token = os.environ.get("OPENAI_API_KEY") | |
| if not github_token and not openai_token: | |
| return None | |
| if github_token: | |
| return ( | |
| ChatOpenAI( | |
| model=DEFAULT_MODEL, | |
| base_url=GITHUB_MODELS_BASE_URL, | |
| api_key=github_token, | |
| temperature=0.1, | |
| ), | |
| "github-models", | |
| ) | |
| return ( | |
| ChatOpenAI( | |
| model=DEFAULT_MODEL, | |
| api_key=openai_token, | |
| temperature=0.1, | |
| ), | |
| "openai", | |
| ) | |
| """Get LangChain LLM client using the shared provider configuration.""" | |
| try: | |
| from tools.llm_provider import get_llm_client | |
| except ImportError: | |
| print("tools.llm_provider not available; cannot initialize LLM client", file=sys.stderr) | |
| return None | |
| # Delegate to the shared LLM provider to ensure consistent configuration. | |
| return get_llm_client() |
| except ImportError: | ||
| print("langchain_openai not installed", file=sys.stderr) | ||
| return None |
There was a problem hiding this comment.
The error message printed to stderr when langchain_openai is not installed doesn't match the actual error handling. The function returns None on ImportError, but the calling code in split_topics_with_llm raises a RuntimeError with a message about setting GITHUB_TOKEN or OPENAI_API_KEY, which won't help users understand they need to install the langchain_openai package. This could confuse users during troubleshooting.
| except ImportError: | |
| print("langchain_openai not installed", file=sys.stderr) | |
| return None | |
| except ImportError as exc: | |
| raise RuntimeError( | |
| "The 'langchain_openai' package is required to use the topic splitter. " | |
| "Install it with 'pip install langchain-openai'." | |
| ) from exc |
| client_info = _get_llm_client() | ||
| if not client_info: | ||
| raise RuntimeError("No LLM client available. Set GITHUB_TOKEN or OPENAI_API_KEY.") |
There was a problem hiding this comment.
When both github_token and openai_token are not set, the function returns None and the caller raises a generic error about setting tokens. However, if langchain_openai failed to import, the user would also see a message about the missing package. The error flow should distinguish between missing dependencies and missing credentials to provide clearer guidance to users.
| def _generate_guid(title: str) -> str: | ||
| """Generate a stable GUID from the title.""" | ||
| normalized = re.sub(r"\s+", " ", title.strip().lower()) | ||
| return str(uuid.uuid5(uuid.NAMESPACE_DNS, normalized)) |
There was a problem hiding this comment.
The GUID generation function _generate_guid duplicates logic from parse_chatgpt_topics.py (lines 218-219). Both use uuid.uuid5 with NAMESPACE_DNS and normalize titles in the same way. However, the normalization here uses re.sub(r"\s+", " ", ...) while parse_chatgpt_topics.py uses a different pattern. Any difference in normalization will result in different GUIDs for the same topic title when comparing outputs from the LLM splitter vs regex parser, which could cause duplicate issues to be created.
| data = json.loads(json_str) | ||
| except json.JSONDecodeError as e: | ||
| print(f"Failed to parse LLM response as JSON: {e}", file=sys.stderr) | ||
| print(f"Response was: {content[:500]}", file=sys.stderr) |
There was a problem hiding this comment.
The JSON error message truncates the LLM response to only 500 characters, which may not be sufficient to diagnose why the JSON parsing failed. The full response content should be logged (possibly to a debug file) or the truncation limit should be increased to help with troubleshooting LLM response issues.
| print(f"Response was: {content[:500]}", file=sys.stderr) | |
| # Save full response to a debug file for easier troubleshooting | |
| debug_path = Path(os.getenv("TOPIC_SPLITTER_DEBUG_FILE", "topic_splitter_llm_response_debug.txt")) | |
| try: | |
| debug_path.write_text(content, encoding="utf-8") | |
| print(f"Full LLM response has been written to: {debug_path}", file=sys.stderr) | |
| except Exception as write_err: | |
| print(f"Additionally failed to write full LLM response to debug file: {write_err}", file=sys.stderr) | |
| # Still show a truncated version on stderr to avoid overwhelming the terminal | |
| print(f"Response was (truncated): {content[:2000]}", file=sys.stderr) |
| topic = { | ||
| "title": title, | ||
| "guid": _generate_guid(title), | ||
| "labels": [], | ||
| "sections": { | ||
| "why": "", | ||
| "tasks": "", | ||
| "acceptance_criteria": "", | ||
| "implementation_notes": "", | ||
| }, | ||
| "extras": body, | ||
| "enumerator": str(i + 1), | ||
| "continuity_break": False, | ||
| } | ||
| topics.append(topic) |
There was a problem hiding this comment.
The output structure differs from parse_chatgpt_topics.py in that it puts all content in the "extras" field and leaves all section fields empty. According to parse_chatgpt_topics.py lines 112-166, sections like "why", "tasks", "acceptance_criteria", and "implementation_notes" are parsed from the body content. This structural difference means downstream consumers expecting parsed sections will receive empty sections from the LLM splitter but populated sections from the regex parser. This inconsistency could break workflows that depend on these fields being populated.
| try: | ||
| topics = split_topics_with_llm(input_text) | ||
| except RuntimeError as e: | ||
| print(f"Error: {e}", file=sys.stderr) |
There was a problem hiding this comment.
The main function doesn't handle RuntimeError exceptions raised by split_topics_with_llm for LLM call failures (line 122). It only catches RuntimeError at line 194, but other exceptions (like json.JSONDecodeError, which is wrapped in RuntimeError at line 133) will still result in exit code 1. The workflow expects consistent error codes for different failure modes to provide helpful error messages.
| print(f"Error: {e}", file=sys.stderr) | |
| # Distinguish between JSON parsing errors and other LLM-related failures | |
| cause = getattr(e, "__cause__", None) | |
| if isinstance(cause, json.JSONDecodeError): | |
| print(f"Error parsing LLM response: {e}", file=sys.stderr) | |
| # Exit code 3: LLM response could not be parsed | |
| sys.exit(3) | |
| else: | |
| print(f"Error calling LLM: {e}", file=sys.stderr) | |
| # Exit code 4: LLM call failed or returned an invalid result | |
| sys.exit(4) | |
| except Exception as e: | |
| # Catch-all for any other unexpected error | |
| print(f"Unexpected error: {e}", file=sys.stderr) |
| if: needs.normalize_inputs.outputs.apply_langchain_formatting == 'true' | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install langchain langchain-core langchain-openai langchain-community |
There was a problem hiding this comment.
The workflow installs LangChain dependencies with a broad set of packages (langchain, langchain-core, langchain-openai, langchain-community) when only langchain-openai is actually required by topic_splitter.py. Installing unnecessary packages increases build time and potential attack surface. Consider installing only the required langchain-openai package, or verify if the other packages are needed for other parts of the workflow.
| pip install langchain langchain-core langchain-openai langchain-community | |
| pip install langchain-openai |
| if python scripts/langchain/topic_splitter.py --input-file input.txt --output-file topics.json; then | ||
| echo 'LLM splitter succeeded.' | ||
| else | ||
| echo '::warning::LLM splitter failed, falling back to regex parser' | ||
| APPLY_LANGCHAIN=false | ||
| fi |
There was a problem hiding this comment.
The warning message when LLM splitter fails uses '::warning::' but the error doesn't propagate properly. When the LLM splitter fails, APPLY_LANGCHAIN is set to false, but the script continues. However, if the subsequent regex parser also fails, the workflow will exit with an error code, losing context about the LLM failure. Consider logging more details about why the LLM splitter failed to aid debugging.
| #!/usr/bin/env python3 | ||
| """ | ||
| Split raw multi-issue text into individual topics using LLM. | ||
|
|
||
| This replaces regex-based parsing with intelligent LLM-based splitting, | ||
| allowing flexible input formats without adding pattern after pattern. | ||
|
|
||
| Run with: | ||
| python scripts/langchain/topic_splitter.py \ | ||
| --input-file issues.txt --output-file topics.json | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import argparse | ||
| import json | ||
| import os | ||
| import re | ||
| import sys | ||
| import uuid | ||
| from pathlib import Path | ||
| from typing import Any | ||
|
|
||
| # LLM configuration | ||
| GITHUB_MODELS_BASE_URL = "https://models.inference.ai.azure.com" | ||
| DEFAULT_MODEL = "gpt-4o-mini" | ||
|
|
||
| TOPIC_SPLITTER_PROMPT = """ | ||
| You are a text parsing assistant. The input contains one or more GitHub issues | ||
| or feature requests. Your job is to split them into separate, individual issues. | ||
|
|
||
| For EACH issue you identify, extract: | ||
| 1. title: A concise title (the issue heading or first line describing it) | ||
| 2. body: The full content of that issue (Why, Scope, Tasks, etc.) | ||
|
|
||
| Rules: | ||
| - Issues may be numbered ("Issue 1", "1.", "A)") or just separated by headers | ||
| - Preserve ALL content for each issue - don't summarize or truncate | ||
| - Keep markdown formatting, code blocks, and file paths intact | ||
| - If there's only ONE issue in the input, return an array with one item | ||
|
|
||
| Output format - respond with ONLY valid JSON, no other text: | ||
| { | ||
| "issues": [ | ||
| { | ||
| "title": "First issue title", | ||
| "body": "Full body content of first issue..." | ||
| }, | ||
| { | ||
| "title": "Second issue title", | ||
| "body": "Full body content of second issue..." | ||
| } | ||
| ] | ||
| } | ||
|
|
||
| Input text to split: | ||
| {input_text} | ||
| """.strip() | ||
|
|
||
|
|
||
| def _get_llm_client() -> tuple[object, str] | None: | ||
| """Get LangChain LLM client.""" | ||
| try: | ||
| from langchain_openai import ChatOpenAI | ||
| except ImportError: | ||
| print("langchain_openai not installed", file=sys.stderr) | ||
| return None | ||
|
|
||
| github_token = os.environ.get("GITHUB_TOKEN") | ||
| openai_token = os.environ.get("OPENAI_API_KEY") | ||
| if not github_token and not openai_token: | ||
| return None | ||
|
|
||
| if github_token: | ||
| return ( | ||
| ChatOpenAI( | ||
| model=DEFAULT_MODEL, | ||
| base_url=GITHUB_MODELS_BASE_URL, | ||
| api_key=github_token, | ||
| temperature=0.1, | ||
| ), | ||
| "github-models", | ||
| ) | ||
| return ( | ||
| ChatOpenAI( | ||
| model=DEFAULT_MODEL, | ||
| api_key=openai_token, | ||
| temperature=0.1, | ||
| ), | ||
| "openai", | ||
| ) | ||
|
|
||
|
|
||
| def _generate_guid(title: str) -> str: | ||
| """Generate a stable GUID from the title.""" | ||
| normalized = re.sub(r"\s+", " ", title.strip().lower()) | ||
| return str(uuid.uuid5(uuid.NAMESPACE_DNS, normalized)) | ||
|
|
||
|
|
||
| def split_topics_with_llm(input_text: str) -> list[dict[str, Any]]: | ||
| """Use LLM to split raw text into individual topics. | ||
|
|
||
| Args: | ||
| input_text: Raw text containing one or more issues | ||
|
|
||
| Returns: | ||
| List of topic dicts with title, body, guid, labels, sections | ||
| """ | ||
| client_info = _get_llm_client() | ||
| if not client_info: | ||
| raise RuntimeError("No LLM client available. Set GITHUB_TOKEN or OPENAI_API_KEY.") | ||
|
|
||
| llm, provider = client_info | ||
| print(f"Using LLM provider: {provider}", file=sys.stderr) | ||
|
|
||
| prompt = TOPIC_SPLITTER_PROMPT.format(input_text=input_text) | ||
|
|
||
| try: | ||
| response = llm.invoke(prompt) | ||
| content = response.content if hasattr(response, "content") else str(response) | ||
| except Exception as e: | ||
| raise RuntimeError(f"LLM call failed: {e}") from e | ||
|
|
||
| # Extract JSON from response (may be wrapped in markdown code block) | ||
| json_match = re.search(r"```(?:json)?\s*([\s\S]*?)```", content) | ||
| json_str = json_match.group(1).strip() if json_match else content.strip() | ||
|
|
||
| try: | ||
| data = json.loads(json_str) | ||
| except json.JSONDecodeError as e: | ||
| print(f"Failed to parse LLM response as JSON: {e}", file=sys.stderr) | ||
| print(f"Response was: {content[:500]}", file=sys.stderr) | ||
| raise RuntimeError("LLM did not return valid JSON") from e | ||
|
|
||
| issues = data.get("issues", []) | ||
| if not issues: | ||
| raise RuntimeError("LLM returned no issues") | ||
|
|
||
| # Convert to standard topic format | ||
| topics = [] | ||
| for i, issue in enumerate(issues): | ||
| title = issue.get("title", "").strip() | ||
| body = issue.get("body", "").strip() | ||
|
|
||
| if not title: | ||
| title = f"Untitled Issue {i + 1}" | ||
|
|
||
| topic = { | ||
| "title": title, | ||
| "guid": _generate_guid(title), | ||
| "labels": [], | ||
| "sections": { | ||
| "why": "", | ||
| "tasks": "", | ||
| "acceptance_criteria": "", | ||
| "implementation_notes": "", | ||
| }, | ||
| "extras": body, | ||
| "enumerator": str(i + 1), | ||
| "continuity_break": False, | ||
| } | ||
| topics.append(topic) | ||
|
|
||
| return topics | ||
|
|
||
|
|
||
| def main() -> None: | ||
| parser = argparse.ArgumentParser(description="Split raw text into topics using LLM") | ||
| parser.add_argument( | ||
| "--input-file", | ||
| type=Path, | ||
| default=Path("input.txt"), | ||
| help="Input file containing raw issue text", | ||
| ) | ||
| parser.add_argument( | ||
| "--output-file", | ||
| type=Path, | ||
| default=Path("topics.json"), | ||
| help="Output JSON file with split topics", | ||
| ) | ||
| args = parser.parse_args() | ||
|
|
||
| if not args.input_file.exists(): | ||
| print(f"Input file not found: {args.input_file}", file=sys.stderr) | ||
| sys.exit(1) | ||
|
|
||
| input_text = args.input_file.read_text(encoding="utf-8").strip() | ||
| if not input_text: | ||
| print("Input file is empty", file=sys.stderr) | ||
| sys.exit(2) | ||
|
|
||
| try: | ||
| topics = split_topics_with_llm(input_text) | ||
| except RuntimeError as e: | ||
| print(f"Error: {e}", file=sys.stderr) | ||
| sys.exit(1) | ||
|
|
||
| args.output_file.write_text(json.dumps(topics, indent=2), encoding="utf-8") | ||
| print(f"Split into {len(topics)} topic(s). First: {topics[0]['title'][:60]}") | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| main() |
There was a problem hiding this comment.
The new topic_splitter.py script lacks test coverage, while other similar langchain scripts in the same directory (issue_formatter.py, task_decomposer.py, etc.) have comprehensive tests. Tests should cover at minimum: successful LLM splitting, fallback behavior when LLM client is unavailable, JSON parsing from LLM responses, GUID generation, and output format validation against expected structure.
Replace regex parsing with intelligent LLM-based topic splitting when apply_langchain_formatting is enabled.
Changes
scripts/langchain/topic_splitter.py: Uses GitHub Models API to intelligently split multi-issue text into individual topicsagents-63-issue-intake.ymlto use LLM splitter whenapply_langchain_formatting=trueWhy
The regex parser in
parse_chatgpt_topics.pyexpects specific formats like1.,A), etc. But input can come in many formats likeIssue 1 — Title. Instead of adding more regex patterns, we leverage the LLM to intelligently parse any format.Testing
This will be tested with the Travel-Plan-Permission repo's Issue Intake workflow using Issues.txt (5 issues in
Issue N —format).