Uh oh!
There was an error while loading. Please reload this page.
Enhance knowledge extraction with resilience and improved user experience - #9
Merged
Merged
Conversation
…real-time progress Major improvements to the knowledge extraction pipeline: Resilient Processing: - Add graceful degradation for partial failures during extraction - Save partial results when some extractors fail (better than nothing) - Track per-processor status with JSON persistence - Enable selective retry of only failed processors - Add comprehensive error reporting at end of batch runs Focused Extraction: - Split extraction into 4 focused extractors running in parallel: - ConceptExtractor: Technical concepts, patterns, methodologies - RelationshipExtractor: Dependencies and connections - InsightExtractor: Actionable insights and best practices - PatternExtractor: Code patterns and implementations - Each extractor optimized for its specific knowledge type - Better extraction quality through specialized prompts Enhanced User Experience: - Add real-time progress indicators showing parallel extraction - Show which extractors are running vs completed (✓ 2/4 complete) - Display immediate notifications as extractors finish - Fix display glitches with proper line clearing - Remove redundant title lines in output - Add token-based truncation (80k tokens) replacing character limits Error Handling: - Provide clear timeout messages when SDK unavailable - Distinguish between "no data found" and "extraction failed" - Show extraction times for performance monitoring - Continue processing despite individual failures Philosophy: Implements "partial results have value" - a 4-hour batch completing with 80% success is better than early failure with 0% results. This aligns with the project's pragmatic approach to batch processing. Co-Authored-By: Claude <noreply@anthropic.com>
Previously, document classification was implemented but not visible during knowledge-sync processing, causing all documents to default to "general" type. This fix makes classification a visible processing step with clean, animated progress output matching other processors. Changes: - Add _classify_document method to resilient_miner.py with proper async handling - Implement animated spinner progress during classification (matching other extractors) - Update all focused extractors to accept document_type parameter - Change verbose logging from info to debug level to reduce output noise - Fix circular imports using TYPE_CHECKING - Add test file to verify clean classification output The classification now shows a clean animated spinner during processing and displays the document type without extra logging messages, improving user visibility into the knowledge extraction pipeline.
…nd retry defaults Comprehensive improvements to the knowledge extraction system focused on reliability and user experience for processing large batches of articles (350+). Output Display Improvements: - Show clean parallel extraction progress with animated spinner - Display individual processor completions with counts and timing - Properly handle error messages on separate indented lines - Remove duplicate completion reporting between inner/outer scopes - Thread-safe output with proper line clearing Progress Persistence: - Save processing status after each article completes (not just at end) - Skip already-processed articles on restart with clear messages - Show pre-processing summary: "Processing 350 articles (247 complete, 12 partial, 91 unprocessed)" - Support incremental processing with full resume capability - Enable safe interruption with Ctrl+C without losing progress Retry Behavior Changes: - Changed default to retry partial failures (goal: get everything complete) - Added --skip-partial-failures flag for opting out of retry behavior - Deprecated --retry-partial flag (hidden but functional with warning) - Aligns with incremental processing philosophy - common case needs no flags These changes enable reliable unattended processing of large article batches, with the system naturally driving toward 100% completion while providing clear visibility into progress and the ability to resume from any interruption.
…r merge - Rename module from resilient_miner.py to article_processor.py for clarity - Rename class from ResilientKnowledgeMiner to ArticleProcessor - Remove deprecated --retry-partial flag and backwards compatibility code - Fix duplicate completion messages in output by removing redundant logging - Simplify CLI flag handling to use only --skip-partial-failures - Update all imports and references throughout the codebase The new naming better reflects the module's purpose of processing articles through various extractors. Removed technical debt from deprecated flags and fixed output formatting issues for cleaner user experience. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Michael J. Jabbour (michaeljabbour) added a commit
to michaeljabbour/amplifier
that referenced
this pull request
Sep 30, 2025
Complete comprehensive technical specification library with all 44 principles: **People (6 specs)** - #1 Small AI-first working groups - #2 Strategic human touchpoints only - #3 Prompt engineering as core skill - microsoft#4 Test-based verification over code review - microsoft#5 Conversation-driven development - microsoft#6 Human escape hatches always available **Process (13 specs)** - microsoft#7 Regenerate, don't edit - microsoft#8 Contract-first everything - microsoft#9 Tests as the quality gate - microsoft#10 Git as safety net - microsoft#11 Continuous validation with fast feedback - microsoft#12 Incremental processing as default - microsoft#13 Parallel exploration by default - microsoft#14 Context management as discipline - microsoft#15 Git-based everything - microsoft#16 Docs define, not describe - microsoft#17 Prompt versioning and testing - microsoft#18 Contract evolution with migration paths - microsoft#19 Cost and token budgeting **Technology (18 specs)** - microsoft#20 Self-modifying AI-first codebase - microsoft#21 Limited and domain-specific by design - microsoft#22 Layered virtualization - microsoft#23 Protected self-healing kernel - microsoft#24 Long-running agent processes - microsoft#25 Simple interfaces by design - microsoft#26 Stateless by default - microsoft#27 Disposable components everywhere - microsoft#28 CLI-first design - microsoft#29 Tool ecosystems as extensions - microsoft#30 Observability baked in - microsoft#31 Idempotency by design (reference) - microsoft#32 Error recovery patterns built in - microsoft#33 Graceful degradation by design - microsoft#34 Feature flags as deployment strategy - microsoft#35 Least-privilege automation - microsoft#36 Dependency pinning and security scanning - microsoft#37 Declarative over imperative **Governance (7 specs)** - microsoft#38 Access control and compliance - microsoft#39 Metrics and evaluation everywhere - microsoft#40 Knowledge stewardship and institutional memory - microsoft#41 Adaptive sandboxing with explicit approvals - microsoft#42 Data governance and privacy controls - microsoft#43 Model lifecycle management - microsoft#44 Self-serve recovery with known-good snapshots Each specification includes: - Plain-language definition - AI-first development rationale - 4-6 implementation approaches - 5 good/bad example pairs with working code - 6 related principles with relationships - 7 common pitfalls with examples - Tools organized by category - 12 actionable checklist items Statistics: - 44 specifications totaling ~10,000+ lines - 220+ good/bad code example pairs - 240+ implementation approaches - 300+ documented anti-patterns - 500+ tools and frameworks - 250+ cross-principle relationships Created through parallel AI agent execution demonstrating Principle microsoft#13 (Parallel Exploration by Default). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Michael J. Jabbour (michaeljabbour) added a commit
to michaeljabbour/amplifier
that referenced
this pull request
Sep 30, 2025
Add comprehensive CLI tool for managing AI-first principle specifications. **Features:** - List principles (with filtering by category/status) - Validate specifications against quality standards - Quality scoring with comprehensive checks - Progress tracking across all specifications - Stub generation for new principles **Tool Commands:** - list: View all principles with filtering - validate: Check specification structure - check-quality: Comprehensive quality scoring - update-progress: Statistics by category - create: Generate new principle stubs **Quality Checks:** - Required sections present - Minimum 5 example pairs - 6+ related principles - 8-12 checklist items - 5-7 common pitfalls - Complete metadata **Documentation:** - tools/README.md: Complete tool guide - Main README updated with Quick Start and usage **Demonstrates Principles:** - microsoft#28 CLI-First Design - microsoft#29 Tool Ecosystems as Extensions - microsoft#25 Simple Interfaces by Design - microsoft#31 Idempotency by Design - microsoft#9 Tests as Quality Gate Provides path for maintaining and expanding the specification library as Amplifier grows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Joi Ito (Joi) pushed a commit
to Joi/amplifier
that referenced
this pull request
Oct 31, 2025
…ence (microsoft#9) * feat: add resilient knowledge extraction with focused extractors and real-time progress Major improvements to the knowledge extraction pipeline: Resilient Processing: - Add graceful degradation for partial failures during extraction - Save partial results when some extractors fail (better than nothing) - Track per-processor status with JSON persistence - Enable selective retry of only failed processors - Add comprehensive error reporting at end of batch runs Focused Extraction: - Split extraction into 4 focused extractors running in parallel: - ConceptExtractor: Technical concepts, patterns, methodologies - RelationshipExtractor: Dependencies and connections - InsightExtractor: Actionable insights and best practices - PatternExtractor: Code patterns and implementations - Each extractor optimized for its specific knowledge type - Better extraction quality through specialized prompts Enhanced User Experience: - Add real-time progress indicators showing parallel extraction - Show which extractors are running vs completed (✓ 2/4 complete) - Display immediate notifications as extractors finish - Fix display glitches with proper line clearing - Remove redundant title lines in output - Add token-based truncation (80k tokens) replacing character limits Error Handling: - Provide clear timeout messages when SDK unavailable - Distinguish between "no data found" and "extraction failed" - Show extraction times for performance monitoring - Continue processing despite individual failures Philosophy: Implements "partial results have value" - a 4-hour batch completing with 80% success is better than early failure with 0% results. This aligns with the project's pragmatic approach to batch processing. Co-Authored-By: Claude <noreply@anthropic.com> * fix: add visible document classification step with clean progress output Previously, document classification was implemented but not visible during knowledge-sync processing, causing all documents to default to "general" type. This fix makes classification a visible processing step with clean, animated progress output matching other processors. Changes: - Add _classify_document method to resilient_miner.py with proper async handling - Implement animated spinner progress during classification (matching other extractors) - Update all focused extractors to accept document_type parameter - Change verbose logging from info to debug level to reduce output noise - Fix circular imports using TYPE_CHECKING - Add test file to verify clean classification output The classification now shows a clean animated spinner during processing and displays the document type without extra logging messages, improving user visibility into the knowledge extraction pipeline. * feat: enhance knowledge extraction with better output, persistence, and retry defaults Comprehensive improvements to the knowledge extraction system focused on reliability and user experience for processing large batches of articles (350+). Output Display Improvements: - Show clean parallel extraction progress with animated spinner - Display individual processor completions with counts and timing - Properly handle error messages on separate indented lines - Remove duplicate completion reporting between inner/outer scopes - Thread-safe output with proper line clearing Progress Persistence: - Save processing status after each article completes (not just at end) - Skip already-processed articles on restart with clear messages - Show pre-processing summary: "Processing 350 articles (247 complete, 12 partial, 91 unprocessed)" - Support incremental processing with full resume capability - Enable safe interruption with Ctrl+C without losing progress Retry Behavior Changes: - Changed default to retry partial failures (goal: get everything complete) - Added --skip-partial-failures flag for opting out of retry behavior - Deprecated --retry-partial flag (hidden but functional with warning) - Aligns with incremental processing philosophy - common case needs no flags These changes enable reliable unattended processing of large article batches, with the system naturally driving toward 100% completion while providing clear visibility into progress and the ability to resume from any interruption. * refactor: rename resilient_miner to article_processor and clean up for merge - Rename module from resilient_miner.py to article_processor.py for clarity - Rename class from ResilientKnowledgeMiner to ArticleProcessor - Remove deprecated --retry-partial flag and backwards compatibility code - Fix duplicate completion messages in output by removing redundant logging - Simplify CLI flag handling to use only --skip-partial-failures - Update all imports and references throughout the codebase The new naming better reflects the module's purpose of processing articles through various extractors. Removed technical debt from deprecated flags and fixed output formatting issues for cleaner user experience. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Merge remote-tracking branch 'origin/main' into knowledge-recovery
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce a robust knowledge extraction pipeline featuring focused extractors, real-time progress indicators, and enhanced error handling. Implement visible document classification and improve output persistence for better user visibility. Rename and refactor components for clarity and maintainability. These changes align with the project's goal of maximizing value from batch processing, ensuring that partial results are captured even in the event of individual failures.