Uh oh!
There was an error while loading. Please reload this page.
Enhance smoke testing framework and clean up dependencies - #6
Merged
Conversation
- Remove unnecessary documentation file for PostgreSQL removal - Clean up unused dependencies from pyproject.toml - Update test package description to be generic - Update uv.lock to reflect dependency changes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Implemented a lightweight smoke test system that validates all Makefile commands with minimal overhead. The test suite runs in under 2 minutes and provides clear visibility into what's being tested and validated. Key features: - 31 smoke tests covering all user-facing commands - Enhanced output visibility showing actual command output (2-3 lines) - Smart filtering to skip boilerplate and show meaningful content - Startup tests for heavy commands that verify processing begins - Context display showing where validation patterns are found - Clear pass/fail indicators with informative messages - No external test framework dependencies (uses Python stdlib + YAML) The implementation follows the ruthless simplicity philosophy: - Direct subprocess execution without complex abstractions - Simple YAML configuration for test definitions - Minimal validation that catches 90% of issues - Fast execution (< 2 minutes for full suite) Tests are divided into categories: - Basic commands (help, check, worktree operations) - Content commands (scan, search, status) - Knowledge commands (sync, search, export, events) - Graph commands (build, search, neighbors, export) - Heavy startup tests (verify processing begins within 10 seconds) The enhanced visibility improvements address user feedback by showing what commands actually output and what patterns are being validated, making it clear what each test is checking without overwhelming the observer with unnecessary details. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed Triage Test to create .md file instead of .txt (triage expects markdown) - Removed trivial AI Validation Example test that only echoed text - Fixed Help Command pattern to match 'QUICK START' (uppercase) - Added meaningful validation patterns for tests that only checked exit codes - Fixed Knowledge Events validation to match actual log output format - All 30 smoke tests now pass successfully Note: Content tests take 15-18s because they scan actual filesystem, which is expected behavior for content scanning operations.
…endent tests - Remove Worktree List test that depended on specific branch names - Fix Content Scan validation to check for actual output patterns - Simplify Clean WSL Files test to only verify exit code (cleanup should be silent) - Remove year-specific checks from Knowledge Events test (fragile date dependencies) - Fix Run Tests validation to check for session start instead of PASSED/FAILED These changes make the smoke tests more reliable across different environments and reduce false failures from transient state dependencies. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Replace complex regex-based smoke tests with AI evaluation approach: - Add model configuration system (fast/default/thinking categories) - Create AI-driven test runner that evaluates outputs like a human would - Use isolated test environments to prevent side effects - Simplify test definitions to just commands and success criteria - Remove brittle pattern matching and complex validation logic Key improvements: - Tests complete in ~11 seconds vs previous implementation - Gracefully handles unavailable AI (falls back to basic checks) - No regex patterns or hardcoded validation rules - Reduced from 400+ lines to modular components totaling 413 lines - Follows ruthless simplicity and modular design philosophies 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Fast: claude-3-5-haiku-20241022 (unchanged) - Default: claude-sonnet-4-20250514 (updated from claude-3-5-sonnet-20241022) - Thinking: claude-opus-4-1-20250805 (updated from claude-3-opus-20240229) Updates model configuration to use the latest Claude model versions as specified by the user.
- Replaced Anthropic API with Claude Code SDK for AI evaluation - Made test runner async to support SDK's async operations - Fixed empty pass statement to comply with stub checker - Tests now work within Claude Code environment using SDK - AI provides meaningful pass/fail evaluation for each test - Gracefully handles SDK unavailability outside Claude Code The smoke tests now use the same Claude Code SDK that's already working in the knowledge extraction system, providing consistent AI evaluation across the codebase.
- Add test_article.md with structured content for knowledge extraction testing - Add test_code.py with sample Python function for code processing testing - These files are used by the smoke test isolated environment - Provides consistent test data that won't change over time These test files replace the need to process production content during smoke tests, making tests more predictable and faster while still validating the core functionality works correctly.
- Create data subdirectory for amplifier data isolation - Only create sample files if they don't already exist (preserve test data) - Improve cleanup to only remove cache files, not test data - Add environment variables for model selection and AI skip behavior - Ensure test data directory uses absolute paths These changes make smoke tests more reliable and preserve test data between runs for easier debugging. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
- Add test_article.md with sample content about testing - Add test_code.py with simple Python functions - These files ensure smoke tests have data to work with
- Fix datetime arithmetic in test_parallel_execution.py by using timedelta instead of datetime.replace() - Update graph export to use GEXF format instead of unsupported JSON format - Handle node attributes properly for GEXF export compatibility - Add proper test data structure with knowledge extractions and events - Create workspace detection Makefile for smoke tests - All 19 smoke tests now pass successfully 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Remove previously tracked runtime artifacts from .smoke_test_data/data/ and add the directory to .gitignore to prevent future tracking. These files (query logs, knowledge graphs, extractions) are generated during smoke test execution and should remain local-only. Previously tracked files removed: - .smoke_test_data/data/knowledge/events.jsonl - .smoke_test_data/data/knowledge/extractions.jsonl - .smoke_test_data/data/knowledge/graph.gexf - .smoke_test_data/data/knowledge/query_log.jsonl These runtime artifacts were inadvertently committed but are now properly excluded from version control while preserving test fixture files in the parent .smoke_test_data/ directory. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Michael J. Jabbour (michaeljabbour) added a commit
to michaeljabbour/amplifier
that referenced
this pull request
Sep 30, 2025
Complete comprehensive technical specification library with all 44 principles: **People (6 specs)** - #1 Small AI-first working groups - #2 Strategic human touchpoints only - #3 Prompt engineering as core skill - microsoft#4 Test-based verification over code review - microsoft#5 Conversation-driven development - microsoft#6 Human escape hatches always available **Process (13 specs)** - microsoft#7 Regenerate, don't edit - microsoft#8 Contract-first everything - microsoft#9 Tests as the quality gate - microsoft#10 Git as safety net - microsoft#11 Continuous validation with fast feedback - microsoft#12 Incremental processing as default - microsoft#13 Parallel exploration by default - microsoft#14 Context management as discipline - microsoft#15 Git-based everything - microsoft#16 Docs define, not describe - microsoft#17 Prompt versioning and testing - microsoft#18 Contract evolution with migration paths - microsoft#19 Cost and token budgeting **Technology (18 specs)** - microsoft#20 Self-modifying AI-first codebase - microsoft#21 Limited and domain-specific by design - microsoft#22 Layered virtualization - microsoft#23 Protected self-healing kernel - microsoft#24 Long-running agent processes - microsoft#25 Simple interfaces by design - microsoft#26 Stateless by default - microsoft#27 Disposable components everywhere - microsoft#28 CLI-first design - microsoft#29 Tool ecosystems as extensions - microsoft#30 Observability baked in - microsoft#31 Idempotency by design (reference) - microsoft#32 Error recovery patterns built in - microsoft#33 Graceful degradation by design - microsoft#34 Feature flags as deployment strategy - microsoft#35 Least-privilege automation - microsoft#36 Dependency pinning and security scanning - microsoft#37 Declarative over imperative **Governance (7 specs)** - microsoft#38 Access control and compliance - microsoft#39 Metrics and evaluation everywhere - microsoft#40 Knowledge stewardship and institutional memory - microsoft#41 Adaptive sandboxing with explicit approvals - microsoft#42 Data governance and privacy controls - microsoft#43 Model lifecycle management - microsoft#44 Self-serve recovery with known-good snapshots Each specification includes: - Plain-language definition - AI-first development rationale - 4-6 implementation approaches - 5 good/bad example pairs with working code - 6 related principles with relationships - 7 common pitfalls with examples - Tools organized by category - 12 actionable checklist items Statistics: - 44 specifications totaling ~10,000+ lines - 220+ good/bad code example pairs - 240+ implementation approaches - 300+ documented anti-patterns - 500+ tools and frameworks - 250+ cross-principle relationships Created through parallel AI agent execution demonstrating Principle microsoft#13 (Parallel Exploration by Default). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Joi Ito (Joi) pushed a commit
to Joi/amplifier
that referenced
this pull request
Oct 31, 2025
* chore: complete cleanup of removed dependencies - Remove unnecessary documentation file for PostgreSQL removal - Clean up unused dependencies from pyproject.toml - Update test package description to be generic - Update uv.lock to reflect dependency changes 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: add comprehensive smoke test suite with enhanced visibility Implemented a lightweight smoke test system that validates all Makefile commands with minimal overhead. The test suite runs in under 2 minutes and provides clear visibility into what's being tested and validated. Key features: - 31 smoke tests covering all user-facing commands - Enhanced output visibility showing actual command output (2-3 lines) - Smart filtering to skip boilerplate and show meaningful content - Startup tests for heavy commands that verify processing begins - Context display showing where validation patterns are found - Clear pass/fail indicators with informative messages - No external test framework dependencies (uses Python stdlib + YAML) The implementation follows the ruthless simplicity philosophy: - Direct subprocess execution without complex abstractions - Simple YAML configuration for test definitions - Minimal validation that catches 90% of issues - Fast execution (< 2 minutes for full suite) Tests are divided into categories: - Basic commands (help, check, worktree operations) - Content commands (scan, search, status) - Knowledge commands (sync, search, export, events) - Graph commands (build, search, neighbors, export) - Heavy startup tests (verify processing begins within 10 seconds) The enhanced visibility improvements address user feedback by showing what commands actually output and what patterns are being validated, making it clear what each test is checking without overwhelming the observer with unnecessary details. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * Merge branch 'main' of https://github.com/microsoft/amplifier into clean-up * fix: improve smoke test validation and remove unhelpful tests - Fixed Triage Test to create .md file instead of .txt (triage expects markdown) - Removed trivial AI Validation Example test that only echoed text - Fixed Help Command pattern to match 'QUICK START' (uppercase) - Added meaningful validation patterns for tests that only checked exit codes - Fixed Knowledge Events validation to match actual log output format - All 30 smoke tests now pass successfully Note: Content tests take 15-18s because they scan actual filesystem, which is expected behavior for content scanning operations. * fix: improve smoke test robustness and remove fragile environment-dependent tests - Remove Worktree List test that depended on specific branch names - Fix Content Scan validation to check for actual output patterns - Simplify Clean WSL Files test to only verify exit code (cleanup should be silent) - Remove year-specific checks from Knowledge Events test (fragile date dependencies) - Fix Run Tests validation to check for session start instead of PASSED/FAILED These changes make the smoke tests more reliable across different environments and reduce false failures from transient state dependencies. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * feat: implement AI-driven smoke test system with ruthless simplicity Replace complex regex-based smoke tests with AI evaluation approach: - Add model configuration system (fast/default/thinking categories) - Create AI-driven test runner that evaluates outputs like a human would - Use isolated test environments to prevent side effects - Simplify test definitions to just commands and success criteria - Remove brittle pattern matching and complex validation logic Key improvements: - Tests complete in ~11 seconds vs previous implementation - Gracefully handles unavailable AI (falls back to basic checks) - No regex patterns or hardcoded validation rules - Reduced from 400+ lines to modular components totaling 413 lines - Follows ruthless simplicity and modular design philosophies 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * fix: update model versions to latest Claude releases - Fast: claude-3-5-haiku-20241022 (unchanged) - Default: claude-sonnet-4-20250514 (updated from claude-3-5-sonnet-20241022) - Thinking: claude-opus-4-1-20250805 (updated from claude-3-opus-20240229) Updates model configuration to use the latest Claude model versions as specified by the user. * fix: update smoke tests to use Claude Code SDK for AI evaluation - Replaced Anthropic API with Claude Code SDK for AI evaluation - Made test runner async to support SDK's async operations - Fixed empty pass statement to comply with stub checker - Tests now work within Claude Code environment using SDK - AI provides meaningful pass/fail evaluation for each test - Gracefully handles SDK unavailability outside Claude Code The smoke tests now use the same Claude Code SDK that's already working in the knowledge extraction system, providing consistent AI evaluation across the codebase. * test: add sample test data files for smoke testing - Add test_article.md with structured content for knowledge extraction testing - Add test_code.py with sample Python function for code processing testing - These files are used by the smoke test isolated environment - Provides consistent test data that won't change over time These test files replace the need to process production content during smoke tests, making tests more predictable and faster while still validating the core functionality works correctly. * fix: improve smoke test configuration and data handling - Create data subdirectory for amplifier data isolation - Only create sample files if they don't already exist (preserve test data) - Improve cleanup to only remove cache files, not test data - Add environment variables for model selection and AI skip behavior - Ensure test data directory uses absolute paths These changes make smoke tests more reliable and preserve test data between runs for easier debugging. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * test: add sample test data files for smoke testing - Add test_article.md with sample content about testing - Add test_code.py with simple Python functions - These files ensure smoke tests have data to work with * fix: resolve all smoke test failures and ensure test data integrity - Fix datetime arithmetic in test_parallel_execution.py by using timedelta instead of datetime.replace() - Update graph export to use GEXF format instead of unsupported JSON format - Handle node attributes properly for GEXF export compatibility - Add proper test data structure with knowledge extractions and events - Create workspace detection Makefile for smoke tests - All 19 smoke tests now pass successfully 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: remove smoke test runtime data from git tracking Remove previously tracked runtime artifacts from .smoke_test_data/data/ and add the directory to .gitignore to prevent future tracking. These files (query logs, knowledge graphs, extractions) are generated during smoke test execution and should remain local-only. Previously tracked files removed: - .smoke_test_data/data/knowledge/events.jsonl - .smoke_test_data/data/knowledge/extractions.jsonl - .smoke_test_data/data/knowledge/graph.gexf - .smoke_test_data/data/knowledge/query_log.jsonl These runtime artifacts were inadvertently committed but are now properly excluded from version control while preserving test fixture files in the parent .smoke_test_data/ directory. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Mollie Munoz (momuno) added a commit
to momuno/amplifier
that referenced
this pull request
Nov 20, 2025
… for v0.3.0 Created feature scope and sprint plan for v0.3.0 "Test Case + Basic Regeneration" focusing on template-based document regeneration with diff preview workflow. Convergence Session (2025-11-19): - Defined 5 must-have features for reliable doc regeneration - Created test case template for amplifier/README.md - Deferred 21 features to backlog (updated MASTER_BACKLOG to 57 items) - Captured convergence completion summary Sprint Planning (v0.3.0): - Sprint 8 (4 days): Template parser & manual regen command - Sprint 9 (3 days): Source context & structure preservation - Sprint 10 (3 days): Change detection & user confirmation - Integrated issues microsoft#4, microsoft#5, microsoft#6, microsoft#7 into sprint work Workflow Improvements: - Fixed convergence-architect to create all 4 required outputs - Fixed sprint-planner to use proper versioned directory structure - Added test case template creation to convergence Phase 3 - Added validation checklists to prevent incomplete workflows Test Case: - Created templates/amplifier_readme.json as primary test case - 9 sections covering real-world README structure - Will validate all v0.3.0 features against actual project 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Mollie Munoz (momuno) added a commit
to momuno/amplifier
that referenced
this pull request
Nov 21, 2025
…ed error messages Significantly expand source documentation and improve error messages with actionable guidance. Documentation Changes (TEMPLATES.md): - Add comprehensive "Source Specification" section (280+ lines) - Explain per-section source specification clearly - Complete glob pattern reference with examples - Document source resolution behavior (relative to template) - Add common source patterns for different use cases - Include source file limits and best practices - Expand troubleshooting with specific solutions Error Message Enhancements: - chunked_generator.py: Improve "no prompt" error with fix guidance - source_validator.py: Enhance "no sources found" error with: - Show which patterns were tried - List possible causes (relative paths, missing files, syntax) - Provide actionable fixes with examples - Link to documentation Sprint 9 Day 2 Status: ✅ Complete - Issue microsoft#6 (sources clarity) resolved - 60/60 tests passing - Comprehensive source documentation - Actionable error messages throughout 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement a comprehensive AI-driven smoke test suite while removing unused dependencies and improving test reliability. The changes streamline the testing process, enhance visibility, and ensure consistent evaluation across different environments. Additionally, the configuration for model versions and smoke test settings has been updated for better performance and maintainability.