🤖 TestAgent - Automated UI Testing for GitHub

TestAgent automatically generates and runs UI tests for your web applications using AI. Simply add it to your GitHub repository and it will test your UI on every pull request!

🚀 Quick Start for GitHub Actions

1. Create Your Test Configuration

Use the TestAgent frontend to generate your test configuration:

🎨 Open TestAgent Configurator

1. Define your application's pages (Login, Dashboard, etc.)
2. Configure test preferences for each page
3. Set up page dependencies and login credentials
4. Download the generated config.yml file

2. Add Configuration to Your Repository

Save the generated config.yml file to your repository root or .github/ directory.

3. Add TestAgent GitHub Action

Create .github/workflows/testagent.yml in your repository:

name: TestAgent UI Testingon:
pull_request:
branches: [ main, master, develop ]jobs:
ui-tests:
runs-on: ubuntu-latestname: Automated UI Testingsteps:
- name: Checkout repositoryuses: actions/checkout@v4
- name: Run TestAgentuses: abalakrishnan1/testagent@mainwith:
url: 'https://your-app-url.com'# Replace with your app URLconfig-file: 'config.yml'# Path to your config filegoogle-api-key: ${{ secrets.GOOGLE_API_KEY }}comment-on-pr: 'true'

4. Add API Key to Repository Secrets

1. Go to your repository → Settings → Secrets and variables → Actions
2. Click "New repository secret"
3. Name: GOOGLE_API_KEY
4. Value: Your Google Gemini API key (Get one here)

5. That's it! 🎉

TestAgent will now run on every pull request and:

• 🔍 Use your configuration to test specific pages and workflows
• 🧪 Generate intelligent test cases based on your UI structure
• 🤖 Execute tests automatically with proper page dependencies
• 🏃‍♂️ Execute tests in a real browser
• 📊 Save test results, screenshots, and detailed reports as GitHub Action artifacts

🎨 Configuration Made Easy

🔗 Open TestAgent Configurator

Our visual configurator makes it easy to set up your tests:

• Page Definition: Define your app's pages (Login, Dashboard, Profile, etc.)
• Test Preferences: Configure what to test on each page (buttons, forms, links)
• Page Dependencies: Set up realistic user flows (login → dashboard → profile)
• Login Credentials: Configure both success and failure scenarios
• Selector Targeting: Specify which elements to prioritize for testing
• Fuzzing Options: Enable input fuzzing for form-heavy pages

The configurator generates a simple YAML file that TestAgent uses to run your tests.

🛠️ Local Development

git clone https://github.com/abalakrishnan1/testagent.git
cd testagent
# Set up development environment
make setup
# Configure your API key
cp .env.example .env
# Edit .env and add your GOOGLE_API_KEY

🚦 Usage

With Configuration File

# Create config using the frontend, then run:
python main.py --config config.yml --url https://your-app.com
# Or test a specific URL without config
python main.py --url https://www.saucedemo.com/
# Use with make for convenience
make run
# Run with Docker
make docker-run

Command Line Options

python main.py [OPTIONS]
Options:
--config PATH Path to configuration YAML file generated by frontend
--url URL URL to test --browser BROWSER Browser to use (chromium, firefox, webkit)
--headless BOOL Run in headless mode (true/false)
--timeout INT Test timeout in seconds
--max-tests INT Maximum number of tests to generate
--output-format STR Output format (console, github-action)

Configuration File Format

While the TestAgent Configurator is the recommended way to create configuration files, here's the expected YAML structure:

# config.ymlLogin:
Buttons: true # Test button interactionsLinks: false # Skip link testing Input_fields: true # Test input fieldspage_fuzzing: false # Skip fuzzing testsselectors: # CSS selectors to prioritize
- "#username"
- "#password"
- "#login-btn"depends_on: [] # Page dependenciesError: # Login failure credentialsUsername: "invalid@example.com"Password: "wrongpassword"Success: # Login success credentials Username: "test@example.com"Password: "validpassword"Dashboard:
Buttons: trueLinks: trueInput_fields: truepage_fuzzing: falseselectors:
- "#navigation"
- ".action-button"depends_on: ["Login"] # Must login first

See config.example.yml for a complete example.

### Advanced Usage
```bash
# Run with performance monitoring
make perf-test
# Run full CI pipeline locally
make ci
# Run specific test types
make test-unit # Unit tests only
make test-integration # Integration tests only
# Code quality checks
make lint # Run linting
make format # Format code
make security # Security scan

🏗 How it works

1. UI Capture (ui_state.py): Captures your app's HTML and takes screenshots
2. Test Planning (planner.py): Uses Gemini with few-shot examples to generate intelligent test cases
3. Test Execution (execute.py): Runs Playwright tests with detailed reporting
4. Result Analysis: Saves screenshots, logs, and generates comprehensive reports

📁 Project Structure

testagent/
├── main.py # Entry point and orchestration
├── containers.py # Container management for test apps
├── ui_state.py # UI capture and screenshot functionality
├── planner.py # LLM-powered test plan generation
├── execute.py # Playwright test execution engine
├── few_shot_examples.py # Prompt engineering examples
├── requirements.txt # Production dependencies
├── requirements-dev.txt # Development dependencies
├── Dockerfile # Multi-stage Docker build
├── docker-compose.yml # Container orchestration
├── Makefile # Development automation
├── pyproject.toml # Python project configuration
├── .github/workflows/ # CI/CD pipelines
│ ├── ci.yml # Main CI/CD pipeline
│ ├── scheduled.yml # Scheduled testing
│ └── dependencies.yml # Dependency updates
├── tests/ # Comprehensive test suite
├── frontend/ # React frontend (configuration UI)
└── results/ # Test outputs and screenshots

🧪 Testing

Running Tests

# Run all tests
make test# Run specific test categories
pytest tests/test_ui_state.py -v # UI capture tests
pytest tests/test_planner.py -v # Test planning tests
pytest tests/test_execute.py -v # Execution tests
pytest tests/test_integration.py -v # End-to-end tests# Run with coverage
make test-unit

Test Categories

• Unit Tests: Individual component testing
• Integration Tests: Full workflow testing
• Security Tests: Vulnerability scanning
• Performance Tests: Memory and execution profiling

🚀 CI/CD Pipeline

The project includes comprehensive GitHub Actions workflows:

Main CI Pipeline (.github/workflows/ci.yml)

• ✅ Python code testing and linting
• ✅ Frontend testing and building
• ✅ Integration testing
• ✅ Security vulnerability scanning
• ✅ Automated deployment to staging/production

Scheduled Testing (.github/workflows/scheduled.yml)

• 🕐 Daily test runs against multiple demo sites
• 📊 Performance monitoring
• 🔔 Slack notifications for failures

Dependency Management (.github/workflows/dependencies.yml)

• 📦 Weekly dependency updates
• 🛡️ Security vulnerability checks
• 🔄 Automated pull requests for updates

📊 Example Output

✅ Running test agent on https://www.saucedemo.com/
✅ UI captured
Got test plan from LLM
Generated tests:
{
"description": "SauceDemo login page tests",
"tests": [
{
"description": "Test successful login with valid credentials",
"steps": [
{"action": "type", "selector": "#user-name", "value": "standard_user"},
{"action": "type", "selector": "#password", "value": "secret_sauce"},
{"action": "click", "selector": "#login-button"}
],
"expect": {
"url": "/inventory.html"
}
},
{
"description": "Test navigation to product page",
"steps": [
{"action": "click", "selector": ".inventory_item_name"}
],
"expect": {
"selectorVisible": ".inventory_details"
}
}
]
}
▶ Running: Test successful login with valid credentials
✅ Passed
▶ Running: Test navigation to product page ✅ Passed
All tests executed successfully

🎯 Roadmap

Current Focus

• ✅ CI/CD Integration
• ✅ Comprehensive testing suite
• ✅ Security scanning
• ✅ Docker containerization

Upcoming Features

• 🔄 Config YAML Integration: Standardized test case definitions
• 🎯 Base Case Coverage: Ensure consistent test coverage
• 🔄 Retry Logic: JSON parsing with LangChain retries
• 🌐 Multi-page Testing: Complex user journey support
• 🎲 Advanced Fuzzing: Input validation and edge case testing
• 📊 Test Coverage Validation: Post-prompt processing
• 🔍 Visual Regression Testing: Screenshot comparison
• 📈 Performance Benchmarking: Load and stress testing

🤝 Contributing

Development Setup

# Clone and setup
git clone https://github.com/abalakrishnan1/testagent.git
cd testagent
make setup
# Make changes and test
make ci
# Submit pull request

Code Quality Standards

• Code Formatting: Black + isort
• Linting: flake8
• Security: Bandit scanning
• Testing: pytest with >90% coverage
• Documentation: Comprehensive docstrings

Pre-commit Hooks

# Setup pre-commit hooks
make setup-hooks
# Manual run
pre-commit run --all-files

📄 License

MIT License - see LICENSE file for details.

🙏 Acknowledgments

• Playwright for browser automation
• Google Gemini for intelligent test generation
• LangChain for LLM orchestration