Uh oh!
There was an error while loading. Please reload this page.
feat(ci): add GitHub Actions integration smoke test - #149
Conversation
Move vllm, flwr, autogen-agentchat, transformers, groq, azure-ai-ml into [tool.poetry.group.clients] so CI can install without CUDA/ML runtime dependencies: poetry install --without clients --with dev Declare packages used by aiopslab/ core modules that were previously only available as undeclared transitive dependencies: docker - aiopslab/service/dock.py pandas - aiopslab/orchestrator/actions/base.py, observer/*.py pytz - aiopslab/observer/__init__.py, metric_api.py requests - aiopslab/observer/trace_api.py, service/telemetry/loki.py pyyaml - aiopslab/config.py, generators/fault/*.py, service/*.py tiktoken - aiopslab/orchestrator/evaluators/quantitative.py Add [tool.poetry.group.dev] with pytest + pytest-asyncio and register the 'integration' pytest marker for the new smoke test.
Adds tests/integration/smoke_test.py — a zero-cost integration test
that exercises the full orchestrator pipeline using a DummyAgent that
makes no LLM or API calls.
Problem: noop_detection_hotel_reservation-1 (lightest in the registry)
Pipeline: deploy HotelReservation → inject no-op fault → run workload
→ DummyAgent calls submit("No") → evaluate → recover → cleanup
The test asserts:
- init_problem() returns non-empty desc, instructions, and actions
- 'submit' is present in available actions
- Detection Accuracy == 'Correct' (no fault injected, answer is 'No')
Run locally after cluster setup:
poetry run pytest tests/integration/smoke_test.py -v -s -m integrationTriggers on every push and pull request to any branch, ensuring regressions in the core orchestrator pipeline are caught before merge. Setup: - Checks out repo with submodules (aiopslab-applications) - Installs kind v0.27.0 (to /tmp to avoid collision with kind/ dir) - Creates two-node kind cluster with kind/kind-config-x86.yaml - Creates /run/udev as a directory so OpenEBS NDM hostPath mount succeeds on runners where it may be absent or a socket file - Pre-installs OpenEBS and Prometheus before pytest to avoid hitting the 5-minute wait_for_ready() hard timeout on cold runners - Installs Python deps excluding CUDA clients (--without clients) - Generates aiopslab/config.yml (k8s_host: kind) - Runs: pytest tests/integration/smoke_test.py -m integration On failure: dumps cluster state and uploads kind logs as an artifact (retention: 7 days).
feat(ci): add GitHub Actions integration smoke testAsh Manda (srmanda-cs)
commented
Feb 26, 2026
Latest run for the PR available at: https://github.com/srmanda-cs/AIOpsLab/actions/runs/22465817762/job/65071423830 |
There was a problem hiding this comment.
Pull request overview
This PR introduces a zero-cost GitHub Actions smoke test that validates the complete orchestrator pipeline without requiring LLM APIs or CUDA dependencies. The test deploys a kind cluster, runs the HotelReservation application with no-op fault injection, and verifies correctness using a lightweight DummyAgent.
Changes:
- Added GitHub Actions workflow with pre-installed OpenEBS/Prometheus to avoid timeout issues during cold image pulls
- Created integration smoke test using DummyAgent against the lightest problem in the registry
- Restructured dependencies to separate heavy ML packages into optional
[clients]group
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
.github/workflows/integration-test.yml | New CI workflow that sets up kind cluster, pre-installs infrastructure, and runs smoke test with failure diagnostics |
tests/integration/smoke_test.py | New smoke test exercising full orchestrator pipeline with DummyAgent requiring no external APIs |
pyproject.toml | Reorganized dependencies: moved ML packages to optional [clients] group, declared previously implicit core deps, added [dev] group |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Uh oh!
There was an error while loading. Please reload this page.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
| # Trigger on every push to any branch and on every pull request. | ||
| # This ensures regressions are caught before merging to main. | ||
| on: | ||
| push: |
There was a problem hiding this comment.
Let us restrict the triggers
on:
push:
branches: [main]
pull_request:
branches: [main]
I think the current trigger is too broad.
Ash Manda (srmanda-cs)
commented
Mar 2, 2026
We might want something like this too: SREGym/SREGym#616 (comment) |
Gagan Somashekar (gaganso)
commented
Mar 3, 2026
Hi Ash Manda (@srmanda-cs), do you mind summarizing the changes in SREGym/SREGym#616 (comment)? |
Jackson Clark (HacksonClark)
commented
Mar 3, 2026
Gagan Somashekar (@gaganso) we added concurrency control to our action over there. That way, if there are lot of commits pushed rapidly it doesn't keep triggering new smoke test runs. |
Gagan Somashekar (gaganso)
commented
Mar 3, 2026
Thanks Jackson Clark (@HacksonClark)! |
Gagan Somashekar (gaganso)
commented
Mar 3, 2026
Ash Manda (@srmanda-cs) , with the restricted triggers, is might not be needed but nice to have. Please go ahead, thanks! |
Ash Manda (srmanda-cs)
commented
Mar 4, 2026
Should be added, please suggest changes if any! |
Uh oh!
There was an error while loading. Please reload this page.
Summary
Closes#19
Adds a GitHub Actions workflow (
.github/workflows/integration-test.yml) that runs ano-cost, end-to-end integration smoke test on every push and pull request.
The test deploys a real kind cluster, runs the full orchestrator pipeline
(HotelReservation app → no-op fault injection → workload → evaluation → teardown),
and asserts correctness — using a
DummyAgentthat requires no LLM or API keys.Changes
.github/workflows/integration-test.ymltests/integration/smoke_test.pyDummyAgentagainstnoop_detection_hotel_reservation-1pyproject.toml[clients]group; declared previously-implicit core deps (docker, pandas, pytz, requests, pyyaml, tiktoken); added[dev]group with pytestpoetry.lockpyproject.tomlBreaking Changes
None. All changes are purely additive. The
[clients]group is installed by default(
poetry install) so existing developer workflows are unaffected. CI usespoetry install --without clients --with devto avoid pulling CUDA/ML runtimes.Notes for Reviewers
Happy to revise anything before it's considered ready!
A few things I'd specifically appreciate input on:
wait_for_ready()has a hardmax_wait=300s. To handle cold Docker Hub pulls on CI runners, the workflow pre-installs OpenEBS and Prometheus before pytest runs. An alternative would be to increasemax_waitor make it configurable — open to whichever approach the team prefers.kind/kind-config-x86.yamldirectly. I initially created a separatekind-config-ci.yamlbut removed it to avoid duplication. Let me know if a CI-specific config is preferred.noop_detection_hotel_reservation-1was chosen as the lightest problem in the registry (no real fault, minimal workload). If there's a more idiomatic choice for a smoke test, I'm happy to change it.pyproject.tomldeps: The implicit-dependency declarations I added (docker,pandas, etc.) were missing from the file but working on developer machines via transitive deps. I can split these into a separate PR if preferred.