feat(ci): monitors marker + three-way Better Stack heartbeat split by ari-nz · Pull Request #641 · aignostics/python-sdk

ari-nz · 2026-05-12T08:36:56Z

Summary

Replaces the parameterised monitors(\"name\") marker with two boolean markers that pytest `-m` expressions can filter on directly:
- platform_api — test monitors Platform API layer (auth, listing, connectivity)
- platform_applications — test monitors a platform application (he-tme, test-app)
- Tests with neither marker are SDK-layer health checks
Splits the hourly scheduled workflow into three independent runs, each feeding its own Better Stack heartbeat:
- SDK (`not platform_api and not platform_applications`): token management, service wiring
- Platform API (`platform_api`): health check, application listing, run listing
- Platform Applications (`platform_applications`): he-tme and test-app processing tests
Extracts heartbeat send logic into `.github/workflows/_betterstack_heartbeat.py` (stdlib urllib, no jq dependency)
Heartbeat steps never fail the CI job (errors are logged as warnings)
`if: always()` added to the final "Fail job if any tests failed" step
JUnit artifact upload uses `reports/junit_*.xml` wildcard (nox controls filenames)
New optional secrets: `BETTERSTACK_HEARTBEAT_URL_PLATFORM_API_{STAGING|PRODUCTION}`, `BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_{STAGING|PRODUCTION}`

Action required

Rename GitHub secret (repository settings → Secrets):

BETTERSTACK_HEARTBEAT_URL_HE_TME_STAGING → BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_STAGING
BETTERSTACK_HEARTBEAT_URL_HE_TME_PRODUCTION → BETTERSTACK_HEARTBEAT_URL_PLATFORM_APPLICATIONS_PRODUCTION

Until the secret is renamed the Platform Applications heartbeat will silently skip (step logs a warning).

Test plan

`uv run pytest --collect-only -m "platform_api"` → 3 tests
`uv run pytest --collect-only -m "platform_applications"` → 6 tests
`uv run pytest --collect-only -m "not platform_api and not platform_applications" -m "scheduled or scheduled_only"` → SDK-only scheduled tests
Better Stack heartbeat steps each skip gracefully when URL secrets are absent

Copilot

Pull request overview

This PR introduces a monitors(...) pytest marker intended to tag scheduled E2E tests by the Better Stack monitor they should feed, and updates the hourly scheduled CI workflow to split test execution and emit separate heartbeats (SDK vs. application-specific).

Changes:

Added @pytest.mark.monitors("he-tme" | "test-app") to selected platform E2E scheduled tests.
Registered the new monitors marker in pyproject.toml pytest configuration.
Split the hourly scheduled workflow into multiple pytest invocations and added a dedicated Better Stack heartbeat URL secret for HE-TME (propagated via the staging/production wrapper workflows).

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`tests/aignostics/platform/e2e_test.py`	Tags scheduled E2E tests with `monitors(...)` to support routing results/heartbeats by monitored system.
`pyproject.toml`	Registers the new pytest marker so `--strict-markers` runs remain valid.
`.github/workflows/_scheduled-test-hourly.yml`	Splits scheduled tests into separate runs and sends separate Better Stack heartbeats (SDK + HE-TME), plus combined status to Sentry.
`.github/workflows/scheduled-testing-staging-hourly.yml`	Passes through the new HE-TME Better Stack heartbeat secret to the reusable workflow.
`.github/workflows/scheduled-testing-production-hourly.yml`	Passes through the new HE-TME Better Stack heartbeat secret to the reusable workflow.

+ # set +e so a test failure does not abort the step — we capture the exit code
+ # manually and send it to Better Stack regardless of outcome.
 set +e
- make test_scheduled
- EXIT_CODE=$?
+ XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+ -m "(scheduled or scheduled_only) and monitors and not stress_only" \
+ --junit-xml=reports/junit_he_tme.xml


+ - name: Test / scheduled / he-tme
+ id: test_he_tme
 env:
- BETTERSTACK_HEARTBEAT_URL: "${{ inputs.platform_environment == 'staging' && secrets.BETTERSTACK_HEARTBEAT_URL_STAGING || secrets.BETTERSTACK_HEARTBEAT_URL_PRODUCTION }}"
 SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
 shell: bash
 run: |
+ # set +e so a test failure does not abort the step — we capture the exit code
+ # manually and send it to Better Stack regardless of outcome.
 set +e
- make test_scheduled
- EXIT_CODE=$?
+ XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+ -m "(scheduled or scheduled_only) and monitors and not stress_only" \
+ --junit-xml=reports/junit_he_tme.xml
+ echo "exit_code=$?" >> $GITHUB_OUTPUT


 "unit: Solitary unit tests - test a layer of a module in isolation with all dependencies mocked, except interaction with shared utils and the systems module. Unit tests must be able to pass offline, i.e. not calls to external services. The timeout should not be bigger than the default 10s, and must be <5 min.",
 "integration: Sociable integration tests - test interactions across architectural layers (e.g. CLI/GUI→Service, Service→Utils) or between modules (e.g. Application→Platform), using real SDK collaborators, real file I/O, real subprocesses, and real Docker containers. Integration test must be able to pass offline, i.e. mock external services (Aignostics Platform API, Auth0, S3/GCS buckets, IDC). The timeout should not be bigger than the default 10s, and must be <5 min.",
 "e2e: End-to-end tests - test complete workflows with real external network services (Aignostics Platform API, cloud storage, IDC, etc). If the test timeout is >= 5 min and < 60 min, additionally mark as `long_running`, if >= 60min mark as 'very_long_running'.",
+ "monitors: Tag a scheduled test with the application it monitors, e.g. @pytest.mark.monitors('he-tme'). Tests without this marker are considered SDK-layer health checks. Used to route Better Stack heartbeats to the correct monitor.",


+ reports/junit_sdk.xml
+ reports/junit_he_tme.xml


codecov · 2026-05-12T09:13:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.
see 2 files with indirect coverage changes

Copilot

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

+ - name: Test / scheduled / he-tme
+ id: test_he_tme
+ env:
+ SENTRY_DSN: ${{ secrets.SENTRY_DSN }}
+ shell: bash
+ run: |
+ # set +e so a test failure does not abort the step — we capture the exit code
+ # manually and send it to Better Stack regardless of outcome.
+ set +e
+ XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+ -m "(scheduled or scheduled_only) and monitors and not monitors_platform_api and not stress_only" \
+ --junit-xml=reports/junit_he_tme.xml


+ --request POST \
+ --header "Content-Type: application/json" \
+ --data-binary "${BETTERSTACK_METADATA_PAYLOAD}" \
+ "${BETTERSTACK_HEARTBEAT_URL}/${SDK_EXIT}"


+ curl \
+ --fail-with-body \
+ --silent \
+ --request POST \
+ --header "Content-Type: application/json" \
+ --data-binary "${BETTERSTACK_METADATA_PAYLOAD}" \
+ "${BETTERSTACK_HEARTBEAT_URL_PLATFORM_API}/${PLATFORM_API_EXIT}"
+ echo "INFO: Sent Platform API heartbeat to BetterStack with exit code '${PLATFORM_API_EXIT}'"


+ curl \
+ --fail-with-body \
+ --silent \
+ --request POST \
+ --header "Content-Type: application/json" \
+ --data-binary "${BETTERSTACK_METADATA_PAYLOAD}" \
+ "${BETTERSTACK_HEARTBEAT_URL_HE_TME}/${HE_TME_EXIT}"
+ echo "INFO: Sent HE-TME heartbeat to BetterStack with exit code '${HE_TME_EXIT}'"


 retention-days: 7
+
+ - name: Fail job if any tests failed
+ shell: bash


+ XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+ -m "(scheduled or scheduled_only) and not monitors and not monitors_platform_api and not stress_only" \
+ --junit-xml=reports/junit_sdk.xml
+ echo "exit_code=$?" >> $GITHUB_OUTPUT


+ XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+ -m "(scheduled or scheduled_only) and monitors_platform_api and not stress_only" \
+ --junit-xml=reports/junit_platform_api.xml
+ echo "exit_code=$?" >> $GITHUB_OUTPUT


+ XDIST_WORKER_FACTOR=1 uv run --all-extras nox -s test -- \
+ -m "(scheduled or scheduled_only) and monitors and not monitors_platform_api and not stress_only" \
+ --junit-xml=reports/junit_he_tme.xml
+ echo "exit_code=$?" >> $GITHUB_OUTPUT


+ reports/junit_sdk.xml
+ reports/junit_platform_api.xml
+ reports/junit_he_tme.xml


Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.

…I / applications

sonarqubecloud · 2026-05-19T17:58:47Z

Quality Gate failed

Failed conditions
E Security Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

CopilotAI review requested due to automatic review settings May 12, 2026 08:36

ari-nz requested review from a team and helmut-hoffer-von-ankershoffen as code owners May 12, 2026 08:36

ari-nz added the skip:test:long_running Skip long-running tests (≥5min) label May 12, 2026

Copilotstarted reviewing on behalf of ari-nz May 12, 2026 08:37 View session

CopilotAI reviewed May 12, 2026
View reviewed changes

ari-nz changed the title ~~feat(ci): split hourly heartbeat by concern and add monitors marker~~feat(ci): monitors marker + three-way Better Stack heartbeat splitMay 12, 2026

ari-nz force-pushed the feat/monitors-marker-split branch from 129c9aa to 0683245Compare May 12, 2026 13:41

ari-nz removed the skip:test:long_running Skip long-running tests (≥5min) label May 19, 2026

ari-nz marked this pull request as draft May 19, 2026 11:07

CopilotAI review requested due to automatic review settings May 19, 2026 11:15

Copilotstarted reviewing on behalf of ari-nz May 19, 2026 11:15 View session

CopilotAI reviewed May 19, 2026
View reviewed changes

ari-nz force-pushed the feat/monitors-marker-split branch from eaa4b76 to 15c7e5dCompare May 19, 2026 14:18

CopilotAI review requested due to automatic review settings May 19, 2026 17:29

ari-nz force-pushed the feat/monitors-marker-split branch from 15c7e5d to 0fdfed4Compare May 19, 2026 17:29

Copilotstarted reviewing on behalf of ari-nz May 19, 2026 17:30 View session

CopilotAI reviewed May 19, 2026
View reviewed changes

Comment thread.github/CLAUDE.md
Comment thread.github/CLAUDE.md Outdated

feat(ci): split Better Stack heartbeat by concern — SDK / Platform AP…
39f8b21
…I / applications

ari-nz force-pushed the feat/monitors-marker-split branch from 0fdfed4 to 39f8b21Compare May 19, 2026 17:56

Navigation Menu

Uh oh!

feat(ci): monitors marker + three-way Better Stack heartbeat split - #641

feat(ci): monitors marker + three-way Better Stack heartbeat split#641
ari-nz wants to merge 1 commit into
mainfrom
feat/monitors-marker-split

ari-nz commented May 12, 2026•
edited
Loading

Uh oh!

CopilotAI left a comment

Uh oh!

codecovBot commented May 12, 2026•
edited
Loading

Uh oh!

CopilotAI left a comment

Uh oh!

CopilotAI left a comment

Uh oh!

Uh oh!

Uh oh!

sonarqubecloudBot commented May 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ari-nz commented May 12, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Action required

Test plan

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

codecovBot commented May 12, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

CopilotAI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

sonarqubecloudBot commented May 19, 2026

Quality Gate failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ari-nz commented May 12, 2026•
edited
Loading

codecovBot commented May 12, 2026•
edited
Loading