fix: auto-skip real_api-marked tests by default by jrob5756 · Pull Request #328 · microsoft/conductor

Jason Robert (jrob5756) · 2026-07-21T18:12:19Z

Summary

Fixes#326.

@pytest.mark.real_api was registered in pyproject.toml but nothing
deselected it by default — only CI/release workflows hand-filtered with
-m "not real_api and not performance". A plain pytest,
pytest -m "not performance", or make test would run those tests,
spawning real copilot/claude subprocesses. That's more than a hygiene
issue: it can hard-kill a live conductor run --web-bg workflow when run
from inside it (unhandled SIGTERM, no workflow_failed event, PID file
left behind).

Changes

tests/conftest.py: add a pytest_collection_modifyitems hook that
skips any real_api-marked item unless the caller's -m expression
already references real_api (word-boundary match) — covering both
explicit opt-in (-m real_api) and CI's existing
-m "not real_api and not performance", which keep working unchanged
since pytest's own marker-expression evaluation handles those cases.
tests/test_config/test_real_api_marker.py: new pytester-based
regression suite that loads the actual hook by file path (not a copy)
and verifies:
- default run (no -m) skips the real_api test
- -m "not performance" skips it (the issue's exact repro)
- -m real_api runs it
- CI's exact -m "not real_api and not performance" still deselects it
- a reverted/no-op hook would not skip it (load-bearing check, so
  this suite would catch a future regression)

No changes needed to Makefile/CI/release workflows — their existing -m
expressions are unaffected by the hook.

Validation

make lint / make typecheck — clean (pre-existing unrelated ty
warning in dialog_evaluator.py only).
uv run pytest tests/test_config/test_real_api_marker.py -v — 5/5 pass.
uv run pytest tests/test_integration/test_copilot_large_write.py tests/test_integration/test_claude_real_api.py tests/test_config/test_ci_infrastructure.py -v (no -m) — all
real_api tests report skipped, proving no subprocess spawn.
uv run pytest tests/test_config/test_ci_infrastructure.py -m real_api -v
— confirms opt-in still runs them.
Full suite uv run pytest -q -m "not performance" —
4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed
from a real session.create call per the issue repro).

🤖 Generated with Copilot CLI

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Add a pytest_collection_modifyitems hook to tests/conftest.py that skips any @pytest.mark.real_api test unless the caller's -m expression already references real_api (e.g. `-m real_api` to opt in, or CI's exact `-m "not real_api and not performance"`), in which case pytest's own marker-expression filtering is left to decide selection/deselection. Previously nothing deselected real_api tests by default: a plain pytest, `pytest -m "not performance"`, or `make test` would run them, spawning real copilot/claude subprocesses that can collide with (and kill) a live `conductor run --web-bg` session. Adds tests/test_config/test_real_api_marker.py, a pytester-based regression suite that exercises the actual hook (loaded by file path) against a synthetic real_api-marked test, verifying: - default run (no -m) skips it - -m "not performance" skips it (the issue's exact repro) - -m real_api runs it - CI's exact -m "not real_api and not performance" still deselects it - a reverted/no-op hook would NOT skip it (load-bearing check) Verified full suite: 4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed from a real session.create call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Update tests/conftest.py module docstring to mention the new pytest_collection_modifyitems hook instead of only "fixtures". - Use config.getoption("markexpr") instead of the "-m" option-string alias, and drop the redundant `or ""` fallback (pytest defaults -m to "" already). - De-duplicate the issue #326 rationale between the hook's docstring and the test module's docstring; the test module now points to the hook as the source of truth. - Note in test_ci_marker_expression_still_deselects that its -m string is duplicated in ci.yml/release.yml and should be kept in sync. - Add test_regex_does_not_match_unrelated_marker_name to directly verify the \breal_api\b word-boundary regex doesn't false-positive on an unrelated marker name that merely contains "real_api" as a substring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jason Robert (jrob5756) · 2026-07-21T18:12:19Z

Summary

Fixes#326.

@pytest.mark.real_api was registered in pyproject.toml but nothing
deselected it by default — only CI/release workflows hand-filtered with
-m "not real_api and not performance". A plain pytest,
pytest -m "not performance", or make test would run those tests,
spawning real copilot/claude subprocesses. That's more than a hygiene
issue: it can hard-kill a live conductor run --web-bg workflow when run
from inside it (unhandled SIGTERM, no workflow_failed event, PID file
left behind).

Changes

tests/conftest.py: add a pytest_collection_modifyitems hook that
skips any real_api-marked item unless the caller's -m expression
already references real_api (word-boundary match) — covering both
explicit opt-in (-m real_api) and CI's existing
-m "not real_api and not performance", which keep working unchanged
since pytest's own marker-expression evaluation handles those cases.
tests/test_config/test_real_api_marker.py: new pytester-based
regression suite that loads the actual hook by file path (not a copy)
and verifies:
- default run (no -m) skips the real_api test
- -m "not performance" skips it (the issue's exact repro)
- -m real_api runs it
- CI's exact -m "not real_api and not performance" still deselects it
- a reverted/no-op hook would not skip it (load-bearing check, so
  this suite would catch a future regression)

No changes needed to Makefile/CI/release workflows — their existing -m
expressions are unaffected by the hook.

Validation

make lint / make typecheck — clean (pre-existing unrelated ty
warning in dialog_evaluator.py only).
uv run pytest tests/test_config/test_real_api_marker.py -v — 5/5 pass.
uv run pytest tests/test_integration/test_copilot_large_write.py tests/test_integration/test_claude_real_api.py tests/test_config/test_ci_infrastructure.py -v (no -m) — all
real_api tests report skipped, proving no subprocess spawn.
uv run pytest tests/test_config/test_ci_infrastructure.py -m real_api -v
— confirms opt-in still runs them.
Full suite uv run pytest -q -m "not performance" —
4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed
from a real session.create call per the issue repro).

🤖 Generated with Copilot CLI

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Add a pytest_collection_modifyitems hook to tests/conftest.py that skips any @pytest.mark.real_api test unless the caller's -m expression already references real_api (e.g. `-m real_api` to opt in, or CI's exact `-m "not real_api and not performance"`), in which case pytest's own marker-expression filtering is left to decide selection/deselection. Previously nothing deselected real_api tests by default: a plain pytest, `pytest -m "not performance"`, or `make test` would run them, spawning real copilot/claude subprocesses that can collide with (and kill) a live `conductor run --web-bg` session. Adds tests/test_config/test_real_api_marker.py, a pytester-based regression suite that exercises the actual hook (loaded by file path) against a synthetic real_api-marked test, verifying: - default run (no -m) skips it - -m "not performance" skips it (the issue's exact repro) - -m real_api runs it - CI's exact -m "not real_api and not performance" still deselects it - a reverted/no-op hook would NOT skip it (load-bearing check) Verified full suite: 4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed from a real session.create call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Update tests/conftest.py module docstring to mention the new pytest_collection_modifyitems hook instead of only "fixtures". - Use config.getoption("markexpr") instead of the "-m" option-string alias, and drop the redundant `or ""` fallback (pytest defaults -m to "" already). - De-duplicate the issue #326 rationale between the hook's docstring and the test module's docstring; the test module now points to the hook as the source of truth. - Note in test_ci_marker_expression_still_deselects that its -m string is duplicated in ci.yml/release.yml and should be kept in sync. - Add test_regex_does_not_match_unrelated_marker_name to directly verify the \breal_api\b word-boundary regex doesn't false-positive on an unrelated marker name that merely contains "real_api" as a substring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jason Robert (jrob5756) · 2026-07-21T18:12:19Z

Summary

Fixes#326.

@pytest.mark.real_api was registered in pyproject.toml but nothing
deselected it by default — only CI/release workflows hand-filtered with
-m "not real_api and not performance". A plain pytest,
pytest -m "not performance", or make test would run those tests,
spawning real copilot/claude subprocesses. That's more than a hygiene
issue: it can hard-kill a live conductor run --web-bg workflow when run
from inside it (unhandled SIGTERM, no workflow_failed event, PID file
left behind).

Changes

tests/conftest.py: add a pytest_collection_modifyitems hook that
skips any real_api-marked item unless the caller's -m expression
already references real_api (word-boundary match) — covering both
explicit opt-in (-m real_api) and CI's existing
-m "not real_api and not performance", which keep working unchanged
since pytest's own marker-expression evaluation handles those cases.
tests/test_config/test_real_api_marker.py: new pytester-based
regression suite that loads the actual hook by file path (not a copy)
and verifies:
- default run (no -m) skips the real_api test
- -m "not performance" skips it (the issue's exact repro)
- -m real_api runs it
- CI's exact -m "not real_api and not performance" still deselects it
- a reverted/no-op hook would not skip it (load-bearing check, so
  this suite would catch a future regression)

No changes needed to Makefile/CI/release workflows — their existing -m
expressions are unaffected by the hook.

Validation

make lint / make typecheck — clean (pre-existing unrelated ty
warning in dialog_evaluator.py only).
uv run pytest tests/test_config/test_real_api_marker.py -v — 5/5 pass.
uv run pytest tests/test_integration/test_copilot_large_write.py tests/test_integration/test_claude_real_api.py tests/test_config/test_ci_infrastructure.py -v (no -m) — all
real_api tests report skipped, proving no subprocess spawn.
uv run pytest tests/test_config/test_ci_infrastructure.py -m real_api -v
— confirms opt-in still runs them.
Full suite uv run pytest -q -m "not performance" —
4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed
from a real session.create call per the issue repro).

🤖 Generated with Copilot CLI

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Add a pytest_collection_modifyitems hook to tests/conftest.py that skips any @pytest.mark.real_api test unless the caller's -m expression already references real_api (e.g. `-m real_api` to opt in, or CI's exact `-m "not real_api and not performance"`), in which case pytest's own marker-expression filtering is left to decide selection/deselection. Previously nothing deselected real_api tests by default: a plain pytest, `pytest -m "not performance"`, or `make test` would run them, spawning real copilot/claude subprocesses that can collide with (and kill) a live `conductor run --web-bg` session. Adds tests/test_config/test_real_api_marker.py, a pytester-based regression suite that exercises the actual hook (loaded by file path) against a synthetic real_api-marked test, verifying: - default run (no -m) skips it - -m "not performance" skips it (the issue's exact repro) - -m real_api runs it - CI's exact -m "not real_api and not performance" still deselects it - a reverted/no-op hook would NOT skip it (load-bearing check) Verified full suite: 4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed from a real session.create call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Update tests/conftest.py module docstring to mention the new pytest_collection_modifyitems hook instead of only "fixtures". - Use config.getoption("markexpr") instead of the "-m" option-string alias, and drop the redundant `or ""` fallback (pytest defaults -m to "" already). - De-duplicate the issue #326 rationale between the hook's docstring and the test module's docstring; the test module now points to the hook as the source of truth. - Note in test_ci_marker_expression_still_deselects that its -m string is duplicated in ci.yml/release.yml and should be kept in sync. - Add test_regex_does_not_match_unrelated_marker_name to directly verify the \breal_api\b word-boundary regex doesn't false-positive on an unrelated marker name that merely contains "real_api" as a substring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jason Robert (jrob5756) · 2026-07-21T18:12:19Z

Summary

Fixes#326.

@pytest.mark.real_api was registered in pyproject.toml but nothing
deselected it by default — only CI/release workflows hand-filtered with
-m "not real_api and not performance". A plain pytest,
pytest -m "not performance", or make test would run those tests,
spawning real copilot/claude subprocesses. That's more than a hygiene
issue: it can hard-kill a live conductor run --web-bg workflow when run
from inside it (unhandled SIGTERM, no workflow_failed event, PID file
left behind).

Changes

tests/conftest.py: add a pytest_collection_modifyitems hook that
skips any real_api-marked item unless the caller's -m expression
already references real_api (word-boundary match) — covering both
explicit opt-in (-m real_api) and CI's existing
-m "not real_api and not performance", which keep working unchanged
since pytest's own marker-expression evaluation handles those cases.
tests/test_config/test_real_api_marker.py: new pytester-based
regression suite that loads the actual hook by file path (not a copy)
and verifies:
- default run (no -m) skips the real_api test
- -m "not performance" skips it (the issue's exact repro)
- -m real_api runs it
- CI's exact -m "not real_api and not performance" still deselects it
- a reverted/no-op hook would not skip it (load-bearing check, so
  this suite would catch a future regression)

No changes needed to Makefile/CI/release workflows — their existing -m
expressions are unaffected by the hook.

Validation

make lint / make typecheck — clean (pre-existing unrelated ty
warning in dialog_evaluator.py only).
uv run pytest tests/test_config/test_real_api_marker.py -v — 5/5 pass.
uv run pytest tests/test_integration/test_copilot_large_write.py tests/test_integration/test_claude_real_api.py tests/test_config/test_ci_infrastructure.py -v (no -m) — all
real_api tests report skipped, proving no subprocess spawn.
uv run pytest tests/test_config/test_ci_infrastructure.py -m real_api -v
— confirms opt-in still runs them.
Full suite uv run pytest -q -m "not performance" —
4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed
from a real session.create call per the issue repro).

🤖 Generated with Copilot CLI

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Add a pytest_collection_modifyitems hook to tests/conftest.py that skips any @pytest.mark.real_api test unless the caller's -m expression already references real_api (e.g. `-m real_api` to opt in, or CI's exact `-m "not real_api and not performance"`), in which case pytest's own marker-expression filtering is left to decide selection/deselection. Previously nothing deselected real_api tests by default: a plain pytest, `pytest -m "not performance"`, or `make test` would run them, spawning real copilot/claude subprocesses that can collide with (and kill) a live `conductor run --web-bg` session. Adds tests/test_config/test_real_api_marker.py, a pytester-based regression suite that exercises the actual hook (loaded by file path) against a synthetic real_api-marked test, verifying: - default run (no -m) skips it - -m "not performance" skips it (the issue's exact repro) - -m real_api runs it - CI's exact -m "not real_api and not performance" still deselects it - a reverted/no-op hook would NOT skip it (load-bearing check) Verified full suite: 4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed from a real session.create call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Update tests/conftest.py module docstring to mention the new pytest_collection_modifyitems hook instead of only "fixtures". - Use config.getoption("markexpr") instead of the "-m" option-string alias, and drop the redundant `or ""` fallback (pytest defaults -m to "" already). - De-duplicate the issue #326 rationale between the hook's docstring and the test module's docstring; the test module now points to the hook as the source of truth. - Note in test_ci_marker_expression_still_deselects that its -m string is duplicated in ci.yml/release.yml and should be kept in sync. - Add test_regex_does_not_match_unrelated_marker_name to directly verify the \breal_api\b word-boundary regex doesn't false-positive on an unrelated marker name that merely contains "real_api" as a substring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jason Robert (jrob5756) · 2026-07-21T18:12:19Z

Summary

Fixes#326.

@pytest.mark.real_api was registered in pyproject.toml but nothing
deselected it by default — only CI/release workflows hand-filtered with
-m "not real_api and not performance". A plain pytest,
pytest -m "not performance", or make test would run those tests,
spawning real copilot/claude subprocesses. That's more than a hygiene
issue: it can hard-kill a live conductor run --web-bg workflow when run
from inside it (unhandled SIGTERM, no workflow_failed event, PID file
left behind).

Changes

tests/conftest.py: add a pytest_collection_modifyitems hook that
skips any real_api-marked item unless the caller's -m expression
already references real_api (word-boundary match) — covering both
explicit opt-in (-m real_api) and CI's existing
-m "not real_api and not performance", which keep working unchanged
since pytest's own marker-expression evaluation handles those cases.
tests/test_config/test_real_api_marker.py: new pytester-based
regression suite that loads the actual hook by file path (not a copy)
and verifies:
- default run (no -m) skips the real_api test
- -m "not performance" skips it (the issue's exact repro)
- -m real_api runs it
- CI's exact -m "not real_api and not performance" still deselects it
- a reverted/no-op hook would not skip it (load-bearing check, so
  this suite would catch a future regression)

No changes needed to Makefile/CI/release workflows — their existing -m
expressions are unaffected by the hook.

Validation

make lint / make typecheck — clean (pre-existing unrelated ty
warning in dialog_evaluator.py only).
uv run pytest tests/test_config/test_real_api_marker.py -v — 5/5 pass.
uv run pytest tests/test_integration/test_copilot_large_write.py tests/test_integration/test_claude_real_api.py tests/test_config/test_ci_infrastructure.py -v (no -m) — all
real_api tests report skipped, proving no subprocess spawn.
uv run pytest tests/test_config/test_ci_infrastructure.py -m real_api -v
— confirms opt-in still runs them.
Full suite uv run pytest -q -m "not performance" —
4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed
from a real session.create call per the issue repro).

🤖 Generated with Copilot CLI

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Add a pytest_collection_modifyitems hook to tests/conftest.py that skips any @pytest.mark.real_api test unless the caller's -m expression already references real_api (e.g. `-m real_api` to opt in, or CI's exact `-m "not real_api and not performance"`), in which case pytest's own marker-expression filtering is left to decide selection/deselection. Previously nothing deselected real_api tests by default: a plain pytest, `pytest -m "not performance"`, or `make test` would run them, spawning real copilot/claude subprocesses that can collide with (and kill) a live `conductor run --web-bg` session. Adds tests/test_config/test_real_api_marker.py, a pytester-based regression suite that exercises the actual hook (loaded by file path) against a synthetic real_api-marked test, verifying: - default run (no -m) skips it - -m "not performance" skips it (the issue's exact repro) - -m real_api runs it - CI's exact -m "not real_api and not performance" still deselects it - a reverted/no-op hook would NOT skip it (load-bearing check) Verified full suite: 4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed from a real session.create call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Update tests/conftest.py module docstring to mention the new pytest_collection_modifyitems hook instead of only "fixtures". - Use config.getoption("markexpr") instead of the "-m" option-string alias, and drop the redundant `or ""` fallback (pytest defaults -m to "" already). - De-duplicate the issue #326 rationale between the hook's docstring and the test module's docstring; the test module now points to the hook as the source of truth. - Note in test_ci_marker_expression_still_deselects that its -m string is duplicated in ci.yml/release.yml and should be kept in sync. - Add test_regex_does_not_match_unrelated_marker_name to directly verify the \breal_api\b word-boundary regex doesn't false-positive on an unrelated marker name that merely contains "real_api" as a substring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jason Robert (jrob5756) · 2026-07-21T18:12:19Z

Summary

Fixes#326.

@pytest.mark.real_api was registered in pyproject.toml but nothing
deselected it by default — only CI/release workflows hand-filtered with
-m "not real_api and not performance". A plain pytest,
pytest -m "not performance", or make test would run those tests,
spawning real copilot/claude subprocesses. That's more than a hygiene
issue: it can hard-kill a live conductor run --web-bg workflow when run
from inside it (unhandled SIGTERM, no workflow_failed event, PID file
left behind).

Changes

tests/conftest.py: add a pytest_collection_modifyitems hook that
skips any real_api-marked item unless the caller's -m expression
already references real_api (word-boundary match) — covering both
explicit opt-in (-m real_api) and CI's existing
-m "not real_api and not performance", which keep working unchanged
since pytest's own marker-expression evaluation handles those cases.
tests/test_config/test_real_api_marker.py: new pytester-based
regression suite that loads the actual hook by file path (not a copy)
and verifies:
- default run (no -m) skips the real_api test
- -m "not performance" skips it (the issue's exact repro)
- -m real_api runs it
- CI's exact -m "not real_api and not performance" still deselects it
- a reverted/no-op hook would not skip it (load-bearing check, so
  this suite would catch a future regression)

No changes needed to Makefile/CI/release workflows — their existing -m
expressions are unaffected by the hook.

Validation

make lint / make typecheck — clean (pre-existing unrelated ty
warning in dialog_evaluator.py only).
uv run pytest tests/test_config/test_real_api_marker.py -v — 5/5 pass.
uv run pytest tests/test_integration/test_copilot_large_write.py tests/test_integration/test_claude_real_api.py tests/test_config/test_ci_infrastructure.py -v (no -m) — all
real_api tests report skipped, proving no subprocess spawn.
uv run pytest tests/test_config/test_ci_infrastructure.py -m real_api -v
— confirms opt-in still runs them.
Full suite uv run pytest -q -m "not performance" —
4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed
from a real session.create call per the issue repro).

🤖 Generated with Copilot CLI

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Add a pytest_collection_modifyitems hook to tests/conftest.py that skips any @pytest.mark.real_api test unless the caller's -m expression already references real_api (e.g. `-m real_api` to opt in, or CI's exact `-m "not real_api and not performance"`), in which case pytest's own marker-expression filtering is left to decide selection/deselection. Previously nothing deselected real_api tests by default: a plain pytest, `pytest -m "not performance"`, or `make test` would run them, spawning real copilot/claude subprocesses that can collide with (and kill) a live `conductor run --web-bg` session. Adds tests/test_config/test_real_api_marker.py, a pytester-based regression suite that exercises the actual hook (loaded by file path) against a synthetic real_api-marked test, verifying: - default run (no -m) skips it - -m "not performance" skips it (the issue's exact repro) - -m real_api runs it - CI's exact -m "not real_api and not performance" still deselects it - a reverted/no-op hook would NOT skip it (load-bearing check) Verified full suite: 4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed from a real session.create call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Update tests/conftest.py module docstring to mention the new pytest_collection_modifyitems hook instead of only "fixtures". - Use config.getoption("markexpr") instead of the "-m" option-string alias, and drop the redundant `or ""` fallback (pytest defaults -m to "" already). - De-duplicate the issue #326 rationale between the hook's docstring and the test module's docstring; the test module now points to the hook as the source of truth. - Note in test_ci_marker_expression_still_deselects that its -m string is duplicated in ci.yml/release.yml and should be kept in sync. - Add test_regex_does_not_match_unrelated_marker_name to directly verify the \breal_api\b word-boundary regex doesn't false-positive on an unrelated marker name that merely contains "real_api" as a substring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jason Robert (jrob5756) · 2026-07-21T18:12:19Z

Summary

Fixes#326.

@pytest.mark.real_api was registered in pyproject.toml but nothing
deselected it by default — only CI/release workflows hand-filtered with
-m "not real_api and not performance". A plain pytest,
pytest -m "not performance", or make test would run those tests,
spawning real copilot/claude subprocesses. That's more than a hygiene
issue: it can hard-kill a live conductor run --web-bg workflow when run
from inside it (unhandled SIGTERM, no workflow_failed event, PID file
left behind).

Changes

tests/conftest.py: add a pytest_collection_modifyitems hook that
skips any real_api-marked item unless the caller's -m expression
already references real_api (word-boundary match) — covering both
explicit opt-in (-m real_api) and CI's existing
-m "not real_api and not performance", which keep working unchanged
since pytest's own marker-expression evaluation handles those cases.
tests/test_config/test_real_api_marker.py: new pytester-based
regression suite that loads the actual hook by file path (not a copy)
and verifies:
- default run (no -m) skips the real_api test
- -m "not performance" skips it (the issue's exact repro)
- -m real_api runs it
- CI's exact -m "not real_api and not performance" still deselects it
- a reverted/no-op hook would not skip it (load-bearing check, so
  this suite would catch a future regression)

No changes needed to Makefile/CI/release workflows — their existing -m
expressions are unaffected by the hook.

Validation

make lint / make typecheck — clean (pre-existing unrelated ty
warning in dialog_evaluator.py only).
uv run pytest tests/test_config/test_real_api_marker.py -v — 5/5 pass.
uv run pytest tests/test_integration/test_copilot_large_write.py tests/test_integration/test_claude_real_api.py tests/test_config/test_ci_infrastructure.py -v (no -m) — all
real_api tests report skipped, proving no subprocess spawn.
uv run pytest tests/test_config/test_ci_infrastructure.py -m real_api -v
— confirms opt-in still runs them.
Full suite uv run pytest -q -m "not performance" —
4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed
from a real session.create call per the issue repro).

🤖 Generated with Copilot CLI

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Add a pytest_collection_modifyitems hook to tests/conftest.py that skips any @pytest.mark.real_api test unless the caller's -m expression already references real_api (e.g. `-m real_api` to opt in, or CI's exact `-m "not real_api and not performance"`), in which case pytest's own marker-expression filtering is left to decide selection/deselection. Previously nothing deselected real_api tests by default: a plain pytest, `pytest -m "not performance"`, or `make test` would run them, spawning real copilot/claude subprocesses that can collide with (and kill) a live `conductor run --web-bg` session. Adds tests/test_config/test_real_api_marker.py, a pytester-based regression suite that exercises the actual hook (loaded by file path) against a synthetic real_api-marked test, verifying: - default run (no -m) skips it - -m "not performance" skips it (the issue's exact repro) - -m real_api runs it - CI's exact -m "not real_api and not performance" still deselects it - a reverted/no-op hook would NOT skip it (load-bearing check) Verified full suite: 4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed from a real session.create call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Update tests/conftest.py module docstring to mention the new pytest_collection_modifyitems hook instead of only "fixtures". - Use config.getoption("markexpr") instead of the "-m" option-string alias, and drop the redundant `or ""` fallback (pytest defaults -m to "" already). - De-duplicate the issue #326 rationale between the hook's docstring and the test module's docstring; the test module now points to the hook as the source of truth. - Note in test_ci_marker_expression_still_deselects that its -m string is duplicated in ci.yml/release.yml and should be kept in sync. - Add test_regex_does_not_match_unrelated_marker_name to directly verify the \breal_api\b word-boundary regex doesn't false-positive on an unrelated marker name that merely contains "real_api" as a substring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jason Robert (jrob5756) · 2026-07-21T18:12:19Z

Summary

Fixes#326.

@pytest.mark.real_api was registered in pyproject.toml but nothing
deselected it by default — only CI/release workflows hand-filtered with
-m "not real_api and not performance". A plain pytest,
pytest -m "not performance", or make test would run those tests,
spawning real copilot/claude subprocesses. That's more than a hygiene
issue: it can hard-kill a live conductor run --web-bg workflow when run
from inside it (unhandled SIGTERM, no workflow_failed event, PID file
left behind).

Changes

tests/conftest.py: add a pytest_collection_modifyitems hook that
skips any real_api-marked item unless the caller's -m expression
already references real_api (word-boundary match) — covering both
explicit opt-in (-m real_api) and CI's existing
-m "not real_api and not performance", which keep working unchanged
since pytest's own marker-expression evaluation handles those cases.
tests/test_config/test_real_api_marker.py: new pytester-based
regression suite that loads the actual hook by file path (not a copy)
and verifies:
- default run (no -m) skips the real_api test
- -m "not performance" skips it (the issue's exact repro)
- -m real_api runs it
- CI's exact -m "not real_api and not performance" still deselects it
- a reverted/no-op hook would not skip it (load-bearing check, so
  this suite would catch a future regression)

No changes needed to Makefile/CI/release workflows — their existing -m
expressions are unaffected by the hook.

Validation

make lint / make typecheck — clean (pre-existing unrelated ty
warning in dialog_evaluator.py only).
uv run pytest tests/test_config/test_real_api_marker.py -v — 5/5 pass.
uv run pytest tests/test_integration/test_copilot_large_write.py tests/test_integration/test_claude_real_api.py tests/test_config/test_ci_infrastructure.py -v (no -m) — all
real_api tests report skipped, proving no subprocess spawn.
uv run pytest tests/test_config/test_ci_infrastructure.py -m real_api -v
— confirms opt-in still runs them.
Full suite uv run pytest -q -m "not performance" —
4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed
from a real session.create call per the issue repro).

🤖 Generated with Copilot CLI

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Add a pytest_collection_modifyitems hook to tests/conftest.py that skips any @pytest.mark.real_api test unless the caller's -m expression already references real_api (e.g. `-m real_api` to opt in, or CI's exact `-m "not real_api and not performance"`), in which case pytest's own marker-expression filtering is left to decide selection/deselection. Previously nothing deselected real_api tests by default: a plain pytest, `pytest -m "not performance"`, or `make test` would run them, spawning real copilot/claude subprocesses that can collide with (and kill) a live `conductor run --web-bg` session. Adds tests/test_config/test_real_api_marker.py, a pytester-based regression suite that exercises the actual hook (loaded by file path) against a synthetic real_api-marked test, verifying: - default run (no -m) skips it - -m "not performance" skips it (the issue's exact repro) - -m real_api runs it - CI's exact -m "not real_api and not performance" still deselects it - a reverted/no-op hook would NOT skip it (load-bearing check) Verified full suite: 4161 passed, 29 skipped, 29 deselected, 0 failed (previously 1 failed from a real session.create call). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

- Update tests/conftest.py module docstring to mention the new pytest_collection_modifyitems hook instead of only "fixtures". - Use config.getoption("markexpr") instead of the "-m" option-string alias, and drop the redundant `or ""` fallback (pytest defaults -m to "" already). - De-duplicate the issue #326 rationale between the hook's docstring and the test module's docstring; the test module now points to the hook as the source of truth. - Note in test_ci_marker_expression_still_deselects that its -m string is duplicated in ci.yml/release.yml and should be kept in sync. - Add test_regex_does_not_match_unrelated_marker_name to directly verify the \breal_api\b word-boundary regex doesn't false-positive on an unrelated marker name that merely contains "real_api" as a substring. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Jason Robertand others added 2 commits July 21, 2026 14:11