test(fixtures): add self-contained openai-default test system YAML

[Karn2898]

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

• tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
• Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
• A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
• Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

[Karn2898]

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[AmitAvital1]

maybe lets call this file utils?

[Karn2898]

sure

[AmitAvital1]

maybe lets do it injectable? think about we will added more yamls for tests in your infra

[Karn2898]

That's a good point . I have made changes ,just need to push

[AmitAvital1]

what about test for all of this configurations?

[Karn2898]

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

[AmitAvital1]

Nice work. can we also test generate?

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

[Karn2898]

Nice work. can we also test generate?

Done , added tests/cli/test_generate_command.py with two tests using CliRunner against the fixture's agents.yaml

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

yeah I was ill so couldn't complete all tasks . now i will push

[Asaf-prog]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

[Karn2898]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

echo_tool signature now matches the emitted args (message: str instead of input: dict)
Test now asserts status == "succeeded" and error is None
Branch rebased onto latest main; fallback behavior/tests in test_engine_flow.py preserved

[Asaf-prog]

Please rebase this PR onto main.

[Karn2898]

Please rebase this PR onto main.

Rebased onto the latest main and force-pushed the updated branch. I also resolved the merge conflicts while preserving the newer fake-model fallback coverage and the CLI generate test additions.

[Karn2898]

Thank you sir :)

…

On Wed, Aug 5, 2026, 02:26 Asaf ***@***.***> wrote: Merged #36 <#36> into main. — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGCN6PTEK4A6NXAUWCD5IJEY3A5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBZG42DEMRWGYYTLJTSMVQXG33OUZQXG43JM5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28974226615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGAW6NEKAVJSKYSF46T5IJEY3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[Karn2898]

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

• tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
• Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
• A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
• Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

[Karn2898]

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[AmitAvital1]

maybe lets call this file utils?

[Karn2898]

sure

[AmitAvital1]

maybe lets do it injectable? think about we will added more yamls for tests in your infra

[Karn2898]

That's a good point . I have made changes ,just need to push

[AmitAvital1]

what about test for all of this configurations?

[Karn2898]

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

[AmitAvital1]

Nice work. can we also test generate?

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

[Karn2898]

Nice work. can we also test generate?

Done , added tests/cli/test_generate_command.py with two tests using CliRunner against the fixture's agents.yaml

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

yeah I was ill so couldn't complete all tasks . now i will push

[Asaf-prog]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

[Karn2898]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

echo_tool signature now matches the emitted args (message: str instead of input: dict)
Test now asserts status == "succeeded" and error is None
Branch rebased onto latest main; fallback behavior/tests in test_engine_flow.py preserved

[Asaf-prog]

Please rebase this PR onto main.

[Karn2898]

Please rebase this PR onto main.

Rebased onto the latest main and force-pushed the updated branch. I also resolved the merge conflicts while preserving the newer fake-model fallback coverage and the CLI generate test additions.

[Karn2898]

Thank you sir :)

…

On Wed, Aug 5, 2026, 02:26 Asaf ***@***.***> wrote: Merged #36 <#36> into main. — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGCN6PTEK4A6NXAUWCD5IJEY3A5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBZG42DEMRWGYYTLJTSMVQXG33OUZQXG43JM5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28974226615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGAW6NEKAVJSKYSF46T5IJEY3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[Karn2898]

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

• tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
• Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
• A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
• Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

[Karn2898]

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[AmitAvital1]

maybe lets call this file utils?

[Karn2898]

sure

[AmitAvital1]

maybe lets do it injectable? think about we will added more yamls for tests in your infra

[Karn2898]

That's a good point . I have made changes ,just need to push

[AmitAvital1]

what about test for all of this configurations?

[Karn2898]

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

[AmitAvital1]

Nice work. can we also test generate?

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

[Karn2898]

Nice work. can we also test generate?

Done , added tests/cli/test_generate_command.py with two tests using CliRunner against the fixture's agents.yaml

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

yeah I was ill so couldn't complete all tasks . now i will push

[Asaf-prog]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

[Karn2898]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

echo_tool signature now matches the emitted args (message: str instead of input: dict)
Test now asserts status == "succeeded" and error is None
Branch rebased onto latest main; fallback behavior/tests in test_engine_flow.py preserved

[Asaf-prog]

Please rebase this PR onto main.

[Karn2898]

Please rebase this PR onto main.

Rebased onto the latest main and force-pushed the updated branch. I also resolved the merge conflicts while preserving the newer fake-model fallback coverage and the CLI generate test additions.

[Karn2898]

Thank you sir :)

…

On Wed, Aug 5, 2026, 02:26 Asaf ***@***.***> wrote: Merged #36 <#36> into main. — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGCN6PTEK4A6NXAUWCD5IJEY3A5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBZG42DEMRWGYYTLJTSMVQXG33OUZQXG43JM5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28974226615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGAW6NEKAVJSKYSF46T5IJEY3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[Karn2898]

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

• tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
• Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
• A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
• Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

[Karn2898]

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[AmitAvital1]

maybe lets call this file utils?

[Karn2898]

sure

[AmitAvital1]

maybe lets do it injectable? think about we will added more yamls for tests in your infra

[Karn2898]

That's a good point . I have made changes ,just need to push

[AmitAvital1]

what about test for all of this configurations?

[Karn2898]

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

[AmitAvital1]

Nice work. can we also test generate?

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

[Karn2898]

Nice work. can we also test generate?

Done , added tests/cli/test_generate_command.py with two tests using CliRunner against the fixture's agents.yaml

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

yeah I was ill so couldn't complete all tasks . now i will push

[Asaf-prog]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

[Karn2898]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

echo_tool signature now matches the emitted args (message: str instead of input: dict)
Test now asserts status == "succeeded" and error is None
Branch rebased onto latest main; fallback behavior/tests in test_engine_flow.py preserved

[Asaf-prog]

Please rebase this PR onto main.

[Karn2898]

Please rebase this PR onto main.

Rebased onto the latest main and force-pushed the updated branch. I also resolved the merge conflicts while preserving the newer fake-model fallback coverage and the CLI generate test additions.

[Karn2898]

Thank you sir :)

…

On Wed, Aug 5, 2026, 02:26 Asaf ***@***.***> wrote: Merged #36 <#36> into main. — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGCN6PTEK4A6NXAUWCD5IJEY3A5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBZG42DEMRWGYYTLJTSMVQXG33OUZQXG43JM5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28974226615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGAW6NEKAVJSKYSF46T5IJEY3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[Karn2898]

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

• tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
• Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
• A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
• Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

[Karn2898]

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[AmitAvital1]

maybe lets call this file utils?

[Karn2898]

sure

[AmitAvital1]

maybe lets do it injectable? think about we will added more yamls for tests in your infra

[Karn2898]

That's a good point . I have made changes ,just need to push

[AmitAvital1]

what about test for all of this configurations?

[Karn2898]

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

[AmitAvital1]

Nice work. can we also test generate?

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

[Karn2898]

Nice work. can we also test generate?

Done , added tests/cli/test_generate_command.py with two tests using CliRunner against the fixture's agents.yaml

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

yeah I was ill so couldn't complete all tasks . now i will push

[Asaf-prog]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

[Karn2898]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

echo_tool signature now matches the emitted args (message: str instead of input: dict)
Test now asserts status == "succeeded" and error is None
Branch rebased onto latest main; fallback behavior/tests in test_engine_flow.py preserved

[Asaf-prog]

Please rebase this PR onto main.

[Karn2898]

Please rebase this PR onto main.

Rebased onto the latest main and force-pushed the updated branch. I also resolved the merge conflicts while preserving the newer fake-model fallback coverage and the CLI generate test additions.

[Karn2898]

Thank you sir :)

…

On Wed, Aug 5, 2026, 02:26 Asaf ***@***.***> wrote: Merged #36 <#36> into main. — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGCN6PTEK4A6NXAUWCD5IJEY3A5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBZG42DEMRWGYYTLJTSMVQXG33OUZQXG43JM5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28974226615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGAW6NEKAVJSKYSF46T5IJEY3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[Karn2898]

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

• tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
• Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
• A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
• Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

[Karn2898]

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[AmitAvital1]

maybe lets call this file utils?

[Karn2898]

sure

[AmitAvital1]

maybe lets do it injectable? think about we will added more yamls for tests in your infra

[Karn2898]

That's a good point . I have made changes ,just need to push

[AmitAvital1]

what about test for all of this configurations?

[Karn2898]

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

[AmitAvital1]

Nice work. can we also test generate?

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

[Karn2898]

Nice work. can we also test generate?

Done , added tests/cli/test_generate_command.py with two tests using CliRunner against the fixture's agents.yaml

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

yeah I was ill so couldn't complete all tasks . now i will push

[Asaf-prog]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

[Karn2898]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

echo_tool signature now matches the emitted args (message: str instead of input: dict)
Test now asserts status == "succeeded" and error is None
Branch rebased onto latest main; fallback behavior/tests in test_engine_flow.py preserved

[Asaf-prog]

Please rebase this PR onto main.

[Karn2898]

Please rebase this PR onto main.

Rebased onto the latest main and force-pushed the updated branch. I also resolved the merge conflicts while preserving the newer fake-model fallback coverage and the CLI generate test additions.

[Karn2898]

Thank you sir :)

…

On Wed, Aug 5, 2026, 02:26 Asaf ***@***.***> wrote: Merged #36 <#36> into main. — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGCN6PTEK4A6NXAUWCD5IJEY3A5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBZG42DEMRWGYYTLJTSMVQXG33OUZQXG43JM5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28974226615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGAW6NEKAVJSKYSF46T5IJEY3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[Karn2898]

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

• tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
• Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
• A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
• Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

[Karn2898]

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[AmitAvital1]

maybe lets call this file utils?

[Karn2898]

sure

[AmitAvital1]

maybe lets do it injectable? think about we will added more yamls for tests in your infra

[Karn2898]

That's a good point . I have made changes ,just need to push

[AmitAvital1]

what about test for all of this configurations?

[Karn2898]

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

[AmitAvital1]

Nice work. can we also test generate?

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

[Karn2898]

Nice work. can we also test generate?

Done , added tests/cli/test_generate_command.py with two tests using CliRunner against the fixture's agents.yaml

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

yeah I was ill so couldn't complete all tasks . now i will push

[Asaf-prog]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

[Karn2898]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

echo_tool signature now matches the emitted args (message: str instead of input: dict)
Test now asserts status == "succeeded" and error is None
Branch rebased onto latest main; fallback behavior/tests in test_engine_flow.py preserved

[Asaf-prog]

Please rebase this PR onto main.

[Karn2898]

Please rebase this PR onto main.

Rebased onto the latest main and force-pushed the updated branch. I also resolved the merge conflicts while preserving the newer fake-model fallback coverage and the CLI generate test additions.

[Karn2898]

Thank you sir :)

…

On Wed, Aug 5, 2026, 02:26 Asaf ***@***.***> wrote: Merged #36 <#36> into main. — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGCN6PTEK4A6NXAUWCD5IJEY3A5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBZG42DEMRWGYYTLJTSMVQXG33OUZQXG43JM5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28974226615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGAW6NEKAVJSKYSF46T5IJEY3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[Karn2898]

Summary

Add a minimal, validating agent system under tests/fixtures/ for exercising the platform without the flagship example.

• tests/fixtures/test_system.yaml: root orchestrator routing to greeting_agent and echo_agent, with a shared resolver and an echo_tool.
• Defaults to the openai provider so it runs without an Anthropic key; any of anthropic/openai/gemini/bedrock is selectable via provider:.
• A comment in the YAML makes explicit that the matching API key is supplied via the environment (OPENAI_API_KEY, etc.), never in the YAML. agentctl validate needs no key; agentctl run/serve need a key for the chosen provider.
• Prompts, resolver stubs, and the echo_tool stub are implemented so agentctl validate passes fully offline.

Verification

agentctl validate tests/fixtures/test_system.yaml
# => validation passed

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)

Thanks

honestly this PR only adds the fixture , i doesn't include the test yet .
this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent .
I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.

this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment.
How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

[Karn2898]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

[AmitAvital1]

This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve)
Thanks

honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency.
this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next.

Very good plan! We really need this benefit. we already have issue #24 and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases

apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw

Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇

[Karn2898]

Hi I have pushed some files, hope it helps. Thank you

…

On Fri, Jul 24, 2026, 13:59 AmitAvital1 ***@***.***> wrote: *AmitAvital1* left a comment (extra-org/extra#36) <#36 (comment)> This is look greate example. but can you elaborate more how we running this test as system tests? how its should work? what makes you did it? (really helpfull question also for us to understand how to improve) Thanks honestly this PR only adds the fixture , i doesn't include the test yet . this fixture gives us everything a system test would need, but there is no pytest that actually runs the full flow against it. the good news is that the engine already supports this kind of system test.tests/engine/test_engine_flow.py already follows the pattern, it injects a fake model_factory into LangGraohEngine which returns a FakeChatModel. This fake model simulates a tool call and a final response so there is no API key or network dependency. this fixture would be pretty straight forwrd: load tests/fixtures/test_system.yaml and build the engine with fake model_factory, then send hello and verify it routes root router to greeting agent . send echo test and verify it routes root router to echo agent . I left that test out of this PR intentionally. My goal here was to validate the fixture first and the engine test also touches multiple internals, so I felt it deserved its own focused PR.If you are happy with the approach I can add it next. Very good plan! We really need this benefit. we already have issue #24 <#24> and this is great improvment. How sound make this PR as infra for this issue? with couple infra test, and infra so any commit can be extended? and maybe next PR you can do more e2e tests cases apologies for the late reply , Want me to draft the infra patch now ? I'm working on that already btw Yea when ever you ready for review, push it inside this PR i would love to see it ! 🥇 — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGBGNRCJTPBMUK2DTZD5GMM6BA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMBWG43TQNRXG422M4TFMFZW63VGMFZXG2LHN2SWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5067786775>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGHIGHIEQJLXJRO3XM35GMM6BAVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>

[AmitAvital1]

maybe lets call this file utils?

[Karn2898]

sure

[AmitAvital1]

maybe lets do it injectable? think about we will added more yamls for tests in your infra

[Karn2898]

That's a good point . I have made changes ,just need to push

[AmitAvital1]

what about test for all of this configurations?

[Karn2898]

I added test_fixture_execution_policy_parsed_from_yaml which loads the spec from YAML and asserts all five fields (max_iterations, max_tool_calls, max_tool_calls_per_agent, max_child_agent_calls, allow_duplicate_tool_calls) match what's in the file. This guards against silent parser/schema changes dropping or misparsing those values. Not testing runtime enforcement here because tests/engine/test_execution_limits.py already covers that with inline specs; this fixture test just guarantees the YAML wiring is correct.

[AmitAvital1]

Nice work. can we also test generate?

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

[AmitAvital1]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

[Karn2898]

Nice work. can we also test generate?

Done , added tests/cli/test_generate_command.py with two tests using CliRunner against the fixture's agents.yaml

[Karn2898]

I see it that infra and support.py will be the infra, and test_yaml with generated files will be in dedicated folder that run this tests infra, and then we will add more folder with more yaml.. that extensabily more!

yep ,that's definetly a better structure , I have applied these changes

i added more comments please see :)

yeah I was ill so couldn't complete all tasks . now i will push

[Asaf-prog]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

[Karn2898]

Thanks — the reusable fixture and shared fake-model direction are valuable and align well with the system-test infrastructure described in #24.

I found one blocking issue in the echo-tool flow.

The shared FakeChatModel emits the tool call with:

args={"message": ...}

but the fixture tool is defined as:

defecho_tool(input: dict) ->str:

Since the engine builds a StructuredTool from the function signature, this does not match the expected input schema. The current test only verifies that an echo_tool record exists, so it can still pass when the tool execution actually failed.

Please align the tool signature and generated arguments, and explicitly assert that the tool record has status == "succeeded" and no error.

The PR is also currently not mergeable and needs to be rebased onto the latest main. While resolving that, please preserve the newer fake-model fallback behavior and fallback tests that now exist in tests/engine/test_engine_flow.py.

Once the branch is updated and the tool test proves successful execution rather than only attempted execution, the overall infrastructure looks useful.

echo_tool signature now matches the emitted args (message: str instead of input: dict)
Test now asserts status == "succeeded" and error is None
Branch rebased onto latest main; fallback behavior/tests in test_engine_flow.py preserved

[Asaf-prog]

Please rebase this PR onto main.

[Karn2898]

Please rebase this PR onto main.

Rebased onto the latest main and force-pushed the updated branch. I also resolved the merge conflicts while preserving the newer fake-model fallback coverage and the CLI generate test additions.

[Karn2898]

Thank you sir :)

…

On Wed, Aug 5, 2026, 02:26 Asaf ***@***.***> wrote: Merged #36 <#36> into main. — Reply to this email directly, view it on GitHub <#36?email_source=notifications&email_token=BJUQNGCN6PTEK4A6NXAUWCD5IJEY3A5CNFSNUABQM5UWIORPF5TWS5BNNB2WEL2JONZXKZKFOZSW45CON52GSZTJMNQXI2LPNYXTEOBZG42DEMRWGYYTLJTSMVQXG33OUZQXG43JM5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#event-28974226615>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJUQNGAW6NEKAVJSKYSF46T5IJEY3AVCNFSNUABGKJSXA33TNF2G64TZHMYTENRRGEZDOMJVGY5US43TOVSTWNBZGMYDKMRTG4YTLILWAI> . You are receiving this because you were assigned.Message ID: ***@***.***>