Skip to content
This repository was archived by the owner on Jun 5, 2025. It is now read-only.

Add integration checks for comparing the result of calling the model API directly vs via CodeGate - #1032

Merged
rdimitrov merged 2 commits into
mainfrom
extend-tests
Feb 13, 2025
Merged

Add integration checks for comparing the result of calling the model API directly vs via CodeGate#1032
rdimitrov merged 2 commits into
mainfrom
extend-tests

Conversation

@rdimitrov

@rdimitrovrdimitrov commented Feb 13, 2025

Copy link
Copy Markdown
Member

This introduces the following format:

codegate_enrichment:
provider_url: http://127.0.0.1:11434/api/chatexpect_difference: false

Fixes: #841

@rdimitrov
rdimitrovforce-pushed the extend-tests branch 9 times, most recently from 778e682 to 2ece4d7CompareFebruary 13, 2025 14:43
@rdimitrov
Enable codegate enrichment tests
2da3a74
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
aponcedeleonch
aponcedeleonch reviewed Feb 13, 2025
View reviewed changes
Comment threadtests/integration/integration_tests.py Outdated
aponcedeleonch
aponcedeleonch previously approved these changes Feb 13, 2025
View reviewed changes

@aponcedeleonchaponcedeleonch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one comment. But in general the code looks good. Out of curiosity, are we not going to test also with OpenAI, Anthropic and Openrouter?

@rdimitrov

rdimitrov commented Feb 13, 2025•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Just one comment. But in general the code looks good. Out of curiosity, are we not going to test also with OpenAI, Anthropic and Openrouter?

My understanding is that it is enough to ensure this for one provider and so it doesn't make sense to repeat it for others (and consume API tokens). Is this correct assumption or we should extend these tests for the others too?

@rdimitrov
Re-use call_provider for calling both codegate and the provider
88a591e
Signed-off-by: Radoslav Dimitrov <radoslav@stacklok.com>
@aponcedeleonch

aponcedeleonch commented Feb 13, 2025

Copy link
Copy Markdown
Member

My understanding is that it is enough to ensure this for one provider and so it doesn't make sense to repeat it for others (and consume API tokens). Is this correct assumption or we should extend these tests for the others too?

Yes, that's right

@rdimitrov
rdimitrov merged commit 426105e into mainFeb 13, 2025
@rdimitrov
rdimitrov deleted the extend-tests branch February 13, 2025 17:24
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer
@aponcedeleonchaponcedeleonchaponcedeleonch approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Task]: Add integration test for comparing the result of calling the API directly vs through CodeGate

2 participants

@rdimitrov@aponcedeleonch