fix: AnswerRelevancy silently ignores embedding_model (both sync and async)

[Rehan (spacesheepinternet)]

Fixes#166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.

[Rehan (spacesheepinternet)]

Fixes#166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.

[Rehan (spacesheepinternet)]

Fixes#166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.

[Rehan (spacesheepinternet)]

Fixes#166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.

[Rehan (spacesheepinternet)]

Fixes#166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.

[Rehan (spacesheepinternet)]

Fixes#166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.

[Rehan (spacesheepinternet)]

Fixes#166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.

[Rehan (spacesheepinternet)]

Fixes#166.

The bug is broader than #166 describes: embedding_model is ignored in both the sync and async paths, not just sync.

AnswerRelevancy passes the model at eval time:

EmbeddingSimilarity(client=self.client).eval(output=..., expected=..., model=self.embedding_model)

but EmbeddingSimilarity only reads model in its constructor (stored into self.extra_args). At eval time the kwarg lands in the **kwargs of _run_eval_sync / _run_eval_async and is never used — so every AnswerRelevancy run silently embeds with the default model, regardless of what the caller configured.

#164 fixed half of this (the sync path was passing self.model — the judge LLM — instead of self.embedding_model), but the corrected value still went to the call site, where it's dropped. #167 had the right approach (constructor, not call site) but no test coverage; this PR completes it.

Fix: pass model=self.embedding_model to the EmbeddingSimilarity constructor in both paths. embedding_model=None remains safe (the constructor falls back to get_default_embedding_model()).

Tests: two regression tests (sync + async) following the pattern of test_answer_correctness_uses_custom_embedding_model — respx captures the model field actually sent to /v1/embeddings. On main both fail with ['text-embedding-ada-002', ...]; with the fix both pass. The tests pin model="gpt-4o-mini" for question generation so the mocked chat-completions endpoint is used (gpt-5-class defaults route through the Responses API).

Found while preparing to use AnswerRelevancy with a custom embedding model for a RAG eval harness.