feat(scorers): validate model parameters by viadezo1er · Pull Request #308 · braintrustdata/bt

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:59Z

Instead of accepting any kinds of invalid parameters, now the backend checks them and bt surfaces the error message.

❯ bt scorers create test-invalid-top-k-scorer \
--slug test-invalid-top-k-scorer \
--model gpt-4.1 \
--top-k 10 \
--messages '[{"role":"user","content":"Return A or B."}]' \
--choice-scores '{"A":1,"B":0}' \
--if-exists replace \
--no-input \
--json
{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

send the complete candidate scorer once to POST /insert-functions
parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
preserve structured provider authentication errors while classifying them as user errors
reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Workflow run: https://github.com/braintrustdata/bt/actions/runs/33011675875
Download all artifacts (GitHub CLI): gh run download 33011675875 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

…tion endpoint

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:59Z

Instead of accepting any kinds of invalid parameters, now the backend checks them and bt surfaces the error message.

❯ bt scorers create test-invalid-top-k-scorer \
--slug test-invalid-top-k-scorer \
--model gpt-4.1 \
--top-k 10 \
--messages '[{"role":"user","content":"Return A or B."}]' \
--choice-scores '{"A":1,"B":0}' \
--if-exists replace \
--no-input \
--json
{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

send the complete candidate scorer once to POST /insert-functions
parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
preserve structured provider authentication errors while classifying them as user errors
reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Workflow run: https://github.com/braintrustdata/bt/actions/runs/33011675875
Download all artifacts (GitHub CLI): gh run download 33011675875 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

…tion endpoint

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:59Z

Instead of accepting any kinds of invalid parameters, now the backend checks them and bt surfaces the error message.

❯ bt scorers create test-invalid-top-k-scorer \
--slug test-invalid-top-k-scorer \
--model gpt-4.1 \
--top-k 10 \
--messages '[{"role":"user","content":"Return A or B."}]' \
--choice-scores '{"A":1,"B":0}' \
--if-exists replace \
--no-input \
--json
{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

send the complete candidate scorer once to POST /insert-functions
parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
preserve structured provider authentication errors while classifying them as user errors
reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Workflow run: https://github.com/braintrustdata/bt/actions/runs/33011675875
Download all artifacts (GitHub CLI): gh run download 33011675875 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

…tion endpoint

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:59Z

Instead of accepting any kinds of invalid parameters, now the backend checks them and bt surfaces the error message.

❯ bt scorers create test-invalid-top-k-scorer \
--slug test-invalid-top-k-scorer \
--model gpt-4.1 \
--top-k 10 \
--messages '[{"role":"user","content":"Return A or B."}]' \
--choice-scores '{"A":1,"B":0}' \
--if-exists replace \
--no-input \
--json
{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

send the complete candidate scorer once to POST /insert-functions
parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
preserve structured provider authentication errors while classifying them as user errors
reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Workflow run: https://github.com/braintrustdata/bt/actions/runs/33011675875
Download all artifacts (GitHub CLI): gh run download 33011675875 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

…tion endpoint

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:59Z

Instead of accepting any kinds of invalid parameters, now the backend checks them and bt surfaces the error message.

❯ bt scorers create test-invalid-top-k-scorer \
--slug test-invalid-top-k-scorer \
--model gpt-4.1 \
--top-k 10 \
--messages '[{"role":"user","content":"Return A or B."}]' \
--choice-scores '{"A":1,"B":0}' \
--if-exists replace \
--no-input \
--json
{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

send the complete candidate scorer once to POST /insert-functions
parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
preserve structured provider authentication errors while classifying them as user errors
reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Workflow run: https://github.com/braintrustdata/bt/actions/runs/33011675875
Download all artifacts (GitHub CLI): gh run download 33011675875 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

…tion endpoint

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:59Z

Instead of accepting any kinds of invalid parameters, now the backend checks them and bt surfaces the error message.

❯ bt scorers create test-invalid-top-k-scorer \
--slug test-invalid-top-k-scorer \
--model gpt-4.1 \
--top-k 10 \
--messages '[{"role":"user","content":"Return A or B."}]' \
--choice-scores '{"A":1,"B":0}' \
--if-exists replace \
--no-input \
--json
{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

send the complete candidate scorer once to POST /insert-functions
parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
preserve structured provider authentication errors while classifying them as user errors
reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Workflow run: https://github.com/braintrustdata/bt/actions/runs/33011675875
Download all artifacts (GitHub CLI): gh run download 33011675875 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

…tion endpoint

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:59Z

Instead of accepting any kinds of invalid parameters, now the backend checks them and bt surfaces the error message.

❯ bt scorers create test-invalid-top-k-scorer \
--slug test-invalid-top-k-scorer \
--model gpt-4.1 \
--top-k 10 \
--messages '[{"role":"user","content":"Return A or B."}]' \
--choice-scores '{"A":1,"B":0}' \
--if-exists replace \
--no-input \
--json
{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

send the complete candidate scorer once to POST /insert-functions
parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
preserve structured provider authentication errors while classifying them as user errors
reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Workflow run: https://github.com/braintrustdata/bt/actions/runs/33011675875
Download all artifacts (GitHub CLI): gh run download 33011675875 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

…tion endpoint

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:59Z

Instead of accepting any kinds of invalid parameters, now the backend checks them and bt surfaces the error message.

❯ bt scorers create test-invalid-top-k-scorer \
--slug test-invalid-top-k-scorer \
--model gpt-4.1 \
--top-k 10 \
--messages '[{"role":"user","content":"Return A or B."}]' \
--choice-scores '{"A":1,"B":0}' \
--if-exists replace \
--no-input \
--json
{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

send the complete candidate scorer once to POST /insert-functions
parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
preserve structured provider authentication errors while classifying them as user errors
reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Workflow run: https://github.com/braintrustdata/bt/actions/runs/33011675875
Download all artifacts (GitHub CLI): gh run download 33011675875 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

…tion endpoint

Cedric / ViaDézo1er (viadezo1er) mentioned this pull request Aug 14, 2026
bt scorers create/update #283
Closed

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from 5e13709 to 8f1e4c0Compare August 14, 2026 20:15

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-validate-model-parameters branch from 3c04afd to 89fac1eCompare August 14, 2026 20:15

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch 3 times, most recently from 06883fe to ddc82b0Compare August 14, 2026 21:27

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-validate-model-parameters branch from 89fac1e to 3639e73Compare August 14, 2026 21:30

Cedric / ViaDézo1er (viadezo1er) mentioned this pull request Aug 14, 2026
feat(scorers): create LLM scorers and classifiers #307
Merged

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from 0d53e92 to 9963a33Compare August 14, 2026 22:49

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-validate-model-parameters branch 6 times, most recently from fc892d8 to 4d393b3Compare August 18, 2026 02:17

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch 2 times, most recently from a8f8b84 to e737d2bCompare August 18, 2026 20:57

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-validate-model-parameters branch 2 times, most recently from 4dba7ec to cfd98dbCompare August 19, 2026 00:08

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from c8cf632 to 7fed713Compare August 19, 2026 00:40

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-validate-model-parameters branch from cfd98db to 3ad4a13Compare August 19, 2026 00:40

Base automatically changed from cedric/scorers-create to mainAugust 19, 2026 00:55

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-validate-model-parameters branch 2 times, most recently from 1c94601 to 21447b9Compare August 19, 2026 01:13

Cedric / ViaDézo1er (viadezo1er) marked this pull request as ready for review August 19, 2026 01:59

Cedric / ViaDézo1er (viadezo1er) requested review from Abhijeet Prasad (AbhiPrasad) and Parker Henderson (parkerhendo)August 19, 2026 02:03

This comment was marked as outdated.
Sign in to view

Cedric / ViaDézo1er (viadezo1er) mentioned this pull request Aug 19, 2026
chore: add scorer parameter validation route braintrustdata/terraform-aws-braintrust-data-plane#340
Closed

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-validate-model-parameters branch 3 times, most recently from 69b2754 to 80037f7Compare August 21, 2026 18:52

Luca Forstner (lforst) self-requested a review August 25, 2026 18:17

Stephen Belanger (Qard) approved these changes Aug 26, 2026
View reviewed changes

Cedric / ViaDézo1er (viadezo1er) added 4 commits August 26, 2026 13:41

chore(scorer): validate scorer parameters with the backend
2837eea

chore: Preserve structured provider errors in scorer validation
7f967ff

chore: instead of a new validation endpoint, use the same insert-func…
75973bf
…tion endpoint

chore: fix test
e68ddee

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-validate-model-parameters branch from 80037f7 to e68ddeeCompare August 26, 2026 20:41

Cedric / ViaDézo1er (viadezo1er) merged commit 665992f into mainAug 26, 2026
34 checks passed