feat(scorers): create LLM scorers and classifiers by viadezo1er · Pull Request #307 · braintrustdata/bt

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:56Z

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --temperature 1000 --reasoning-effort none

bt-scorer-messages.json:

[
{
"role": "system",
"content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
},
{
"role": "user",
"content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
}
]

❯ bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --json|jq
{
"found_existing": true,
"id": "6681c294-0ab3-489b-b979-33351add071d",
"ignored": false,
"project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
"slug": "demo-answer-correctness",
"version": "1000197688638740786"
}

github-actions · 2026-08-14T01:52:15Z

Latest downloadable build artifacts for this PR commit 7fed7130c346:

Workflow run: https://github.com/braintrustdata/bt/actions/runs/32202114784
Download all artifacts (GitHub CLI): gh run download 32202114784 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Stephen Belanger (Qard)

Still overall LGTM, though CI still seems to be failing, and looks like it needs a rebase.

Cedric / ViaDézo1er (viadezo1er) · 2026-08-18T20:58:12Z

Rebase done

…n BTQL requests

… in the web ui

…has been ignored when it happens

…se instead of what bt asked to be created

…_api_key responses are classified as user/provider errors, not Braintrust authentication failures

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:56Z

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --temperature 1000 --reasoning-effort none

bt-scorer-messages.json:

[
{
"role": "system",
"content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
},
{
"role": "user",
"content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
}
]

❯ bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --json|jq
{
"found_existing": true,
"id": "6681c294-0ab3-489b-b979-33351add071d",
"ignored": false,
"project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
"slug": "demo-answer-correctness",
"version": "1000197688638740786"
}

github-actions · 2026-08-14T01:52:15Z

Latest downloadable build artifacts for this PR commit 7fed7130c346:

Workflow run: https://github.com/braintrustdata/bt/actions/runs/32202114784
Download all artifacts (GitHub CLI): gh run download 32202114784 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Stephen Belanger (Qard)

Still overall LGTM, though CI still seems to be failing, and looks like it needs a rebase.

Cedric / ViaDézo1er (viadezo1er) · 2026-08-18T20:58:12Z

Rebase done

…n BTQL requests

… in the web ui

…has been ignored when it happens

…se instead of what bt asked to be created

…_api_key responses are classified as user/provider errors, not Braintrust authentication failures

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:56Z

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --temperature 1000 --reasoning-effort none

bt-scorer-messages.json:

[
{
"role": "system",
"content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
},
{
"role": "user",
"content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
}
]

❯ bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --json|jq
{
"found_existing": true,
"id": "6681c294-0ab3-489b-b979-33351add071d",
"ignored": false,
"project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
"slug": "demo-answer-correctness",
"version": "1000197688638740786"
}

github-actions · 2026-08-14T01:52:15Z

Latest downloadable build artifacts for this PR commit 7fed7130c346:

Workflow run: https://github.com/braintrustdata/bt/actions/runs/32202114784
Download all artifacts (GitHub CLI): gh run download 32202114784 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Stephen Belanger (Qard)

Still overall LGTM, though CI still seems to be failing, and looks like it needs a rebase.

Cedric / ViaDézo1er (viadezo1er) · 2026-08-18T20:58:12Z

Rebase done

…n BTQL requests

… in the web ui

…has been ignored when it happens

…se instead of what bt asked to be created

…_api_key responses are classified as user/provider errors, not Braintrust authentication failures

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:56Z

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --temperature 1000 --reasoning-effort none

bt-scorer-messages.json:

[
{
"role": "system",
"content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
},
{
"role": "user",
"content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
}
]

❯ bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --json|jq
{
"found_existing": true,
"id": "6681c294-0ab3-489b-b979-33351add071d",
"ignored": false,
"project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
"slug": "demo-answer-correctness",
"version": "1000197688638740786"
}

github-actions · 2026-08-14T01:52:15Z

Latest downloadable build artifacts for this PR commit 7fed7130c346:

Workflow run: https://github.com/braintrustdata/bt/actions/runs/32202114784
Download all artifacts (GitHub CLI): gh run download 32202114784 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Stephen Belanger (Qard)

Still overall LGTM, though CI still seems to be failing, and looks like it needs a rebase.

Cedric / ViaDézo1er (viadezo1er) · 2026-08-18T20:58:12Z

Rebase done

…n BTQL requests

… in the web ui

…has been ignored when it happens

…se instead of what bt asked to be created

…_api_key responses are classified as user/provider errors, not Braintrust authentication failures

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:56Z

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --temperature 1000 --reasoning-effort none

bt-scorer-messages.json:

[
{
"role": "system",
"content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
},
{
"role": "user",
"content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
}
]

❯ bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --json|jq
{
"found_existing": true,
"id": "6681c294-0ab3-489b-b979-33351add071d",
"ignored": false,
"project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
"slug": "demo-answer-correctness",
"version": "1000197688638740786"
}

github-actions · 2026-08-14T01:52:15Z

Latest downloadable build artifacts for this PR commit 7fed7130c346:

Workflow run: https://github.com/braintrustdata/bt/actions/runs/32202114784
Download all artifacts (GitHub CLI): gh run download 32202114784 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Stephen Belanger (Qard)

Still overall LGTM, though CI still seems to be failing, and looks like it needs a rebase.

Cedric / ViaDézo1er (viadezo1er) · 2026-08-18T20:58:12Z

Rebase done

…n BTQL requests

… in the web ui

…has been ignored when it happens

…se instead of what bt asked to be created

…_api_key responses are classified as user/provider errors, not Braintrust authentication failures

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:56Z

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --temperature 1000 --reasoning-effort none

bt-scorer-messages.json:

[
{
"role": "system",
"content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
},
{
"role": "user",
"content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
}
]

❯ bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --json|jq
{
"found_existing": true,
"id": "6681c294-0ab3-489b-b979-33351add071d",
"ignored": false,
"project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
"slug": "demo-answer-correctness",
"version": "1000197688638740786"
}

github-actions · 2026-08-14T01:52:15Z

Latest downloadable build artifacts for this PR commit 7fed7130c346:

Workflow run: https://github.com/braintrustdata/bt/actions/runs/32202114784
Download all artifacts (GitHub CLI): gh run download 32202114784 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Stephen Belanger (Qard)

Still overall LGTM, though CI still seems to be failing, and looks like it needs a rebase.

Cedric / ViaDézo1er (viadezo1er) · 2026-08-18T20:58:12Z

Rebase done

…n BTQL requests

… in the web ui

…has been ignored when it happens

…se instead of what bt asked to be created

…_api_key responses are classified as user/provider errors, not Braintrust authentication failures

Cedric / ViaDézo1er (viadezo1er) · 2026-08-14T01:38:56Z

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --temperature 1000 --reasoning-effort none

bt-scorer-messages.json:

[
{
"role": "system",
"content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
},
{
"role": "user",
"content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
}
]

❯ bt scorers create "Demo answer correctness" \
--slug demo-answer-correctness \
--description "Synthetic scorer used to demonstrate bt scorers create" \
--model gpt-5.4-nano \
--messages @/tmp/bt-scorer-messages.json \
--choice-scores '{"PASS":1,"FAIL":0}' \
--use-cot=false \
--pass-threshold 0.5 \
--metadata 'purpose: synthetic-demo' \
--if-exists replace --json|jq
{
"found_existing": true,
"id": "6681c294-0ab3-489b-b979-33351add071d",
"ignored": false,
"project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
"slug": "demo-answer-correctness",
"version": "1000197688638740786"
}

github-actions · 2026-08-14T01:52:15Z

Latest downloadable build artifacts for this PR commit 7fed7130c346:

Workflow run: https://github.com/braintrustdata/bt/actions/runs/32202114784
Download all artifacts (GitHub CLI): gh run download 32202114784 --repo braintrustdata/bt
Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

artifacts-build-global
artifacts-build-local-aarch64-pc-windows-msvc
artifacts-build-local-x86_64-pc-windows-msvc
artifacts-build-local-x86_64-apple-darwin
artifacts-build-local-x86_64-unknown-linux-musl
artifacts-build-local-x86_64-unknown-linux-gnu
artifacts-build-local-aarch64-apple-darwin
artifacts-build-local-aarch64-unknown-linux-gnu
artifacts-plan-dist-manifest
cargo-dist-cache

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit:9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Stephen Belanger (Qard)

Still overall LGTM, though CI still seems to be failing, and looks like it needs a rebase.

Cedric / ViaDézo1er (viadezo1er) · 2026-08-18T20:58:12Z

Rebase done

…n BTQL requests

… in the web ui

…has been ignored when it happens

…se instead of what bt asked to be created

…_api_key responses are classified as user/provider errors, not Braintrust authentication failures

Cedric / ViaDézo1er (viadezo1er) mentioned this pull request Aug 14, 2026
bt scorers create/update #283
Closed

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from 5e13709 to 8f1e4c0Compare August 14, 2026 20:15

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from 8f1e4c0 to 9d92d66Compare August 14, 2026 20:32

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from 9d92d66 to 06883feCompare August 14, 2026 21:15

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from 06883fe to ddc82b0Compare August 14, 2026 21:27

Cedric / ViaDézo1er (viadezo1er) changed the base branch from cedric/fix-btql-safety to mainAugust 14, 2026 21:31

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from 0d53e92 to 9963a33Compare August 14, 2026 22:49

Cedric / ViaDézo1er (viadezo1er) marked this pull request as ready for review August 14, 2026 22:50

chatgpt-codex-connectorBot reviewed Aug 14, 2026
View reviewed changes

Comment threadsrc/functions/create.rs

Cedric / ViaDézo1er (viadezo1er) requested review from Stephen Belanger (Qard) and Parker Henderson (parkerhendo)August 14, 2026 23:12

Stephen Belanger (Qard) approved these changes Aug 14, 2026
View reviewed changes

Stephen Belanger (Qard) approved these changes Aug 18, 2026
View reviewed changes

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch 2 times, most recently from a8f8b84 to e737d2bCompare August 18, 2026 20:57

Cedric / ViaDézo1er (viadezo1er) added 8 commits August 18, 2026 17:40

feat(scorers): create LLM scorers and classifiers
a6be250

chore: AGENTS.md had imprecise guidelines about needing a timestamp i…
26b8953
…n BTQL requests

chore(scorer): show classification scorer with bt scorers list like…
1af1fd0
… in the web ui

fix(scorer): --if-exists ignore --json correctly shows the scorers …
0243ac8
…has been ignored when it happens

chore(scorer): scorer creation returns the json of the backend respon…
5451e85
…se instead of what bt asked to be created

fix(scorer): invoking a scorer respects --json and provider invalid…
a7cad9a
…_api_key responses are classified as user/provider errors, not Braintrust authentication failures

chore(scorers): use_cache can only be used when temperature is 0
1ae5341

fix: test expecting output in stderr instead of stdout
7fed713

Cedric / ViaDézo1er (viadezo1er)force-pushed the cedric/scorers-create branch from c8cf632 to 7fed713Compare August 19, 2026 00:40

Cedric / ViaDézo1er (viadezo1er) merged commit 45b92d3 into mainAug 19, 2026
34 checks passed