Add per-input trialCount support to Eval() by dflynn15 · Pull Request #1341 · braintrustdata/braintrust-sdk-javascript

Daniel Flynn (dflynn15) · 2026-02-05T19:25:44Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval("My Project",{data: [{input: "stable query",expected: "..."},// Uses global (3){input: "flaky query",expected: "...",trialCount: 10},// Override to 10{input: "deterministic",expected: "...",trialCount: 1},// Override to 1],task: myTask,scores: [Factuality],trialCount: 3,// Global default});

There is a corollary Python PR up to match it here: #1342

Allow data items to specify their own trialCount, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trialCount` field to `EvalCase` type - Per-item trialCount takes precedence over global trialCount - Items without trialCount use the global value (or 1 if unset)

github-actions · 2026-03-03T00:42:59Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:54:41Z

I have trouble fetching your fork because of a weird GH error:

2026-04-14 14:50:59.791 [error] [Review+0] Checkout failed #{"stdout":"","stderr":"ERROR: Access to this repository has been disabled by GitHub staff due to excessive resource use.\nPlease contact support via https://support.github.com/contact to restore access to this repository.\nRead about how to decrease the size of your repository:\n https://docs.github.com/articles/what-is-my-disk-quota\n\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","exitCode":128,"gitErrorCode":"RemoteConnectionError","gitCommand":"fetch","gitArgs":["fetch","dflynn15","feature/js-per-input-trial-count:pr/dflynn15/1341"]}

Gonna re-implement it and tag you as a co-author. Ty!

Luca Forstner (lforst) · 2026-04-14T12:57:41Z

Superseded by: #1814

Daniel Flynn (dflynn15) · 2026-02-05T19:25:44Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval("My Project",{data: [{input: "stable query",expected: "..."},// Uses global (3){input: "flaky query",expected: "...",trialCount: 10},// Override to 10{input: "deterministic",expected: "...",trialCount: 1},// Override to 1],task: myTask,scores: [Factuality],trialCount: 3,// Global default});

There is a corollary Python PR up to match it here: #1342

Allow data items to specify their own trialCount, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trialCount` field to `EvalCase` type - Per-item trialCount takes precedence over global trialCount - Items without trialCount use the global value (or 1 if unset)

github-actions · 2026-03-03T00:42:59Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:54:41Z

I have trouble fetching your fork because of a weird GH error:

2026-04-14 14:50:59.791 [error] [Review+0] Checkout failed #{"stdout":"","stderr":"ERROR: Access to this repository has been disabled by GitHub staff due to excessive resource use.\nPlease contact support via https://support.github.com/contact to restore access to this repository.\nRead about how to decrease the size of your repository:\n https://docs.github.com/articles/what-is-my-disk-quota\n\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","exitCode":128,"gitErrorCode":"RemoteConnectionError","gitCommand":"fetch","gitArgs":["fetch","dflynn15","feature/js-per-input-trial-count:pr/dflynn15/1341"]}

Gonna re-implement it and tag you as a co-author. Ty!

Luca Forstner (lforst) · 2026-04-14T12:57:41Z

Superseded by: #1814

Daniel Flynn (dflynn15) · 2026-02-05T19:25:44Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval("My Project",{data: [{input: "stable query",expected: "..."},// Uses global (3){input: "flaky query",expected: "...",trialCount: 10},// Override to 10{input: "deterministic",expected: "...",trialCount: 1},// Override to 1],task: myTask,scores: [Factuality],trialCount: 3,// Global default});

There is a corollary Python PR up to match it here: #1342

Allow data items to specify their own trialCount, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trialCount` field to `EvalCase` type - Per-item trialCount takes precedence over global trialCount - Items without trialCount use the global value (or 1 if unset)

github-actions · 2026-03-03T00:42:59Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:54:41Z

I have trouble fetching your fork because of a weird GH error:

2026-04-14 14:50:59.791 [error] [Review+0] Checkout failed #{"stdout":"","stderr":"ERROR: Access to this repository has been disabled by GitHub staff due to excessive resource use.\nPlease contact support via https://support.github.com/contact to restore access to this repository.\nRead about how to decrease the size of your repository:\n https://docs.github.com/articles/what-is-my-disk-quota\n\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","exitCode":128,"gitErrorCode":"RemoteConnectionError","gitCommand":"fetch","gitArgs":["fetch","dflynn15","feature/js-per-input-trial-count:pr/dflynn15/1341"]}

Gonna re-implement it and tag you as a co-author. Ty!

Luca Forstner (lforst) · 2026-04-14T12:57:41Z

Superseded by: #1814

Daniel Flynn (dflynn15) · 2026-02-05T19:25:44Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval("My Project",{data: [{input: "stable query",expected: "..."},// Uses global (3){input: "flaky query",expected: "...",trialCount: 10},// Override to 10{input: "deterministic",expected: "...",trialCount: 1},// Override to 1],task: myTask,scores: [Factuality],trialCount: 3,// Global default});

There is a corollary Python PR up to match it here: #1342

Allow data items to specify their own trialCount, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trialCount` field to `EvalCase` type - Per-item trialCount takes precedence over global trialCount - Items without trialCount use the global value (or 1 if unset)

github-actions · 2026-03-03T00:42:59Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:54:41Z

I have trouble fetching your fork because of a weird GH error:

2026-04-14 14:50:59.791 [error] [Review+0] Checkout failed #{"stdout":"","stderr":"ERROR: Access to this repository has been disabled by GitHub staff due to excessive resource use.\nPlease contact support via https://support.github.com/contact to restore access to this repository.\nRead about how to decrease the size of your repository:\n https://docs.github.com/articles/what-is-my-disk-quota\n\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","exitCode":128,"gitErrorCode":"RemoteConnectionError","gitCommand":"fetch","gitArgs":["fetch","dflynn15","feature/js-per-input-trial-count:pr/dflynn15/1341"]}

Gonna re-implement it and tag you as a co-author. Ty!

Luca Forstner (lforst) · 2026-04-14T12:57:41Z

Superseded by: #1814

Daniel Flynn (dflynn15) · 2026-02-05T19:25:44Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval("My Project",{data: [{input: "stable query",expected: "..."},// Uses global (3){input: "flaky query",expected: "...",trialCount: 10},// Override to 10{input: "deterministic",expected: "...",trialCount: 1},// Override to 1],task: myTask,scores: [Factuality],trialCount: 3,// Global default});

There is a corollary Python PR up to match it here: #1342

Allow data items to specify their own trialCount, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trialCount` field to `EvalCase` type - Per-item trialCount takes precedence over global trialCount - Items without trialCount use the global value (or 1 if unset)

github-actions · 2026-03-03T00:42:59Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:54:41Z

I have trouble fetching your fork because of a weird GH error:

2026-04-14 14:50:59.791 [error] [Review+0] Checkout failed #{"stdout":"","stderr":"ERROR: Access to this repository has been disabled by GitHub staff due to excessive resource use.\nPlease contact support via https://support.github.com/contact to restore access to this repository.\nRead about how to decrease the size of your repository:\n https://docs.github.com/articles/what-is-my-disk-quota\n\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","exitCode":128,"gitErrorCode":"RemoteConnectionError","gitCommand":"fetch","gitArgs":["fetch","dflynn15","feature/js-per-input-trial-count:pr/dflynn15/1341"]}

Gonna re-implement it and tag you as a co-author. Ty!

Luca Forstner (lforst) · 2026-04-14T12:57:41Z

Superseded by: #1814

Daniel Flynn (dflynn15) · 2026-02-05T19:25:44Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval("My Project",{data: [{input: "stable query",expected: "..."},// Uses global (3){input: "flaky query",expected: "...",trialCount: 10},// Override to 10{input: "deterministic",expected: "...",trialCount: 1},// Override to 1],task: myTask,scores: [Factuality],trialCount: 3,// Global default});

There is a corollary Python PR up to match it here: #1342

Allow data items to specify their own trialCount, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trialCount` field to `EvalCase` type - Per-item trialCount takes precedence over global trialCount - Items without trialCount use the global value (or 1 if unset)

github-actions · 2026-03-03T00:42:59Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:54:41Z

I have trouble fetching your fork because of a weird GH error:

2026-04-14 14:50:59.791 [error] [Review+0] Checkout failed #{"stdout":"","stderr":"ERROR: Access to this repository has been disabled by GitHub staff due to excessive resource use.\nPlease contact support via https://support.github.com/contact to restore access to this repository.\nRead about how to decrease the size of your repository:\n https://docs.github.com/articles/what-is-my-disk-quota\n\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","exitCode":128,"gitErrorCode":"RemoteConnectionError","gitCommand":"fetch","gitArgs":["fetch","dflynn15","feature/js-per-input-trial-count:pr/dflynn15/1341"]}

Gonna re-implement it and tag you as a co-author. Ty!

Luca Forstner (lforst) · 2026-04-14T12:57:41Z

Superseded by: #1814

Daniel Flynn (dflynn15) · 2026-02-05T19:25:44Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval("My Project",{data: [{input: "stable query",expected: "..."},// Uses global (3){input: "flaky query",expected: "...",trialCount: 10},// Override to 10{input: "deterministic",expected: "...",trialCount: 1},// Override to 1],task: myTask,scores: [Factuality],trialCount: 3,// Global default});

There is a corollary Python PR up to match it here: #1342

Allow data items to specify their own trialCount, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trialCount` field to `EvalCase` type - Per-item trialCount takes precedence over global trialCount - Items without trialCount use the global value (or 1 if unset)

github-actions · 2026-03-03T00:42:59Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:54:41Z

I have trouble fetching your fork because of a weird GH error:

2026-04-14 14:50:59.791 [error] [Review+0] Checkout failed #{"stdout":"","stderr":"ERROR: Access to this repository has been disabled by GitHub staff due to excessive resource use.\nPlease contact support via https://support.github.com/contact to restore access to this repository.\nRead about how to decrease the size of your repository:\n https://docs.github.com/articles/what-is-my-disk-quota\n\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","exitCode":128,"gitErrorCode":"RemoteConnectionError","gitCommand":"fetch","gitArgs":["fetch","dflynn15","feature/js-per-input-trial-count:pr/dflynn15/1341"]}

Gonna re-implement it and tag you as a co-author. Ty!

Luca Forstner (lforst) · 2026-04-14T12:57:41Z

Superseded by: #1814

Daniel Flynn (dflynn15) · 2026-02-05T19:25:44Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval("My Project",{data: [{input: "stable query",expected: "..."},// Uses global (3){input: "flaky query",expected: "...",trialCount: 10},// Override to 10{input: "deterministic",expected: "...",trialCount: 1},// Override to 1],task: myTask,scores: [Factuality],trialCount: 3,// Global default});

There is a corollary Python PR up to match it here: #1342

Allow data items to specify their own trialCount, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trialCount` field to `EvalCase` type - Per-item trialCount takes precedence over global trialCount - Items without trialCount use the global value (or 1 if unset)

github-actions · 2026-03-03T00:42:59Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:54:41Z

I have trouble fetching your fork because of a weird GH error:

2026-04-14 14:50:59.791 [error] [Review+0] Checkout failed #{"stdout":"","stderr":"ERROR: Access to this repository has been disabled by GitHub staff due to excessive resource use.\nPlease contact support via https://support.github.com/contact to restore access to this repository.\nRead about how to decrease the size of your repository:\n https://docs.github.com/articles/what-is-my-disk-quota\n\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n","exitCode":128,"gitErrorCode":"RemoteConnectionError","gitCommand":"fetch","gitArgs":["fetch","dflynn15","feature/js-per-input-trial-count:pr/dflynn15/1341"]}

Gonna re-implement it and tag you as a co-author. Ty!

Luca Forstner (lforst) · 2026-04-14T12:57:41Z

Superseded by: #1814

Daniel Flynn (dflynn15) mentioned this pull request Feb 5, 2026
Add per-input trial_count support to Eval() #1342
Closed

github-actionsBot added the stale label Mar 3, 2026