Add per-input trial_count support to Eval() by dflynn15 · Pull Request #1342 · braintrustdata/braintrust-sdk-javascript

Daniel Flynn (dflynn15) · 2026-02-05T19:30:19Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval(
"My Project",
data=[
{"input": "stable query", "expected": "..."}, # Uses global (3)
{"input": "flaky query", "expected": "...", "trial_count": 10}, # Override to 10EvalCase(input="deterministic", expected="...", trial_count=1), # Override to 1
],
task=my_task,
scores=[Factuality],
trial_count=3, # Global default
)

There is a corollary JS PR up to match it here: #1341

Allow data items to specify their own trial_count, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trial_count` field to `EvalCase` dataclass and TypedDict - Per-item trial_count takes precedence over global trial_count - Items without trial_count use the global value (or 1 if unset) - Works with both EvalCase objects and dict data

github-actions · 2026-03-03T00:42:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:47:05Z

Hi, this unfortunately became stale because we moved the python sdk into its own repo: https://github.com/braintrustdata/braintrust-sdk-python

We can think about re-introducing the change there.

Daniel Flynn (dflynn15) · 2026-02-05T19:30:19Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval(
"My Project",
data=[
{"input": "stable query", "expected": "..."}, # Uses global (3)
{"input": "flaky query", "expected": "...", "trial_count": 10}, # Override to 10EvalCase(input="deterministic", expected="...", trial_count=1), # Override to 1
],
task=my_task,
scores=[Factuality],
trial_count=3, # Global default
)

There is a corollary JS PR up to match it here: #1341

Allow data items to specify their own trial_count, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trial_count` field to `EvalCase` dataclass and TypedDict - Per-item trial_count takes precedence over global trial_count - Items without trial_count use the global value (or 1 if unset) - Works with both EvalCase objects and dict data

github-actions · 2026-03-03T00:42:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:47:05Z

Hi, this unfortunately became stale because we moved the python sdk into its own repo: https://github.com/braintrustdata/braintrust-sdk-python

We can think about re-introducing the change there.

Daniel Flynn (dflynn15) · 2026-02-05T19:30:19Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval(
"My Project",
data=[
{"input": "stable query", "expected": "..."}, # Uses global (3)
{"input": "flaky query", "expected": "...", "trial_count": 10}, # Override to 10EvalCase(input="deterministic", expected="...", trial_count=1), # Override to 1
],
task=my_task,
scores=[Factuality],
trial_count=3, # Global default
)

There is a corollary JS PR up to match it here: #1341

Allow data items to specify their own trial_count, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trial_count` field to `EvalCase` dataclass and TypedDict - Per-item trial_count takes precedence over global trial_count - Items without trial_count use the global value (or 1 if unset) - Works with both EvalCase objects and dict data

github-actions · 2026-03-03T00:42:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:47:05Z

Hi, this unfortunately became stale because we moved the python sdk into its own repo: https://github.com/braintrustdata/braintrust-sdk-python

We can think about re-introducing the change there.

Daniel Flynn (dflynn15) · 2026-02-05T19:30:19Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval(
"My Project",
data=[
{"input": "stable query", "expected": "..."}, # Uses global (3)
{"input": "flaky query", "expected": "...", "trial_count": 10}, # Override to 10EvalCase(input="deterministic", expected="...", trial_count=1), # Override to 1
],
task=my_task,
scores=[Factuality],
trial_count=3, # Global default
)

There is a corollary JS PR up to match it here: #1341

Allow data items to specify their own trial_count, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trial_count` field to `EvalCase` dataclass and TypedDict - Per-item trial_count takes precedence over global trial_count - Items without trial_count use the global value (or 1 if unset) - Works with both EvalCase objects and dict data

github-actions · 2026-03-03T00:42:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:47:05Z

Hi, this unfortunately became stale because we moved the python sdk into its own repo: https://github.com/braintrustdata/braintrust-sdk-python

We can think about re-introducing the change there.

Daniel Flynn (dflynn15) · 2026-02-05T19:30:19Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval(
"My Project",
data=[
{"input": "stable query", "expected": "..."}, # Uses global (3)
{"input": "flaky query", "expected": "...", "trial_count": 10}, # Override to 10EvalCase(input="deterministic", expected="...", trial_count=1), # Override to 1
],
task=my_task,
scores=[Factuality],
trial_count=3, # Global default
)

There is a corollary JS PR up to match it here: #1341

Allow data items to specify their own trial_count, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trial_count` field to `EvalCase` dataclass and TypedDict - Per-item trial_count takes precedence over global trial_count - Items without trial_count use the global value (or 1 if unset) - Works with both EvalCase objects and dict data

github-actions · 2026-03-03T00:42:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:47:05Z

Hi, this unfortunately became stale because we moved the python sdk into its own repo: https://github.com/braintrustdata/braintrust-sdk-python

We can think about re-introducing the change there.

Daniel Flynn (dflynn15) · 2026-02-05T19:30:19Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval(
"My Project",
data=[
{"input": "stable query", "expected": "..."}, # Uses global (3)
{"input": "flaky query", "expected": "...", "trial_count": 10}, # Override to 10EvalCase(input="deterministic", expected="...", trial_count=1), # Override to 1
],
task=my_task,
scores=[Factuality],
trial_count=3, # Global default
)

There is a corollary JS PR up to match it here: #1341

Allow data items to specify their own trial_count, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trial_count` field to `EvalCase` dataclass and TypedDict - Per-item trial_count takes precedence over global trial_count - Items without trial_count use the global value (or 1 if unset) - Works with both EvalCase objects and dict data

github-actions · 2026-03-03T00:42:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:47:05Z

Hi, this unfortunately became stale because we moved the python sdk into its own repo: https://github.com/braintrustdata/braintrust-sdk-python

We can think about re-introducing the change there.

Daniel Flynn (dflynn15) · 2026-02-05T19:30:19Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval(
"My Project",
data=[
{"input": "stable query", "expected": "..."}, # Uses global (3)
{"input": "flaky query", "expected": "...", "trial_count": 10}, # Override to 10EvalCase(input="deterministic", expected="...", trial_count=1), # Override to 1
],
task=my_task,
scores=[Factuality],
trial_count=3, # Global default
)

There is a corollary JS PR up to match it here: #1341

Allow data items to specify their own trial_count, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trial_count` field to `EvalCase` dataclass and TypedDict - Per-item trial_count takes precedence over global trial_count - Items without trial_count use the global value (or 1 if unset) - Works with both EvalCase objects and dict data

github-actions · 2026-03-03T00:42:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:47:05Z

Hi, this unfortunately became stale because we moved the python sdk into its own repo: https://github.com/braintrustdata/braintrust-sdk-python

We can think about re-introducing the change there.

Daniel Flynn (dflynn15) · 2026-02-05T19:30:19Z

Why?

Braintrust's Eval() function supports a trialCount parameter that runs each input multiple times to measure variance in non-deterministic LLM outputs. However, this setting applies globally to all inputs, which creates some (but minimal) friction in some evaluation workflows. For example:

Targeted Debugging is Expensive: When investigating a single flaky test case, you want to run it 10-20 times to understand the variance pattern. With global trialCount, this means running your entire suite 10-20 times, multiplying costs and wait time unnecessarily.
Mixed Determinism is Common: Real evaluation suites contain a mix of deterministic scenarios (math problems, factual lookups) and non-deterministic ones (creative writing, open-ended reasoning). Forcing the same trial count on both wastes resources.
Cost Scales Linearly: Every additional trial means another LLM API call. A global trialCount: 5 on a 100-item dataset means 500 API calls, even if only 10 items actually need variance analysis.

In order to address this, we've created a custom solution that I want to propose as a contribution. Specifically it, allows each data item to specify its own trialCount, overriding the global default. This gives users fine-grained control over where to invest their evaluation budget.

What?

Eval(
"My Project",
data=[
{"input": "stable query", "expected": "..."}, # Uses global (3)
{"input": "flaky query", "expected": "...", "trial_count": 10}, # Override to 10EvalCase(input="deterministic", expected="...", trial_count=1), # Override to 1
],
task=my_task,
scores=[Factuality],
trial_count=3, # Global default
)

There is a corollary JS PR up to match it here: #1341

Allow data items to specify their own trial_count, overriding the global evaluator setting. This enables targeted debugging of flaky test cases and mixed determinism scenarios without multiplying the entire suite. - Add optional `trial_count` field to `EvalCase` dataclass and TypedDict - Per-item trial_count takes precedence over global trial_count - Items without trial_count use the global value (or 1 if unset) - Works with both EvalCase objects and dict data

github-actions · 2026-03-03T00:42:57Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. If this PR is still relevant, please leave a comment, push an update, or remove the stale label. Thank you for your contributions!

Luca Forstner (lforst) · 2026-04-14T12:47:05Z

Hi, this unfortunately became stale because we moved the python sdk into its own repo: https://github.com/braintrustdata/braintrust-sdk-python

We can think about re-introducing the change there.