feat(api): migrate experiments to the API client

[Abhijeet Prasad (AbhiPrasad)]

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve#639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

[Abhijeet Prasad (AbhiPrasad)]

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

• get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
  generated bindings must not flatten it into a generic 4xx error);
• get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
  call-site audit's POST-based-logical-read class;
• additive backend fields surviving round-trips;
• SummarySuccess / SummarySkipped in logger.py;
• the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
  request targets and additive-field preservation, plus type tests — as the template for every
  subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.

[Abhijeet Prasad (AbhiPrasad)]

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve#639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

[Abhijeet Prasad (AbhiPrasad)]

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

• get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
  generated bindings must not flatten it into a generic 4xx error);
• get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
  call-site audit's POST-based-logical-read class;
• additive backend fields surviving round-trips;
• SummarySuccess / SummarySkipped in logger.py;
• the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
  request targets and additive-field preservation, plus type tests — as the template for every
  subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.

[Abhijeet Prasad (AbhiPrasad)]

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve#639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

[Abhijeet Prasad (AbhiPrasad)]

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

• get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
  generated bindings must not flatten it into a generic 4xx error);
• get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
  call-site audit's POST-based-logical-read class;
• additive backend fields surviving round-trips;
• SummarySuccess / SummarySkipped in logger.py;
• the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
  request targets and additive-field preservation, plus type tests — as the template for every
  subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.

[Abhijeet Prasad (AbhiPrasad)]

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve#639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

[Abhijeet Prasad (AbhiPrasad)]

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

• get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
  generated bindings must not flatten it into a generic 4xx error);
• get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
  call-site audit's POST-based-logical-read class;
• additive backend fields surviving round-trips;
• SummarySuccess / SummarySkipped in logger.py;
• the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
  request targets and additive-field preservation, plus type tests — as the template for every
  subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.

[Abhijeet Prasad (AbhiPrasad)]

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve#639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

[Abhijeet Prasad (AbhiPrasad)]

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

• get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
  generated bindings must not flatten it into a generic 4xx error);
• get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
  call-site audit's POST-based-logical-read class;
• additive backend fields surviving round-trips;
• SummarySuccess / SummarySkipped in logger.py;
• the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
  request targets and additive-field preservation, plus type tests — as the template for every
  subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.

[Abhijeet Prasad (AbhiPrasad)]

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve#639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

[Abhijeet Prasad (AbhiPrasad)]

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

• get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
  generated bindings must not flatten it into a generic 4xx error);
• get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
  call-site audit's POST-based-logical-read class;
• additive backend fields surviving round-trips;
• SummarySuccess / SummarySkipped in logger.py;
• the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
  request targets and additive-field preservation, plus type tests — as the template for every
  subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.

[Abhijeet Prasad (AbhiPrasad)]

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve#639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

[Abhijeet Prasad (AbhiPrasad)]

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

• get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
  generated bindings must not flatten it into a generic 4xx error);
• get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
  call-site audit's POST-based-logical-read class;
• additive backend fields surviving round-trips;
• SummarySuccess / SummarySkipped in logger.py;
• the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
  request targets and additive-field preservation, plus type tests — as the template for every
  subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.

[Abhijeet Prasad (AbhiPrasad)]

ref #673
supercedes #640

Migrate experiment lookup, base resolution, and comparison reads to the policy-aware API workflow with safe-read retries and typed responses.

Resolve#639 by retrying transient comparison failures and raising typed API errors after retries are exhausted instead of presenting failures as empty scores. Represent successful and intentionally skipped summaries explicitly.

[Abhijeet Prasad (AbhiPrasad)]

Closing in favor of the generated API client — tracked in #683.

Why: this PR returns frozen dataclasses (ExperimentRecord, ExperimentComparison, …) with
from_dict validation and a raw verbatim-body passthrough. #683 generates TypedDicts from
braintrust-openapi. Merging this would put
two incompatible return conventions in braintrust.api, so the convention question is settled up
front instead of after the fact: generated TypedDicts, with additive-field tolerance coming from
non-closed TypedDicts rather than a raw field. Recorded as open decision 7 in #683.

Nothing here is wasted — the branch stays readable and is the reference implementation for the redo.
#683 records these as requirements it must satisfy:

• get_base() returning None on HTTP 400 (a fresh project's first eval always takes this path, so
  generated bindings must not flatten it into a generic 4xx error);
• get() / get_base() / compare() classified RetryMode.SAFE_READ — the concrete case of the
  call-site audit's POST-based-logical-read class;
• additive backend fields surviving round-trips;
• SummarySuccess / SummarySkipped in logger.py;
• the coverage shape — real-backend VCR cassettes plus a local http.server test asserting exact
  request targets and additive-field preservation, plus type tests — as the template for every
  subsequent resource migration.

Note on #639: this PR was its only fix in flight, and that bug is user-facing — a failed
comparison read currently looks identical to a successful experiment with no scores. #683 tracks two
ways to unpark it: re-land only the product-level half now (the strict-by-default semantics and the
discriminated SummarySuccess/SummarySkipped/SummaryFailed result in logger.py, which are
eval-summary types rather than REST wire types and so do not touch the convention decision), or wait
for generated Experiments bindings in step 3.