fix(json): stop a skipped sub-score being averaged as a mismatch

[Arthi Arumugam (arthi-arumugam-git)]

The problem

Score documents the contract:

If the score is None, the evaluation is considered to be skipped.

JSONDiff.json_diff filters those out before averaging, but the two branches then divide by
different things:

# dictbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /len(base_scores) # count AFTER filtering# listbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /max(len(o1), len(o2)) # count BEFORE filtering

In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.

The same skip therefore lands differently depending on the container:

input	scorer skips one value	score on main
{"a": "skip", "b": "same"}	yes	1.0
["skip", "same"]	yes	0.5

Same values, same scorer, same skip.

This is reachable with any scorer that can abstain, which is the case the None score exists
for: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.

Second, smaller issue

The dict branch divides by len(base_scores) with no guard. An object whose comparisons are
all skipped leaves that list empty and raises ZeroDivisionError. The empty-object case is
handled above it, so this needs a non-empty object rather than an edge case in the inputs.

The change

Both branches now drop skipped comparisons from the denominator, and return None when
nothing is left to average, which propagates the skip upward instead of inventing a number or
raising.

The list denominator stays max(len(o1), len(o2)) minus the skips. That distinction is
deliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what max(...) contributes over the zip. Only the skips come out.

Tests

py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail on main:

FAILED test_skipped_element_in_a_list_is_not_scored_as_zero
FAILED test_list_and_dict_treat_an_identical_skip_the_same_way
FAILED test_a_fully_skipped_object_reports_a_skip_rather_than_raising
FAILED test_a_fully_skipped_list_reports_a_skip_rather_than_raising

The other four pin behaviour that must not move: missing elements are still penalised
(["a"] vs ["a", "b"] is still 0.5), and unskipped lists, dicts and empty containers score
exactly as before.

py/autoevals/test_json.py: 4 passed. Across py/autoevals the results are identical with
and without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
black and isort clean at line-length 119.

[Arthi Arumugam (arthi-arumugam-git)]

Flagging in case it is not visible from your side: the workflow run on this PR needs maintainer approval before anything executes, so the checks here are empty rather than failing.

Locally the eight added cases pass, four of which fail on main, and the rest of the py/autoevals suite is byte-identical with and without the change (the pre-existing failures there need API keys). Happy for CI to run whenever someone can enable it, and to fix anything it turns up.

[Arthi Arumugam (arthi-arumugam-git)]

The red checks that showed up on this PR today are not test failures. The six runs were created on 2 Aug, sat waiting for maintainer approval, and GitHub expired them at the 30 day mark (created 2026-08-02 02:33 UTC, marked failed 2026-09-01 02:36 UTC). All six have zero jobs and no logs, so nothing actually ran. The same thing has happened to the other fork PRs on this repo, so it does not look specific to this branch.

I re-checked the change locally today on Python 3.13:

• the 8 new tests plus the existing test_json.py pass, 12 passed
• with json.py put back to main and the tests kept, 4 fail, the same 4 listed in the description
• the rest of py/autoevals gives an identical 28 failures with and without this change (all of them missing API keys or missing optional modules)
• black and ruff are clean on both changed files

The branch is still level with main, so there is nothing to rebase. Since these runs have already expired, a fresh one would need to be queued before it can be approved. I can push an empty commit to trigger that if it is useful, just say the word.

[Arthi Arumugam (arthi-arumugam-git)]

The problem

Score documents the contract:

If the score is None, the evaluation is considered to be skipped.

JSONDiff.json_diff filters those out before averaging, but the two branches then divide by
different things:

# dictbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /len(base_scores) # count AFTER filtering# listbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /max(len(o1), len(o2)) # count BEFORE filtering

In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.

The same skip therefore lands differently depending on the container:

input	scorer skips one value	score on main
{"a": "skip", "b": "same"}	yes	1.0
["skip", "same"]	yes	0.5

Same values, same scorer, same skip.

This is reachable with any scorer that can abstain, which is the case the None score exists
for: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.

Second, smaller issue

The dict branch divides by len(base_scores) with no guard. An object whose comparisons are
all skipped leaves that list empty and raises ZeroDivisionError. The empty-object case is
handled above it, so this needs a non-empty object rather than an edge case in the inputs.

The change

Both branches now drop skipped comparisons from the denominator, and return None when
nothing is left to average, which propagates the skip upward instead of inventing a number or
raising.

The list denominator stays max(len(o1), len(o2)) minus the skips. That distinction is
deliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what max(...) contributes over the zip. Only the skips come out.

Tests

py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail on main:

FAILED test_skipped_element_in_a_list_is_not_scored_as_zero
FAILED test_list_and_dict_treat_an_identical_skip_the_same_way
FAILED test_a_fully_skipped_object_reports_a_skip_rather_than_raising
FAILED test_a_fully_skipped_list_reports_a_skip_rather_than_raising

The other four pin behaviour that must not move: missing elements are still penalised
(["a"] vs ["a", "b"] is still 0.5), and unskipped lists, dicts and empty containers score
exactly as before.

py/autoevals/test_json.py: 4 passed. Across py/autoevals the results are identical with
and without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
black and isort clean at line-length 119.

[Arthi Arumugam (arthi-arumugam-git)]

Flagging in case it is not visible from your side: the workflow run on this PR needs maintainer approval before anything executes, so the checks here are empty rather than failing.

Locally the eight added cases pass, four of which fail on main, and the rest of the py/autoevals suite is byte-identical with and without the change (the pre-existing failures there need API keys). Happy for CI to run whenever someone can enable it, and to fix anything it turns up.

[Arthi Arumugam (arthi-arumugam-git)]

The red checks that showed up on this PR today are not test failures. The six runs were created on 2 Aug, sat waiting for maintainer approval, and GitHub expired them at the 30 day mark (created 2026-08-02 02:33 UTC, marked failed 2026-09-01 02:36 UTC). All six have zero jobs and no logs, so nothing actually ran. The same thing has happened to the other fork PRs on this repo, so it does not look specific to this branch.

I re-checked the change locally today on Python 3.13:

• the 8 new tests plus the existing test_json.py pass, 12 passed
• with json.py put back to main and the tests kept, 4 fail, the same 4 listed in the description
• the rest of py/autoevals gives an identical 28 failures with and without this change (all of them missing API keys or missing optional modules)
• black and ruff are clean on both changed files

The branch is still level with main, so there is nothing to rebase. Since these runs have already expired, a fresh one would need to be queued before it can be approved. I can push an empty commit to trigger that if it is useful, just say the word.

[Arthi Arumugam (arthi-arumugam-git)]

The problem

Score documents the contract:

If the score is None, the evaluation is considered to be skipped.

JSONDiff.json_diff filters those out before averaging, but the two branches then divide by
different things:

# dictbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /len(base_scores) # count AFTER filtering# listbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /max(len(o1), len(o2)) # count BEFORE filtering

In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.

The same skip therefore lands differently depending on the container:

input	scorer skips one value	score on main
{"a": "skip", "b": "same"}	yes	1.0
["skip", "same"]	yes	0.5

Same values, same scorer, same skip.

This is reachable with any scorer that can abstain, which is the case the None score exists
for: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.

Second, smaller issue

The dict branch divides by len(base_scores) with no guard. An object whose comparisons are
all skipped leaves that list empty and raises ZeroDivisionError. The empty-object case is
handled above it, so this needs a non-empty object rather than an edge case in the inputs.

The change

Both branches now drop skipped comparisons from the denominator, and return None when
nothing is left to average, which propagates the skip upward instead of inventing a number or
raising.

The list denominator stays max(len(o1), len(o2)) minus the skips. That distinction is
deliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what max(...) contributes over the zip. Only the skips come out.

Tests

py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail on main:

FAILED test_skipped_element_in_a_list_is_not_scored_as_zero
FAILED test_list_and_dict_treat_an_identical_skip_the_same_way
FAILED test_a_fully_skipped_object_reports_a_skip_rather_than_raising
FAILED test_a_fully_skipped_list_reports_a_skip_rather_than_raising

The other four pin behaviour that must not move: missing elements are still penalised
(["a"] vs ["a", "b"] is still 0.5), and unskipped lists, dicts and empty containers score
exactly as before.

py/autoevals/test_json.py: 4 passed. Across py/autoevals the results are identical with
and without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
black and isort clean at line-length 119.

[Arthi Arumugam (arthi-arumugam-git)]

Flagging in case it is not visible from your side: the workflow run on this PR needs maintainer approval before anything executes, so the checks here are empty rather than failing.

Locally the eight added cases pass, four of which fail on main, and the rest of the py/autoevals suite is byte-identical with and without the change (the pre-existing failures there need API keys). Happy for CI to run whenever someone can enable it, and to fix anything it turns up.

[Arthi Arumugam (arthi-arumugam-git)]

The red checks that showed up on this PR today are not test failures. The six runs were created on 2 Aug, sat waiting for maintainer approval, and GitHub expired them at the 30 day mark (created 2026-08-02 02:33 UTC, marked failed 2026-09-01 02:36 UTC). All six have zero jobs and no logs, so nothing actually ran. The same thing has happened to the other fork PRs on this repo, so it does not look specific to this branch.

I re-checked the change locally today on Python 3.13:

• the 8 new tests plus the existing test_json.py pass, 12 passed
• with json.py put back to main and the tests kept, 4 fail, the same 4 listed in the description
• the rest of py/autoevals gives an identical 28 failures with and without this change (all of them missing API keys or missing optional modules)
• black and ruff are clean on both changed files

The branch is still level with main, so there is nothing to rebase. Since these runs have already expired, a fresh one would need to be queued before it can be approved. I can push an empty commit to trigger that if it is useful, just say the word.

[Arthi Arumugam (arthi-arumugam-git)]

The problem

Score documents the contract:

If the score is None, the evaluation is considered to be skipped.

JSONDiff.json_diff filters those out before averaging, but the two branches then divide by
different things:

# dictbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /len(base_scores) # count AFTER filtering# listbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /max(len(o1), len(o2)) # count BEFORE filtering

In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.

The same skip therefore lands differently depending on the container:

input	scorer skips one value	score on main
{"a": "skip", "b": "same"}	yes	1.0
["skip", "same"]	yes	0.5

Same values, same scorer, same skip.

This is reachable with any scorer that can abstain, which is the case the None score exists
for: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.

Second, smaller issue

The dict branch divides by len(base_scores) with no guard. An object whose comparisons are
all skipped leaves that list empty and raises ZeroDivisionError. The empty-object case is
handled above it, so this needs a non-empty object rather than an edge case in the inputs.

The change

Both branches now drop skipped comparisons from the denominator, and return None when
nothing is left to average, which propagates the skip upward instead of inventing a number or
raising.

The list denominator stays max(len(o1), len(o2)) minus the skips. That distinction is
deliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what max(...) contributes over the zip. Only the skips come out.

Tests

py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail on main:

FAILED test_skipped_element_in_a_list_is_not_scored_as_zero
FAILED test_list_and_dict_treat_an_identical_skip_the_same_way
FAILED test_a_fully_skipped_object_reports_a_skip_rather_than_raising
FAILED test_a_fully_skipped_list_reports_a_skip_rather_than_raising

The other four pin behaviour that must not move: missing elements are still penalised
(["a"] vs ["a", "b"] is still 0.5), and unskipped lists, dicts and empty containers score
exactly as before.

py/autoevals/test_json.py: 4 passed. Across py/autoevals the results are identical with
and without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
black and isort clean at line-length 119.

[Arthi Arumugam (arthi-arumugam-git)]

Flagging in case it is not visible from your side: the workflow run on this PR needs maintainer approval before anything executes, so the checks here are empty rather than failing.

Locally the eight added cases pass, four of which fail on main, and the rest of the py/autoevals suite is byte-identical with and without the change (the pre-existing failures there need API keys). Happy for CI to run whenever someone can enable it, and to fix anything it turns up.

[Arthi Arumugam (arthi-arumugam-git)]

The red checks that showed up on this PR today are not test failures. The six runs were created on 2 Aug, sat waiting for maintainer approval, and GitHub expired them at the 30 day mark (created 2026-08-02 02:33 UTC, marked failed 2026-09-01 02:36 UTC). All six have zero jobs and no logs, so nothing actually ran. The same thing has happened to the other fork PRs on this repo, so it does not look specific to this branch.

I re-checked the change locally today on Python 3.13:

• the 8 new tests plus the existing test_json.py pass, 12 passed
• with json.py put back to main and the tests kept, 4 fail, the same 4 listed in the description
• the rest of py/autoevals gives an identical 28 failures with and without this change (all of them missing API keys or missing optional modules)
• black and ruff are clean on both changed files

The branch is still level with main, so there is nothing to rebase. Since these runs have already expired, a fresh one would need to be queued before it can be approved. I can push an empty commit to trigger that if it is useful, just say the word.

[Arthi Arumugam (arthi-arumugam-git)]

The problem

Score documents the contract:

If the score is None, the evaluation is considered to be skipped.

JSONDiff.json_diff filters those out before averaging, but the two branches then divide by
different things:

# dictbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /len(base_scores) # count AFTER filtering# listbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /max(len(o1), len(o2)) # count BEFORE filtering

In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.

The same skip therefore lands differently depending on the container:

input	scorer skips one value	score on main
{"a": "skip", "b": "same"}	yes	1.0
["skip", "same"]	yes	0.5

Same values, same scorer, same skip.

This is reachable with any scorer that can abstain, which is the case the None score exists
for: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.

Second, smaller issue

The dict branch divides by len(base_scores) with no guard. An object whose comparisons are
all skipped leaves that list empty and raises ZeroDivisionError. The empty-object case is
handled above it, so this needs a non-empty object rather than an edge case in the inputs.

The change

Both branches now drop skipped comparisons from the denominator, and return None when
nothing is left to average, which propagates the skip upward instead of inventing a number or
raising.

The list denominator stays max(len(o1), len(o2)) minus the skips. That distinction is
deliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what max(...) contributes over the zip. Only the skips come out.

Tests

py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail on main:

FAILED test_skipped_element_in_a_list_is_not_scored_as_zero
FAILED test_list_and_dict_treat_an_identical_skip_the_same_way
FAILED test_a_fully_skipped_object_reports_a_skip_rather_than_raising
FAILED test_a_fully_skipped_list_reports_a_skip_rather_than_raising

The other four pin behaviour that must not move: missing elements are still penalised
(["a"] vs ["a", "b"] is still 0.5), and unskipped lists, dicts and empty containers score
exactly as before.

py/autoevals/test_json.py: 4 passed. Across py/autoevals the results are identical with
and without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
black and isort clean at line-length 119.

[Arthi Arumugam (arthi-arumugam-git)]

Flagging in case it is not visible from your side: the workflow run on this PR needs maintainer approval before anything executes, so the checks here are empty rather than failing.

Locally the eight added cases pass, four of which fail on main, and the rest of the py/autoevals suite is byte-identical with and without the change (the pre-existing failures there need API keys). Happy for CI to run whenever someone can enable it, and to fix anything it turns up.

[Arthi Arumugam (arthi-arumugam-git)]

The red checks that showed up on this PR today are not test failures. The six runs were created on 2 Aug, sat waiting for maintainer approval, and GitHub expired them at the 30 day mark (created 2026-08-02 02:33 UTC, marked failed 2026-09-01 02:36 UTC). All six have zero jobs and no logs, so nothing actually ran. The same thing has happened to the other fork PRs on this repo, so it does not look specific to this branch.

I re-checked the change locally today on Python 3.13:

• the 8 new tests plus the existing test_json.py pass, 12 passed
• with json.py put back to main and the tests kept, 4 fail, the same 4 listed in the description
• the rest of py/autoevals gives an identical 28 failures with and without this change (all of them missing API keys or missing optional modules)
• black and ruff are clean on both changed files

The branch is still level with main, so there is nothing to rebase. Since these runs have already expired, a fresh one would need to be queued before it can be approved. I can push an empty commit to trigger that if it is useful, just say the word.

[Arthi Arumugam (arthi-arumugam-git)]

The problem

Score documents the contract:

If the score is None, the evaluation is considered to be skipped.

JSONDiff.json_diff filters those out before averaging, but the two branches then divide by
different things:

# dictbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /len(base_scores) # count AFTER filtering# listbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /max(len(o1), len(o2)) # count BEFORE filtering

In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.

The same skip therefore lands differently depending on the container:

input	scorer skips one value	score on main
{"a": "skip", "b": "same"}	yes	1.0
["skip", "same"]	yes	0.5

Same values, same scorer, same skip.

This is reachable with any scorer that can abstain, which is the case the None score exists
for: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.

Second, smaller issue

The dict branch divides by len(base_scores) with no guard. An object whose comparisons are
all skipped leaves that list empty and raises ZeroDivisionError. The empty-object case is
handled above it, so this needs a non-empty object rather than an edge case in the inputs.

The change

Both branches now drop skipped comparisons from the denominator, and return None when
nothing is left to average, which propagates the skip upward instead of inventing a number or
raising.

The list denominator stays max(len(o1), len(o2)) minus the skips. That distinction is
deliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what max(...) contributes over the zip. Only the skips come out.

Tests

py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail on main:

FAILED test_skipped_element_in_a_list_is_not_scored_as_zero
FAILED test_list_and_dict_treat_an_identical_skip_the_same_way
FAILED test_a_fully_skipped_object_reports_a_skip_rather_than_raising
FAILED test_a_fully_skipped_list_reports_a_skip_rather_than_raising

The other four pin behaviour that must not move: missing elements are still penalised
(["a"] vs ["a", "b"] is still 0.5), and unskipped lists, dicts and empty containers score
exactly as before.

py/autoevals/test_json.py: 4 passed. Across py/autoevals the results are identical with
and without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
black and isort clean at line-length 119.

[Arthi Arumugam (arthi-arumugam-git)]

Flagging in case it is not visible from your side: the workflow run on this PR needs maintainer approval before anything executes, so the checks here are empty rather than failing.

Locally the eight added cases pass, four of which fail on main, and the rest of the py/autoevals suite is byte-identical with and without the change (the pre-existing failures there need API keys). Happy for CI to run whenever someone can enable it, and to fix anything it turns up.

[Arthi Arumugam (arthi-arumugam-git)]

The red checks that showed up on this PR today are not test failures. The six runs were created on 2 Aug, sat waiting for maintainer approval, and GitHub expired them at the 30 day mark (created 2026-08-02 02:33 UTC, marked failed 2026-09-01 02:36 UTC). All six have zero jobs and no logs, so nothing actually ran. The same thing has happened to the other fork PRs on this repo, so it does not look specific to this branch.

I re-checked the change locally today on Python 3.13:

• the 8 new tests plus the existing test_json.py pass, 12 passed
• with json.py put back to main and the tests kept, 4 fail, the same 4 listed in the description
• the rest of py/autoevals gives an identical 28 failures with and without this change (all of them missing API keys or missing optional modules)
• black and ruff are clean on both changed files

The branch is still level with main, so there is nothing to rebase. Since these runs have already expired, a fresh one would need to be queued before it can be approved. I can push an empty commit to trigger that if it is useful, just say the word.

[Arthi Arumugam (arthi-arumugam-git)]

The problem

Score documents the contract:

If the score is None, the evaluation is considered to be skipped.

JSONDiff.json_diff filters those out before averaging, but the two branches then divide by
different things:

# dictbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /len(base_scores) # count AFTER filtering# listbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /max(len(o1), len(o2)) # count BEFORE filtering

In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.

The same skip therefore lands differently depending on the container:

input	scorer skips one value	score on main
{"a": "skip", "b": "same"}	yes	1.0
["skip", "same"]	yes	0.5

Same values, same scorer, same skip.

This is reachable with any scorer that can abstain, which is the case the None score exists
for: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.

Second, smaller issue

The dict branch divides by len(base_scores) with no guard. An object whose comparisons are
all skipped leaves that list empty and raises ZeroDivisionError. The empty-object case is
handled above it, so this needs a non-empty object rather than an edge case in the inputs.

The change

Both branches now drop skipped comparisons from the denominator, and return None when
nothing is left to average, which propagates the skip upward instead of inventing a number or
raising.

The list denominator stays max(len(o1), len(o2)) minus the skips. That distinction is
deliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what max(...) contributes over the zip. Only the skips come out.

Tests

py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail on main:

FAILED test_skipped_element_in_a_list_is_not_scored_as_zero
FAILED test_list_and_dict_treat_an_identical_skip_the_same_way
FAILED test_a_fully_skipped_object_reports_a_skip_rather_than_raising
FAILED test_a_fully_skipped_list_reports_a_skip_rather_than_raising

The other four pin behaviour that must not move: missing elements are still penalised
(["a"] vs ["a", "b"] is still 0.5), and unskipped lists, dicts and empty containers score
exactly as before.

py/autoevals/test_json.py: 4 passed. Across py/autoevals the results are identical with
and without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
black and isort clean at line-length 119.

[Arthi Arumugam (arthi-arumugam-git)]

Flagging in case it is not visible from your side: the workflow run on this PR needs maintainer approval before anything executes, so the checks here are empty rather than failing.

Locally the eight added cases pass, four of which fail on main, and the rest of the py/autoevals suite is byte-identical with and without the change (the pre-existing failures there need API keys). Happy for CI to run whenever someone can enable it, and to fix anything it turns up.

[Arthi Arumugam (arthi-arumugam-git)]

The red checks that showed up on this PR today are not test failures. The six runs were created on 2 Aug, sat waiting for maintainer approval, and GitHub expired them at the 30 day mark (created 2026-08-02 02:33 UTC, marked failed 2026-09-01 02:36 UTC). All six have zero jobs and no logs, so nothing actually ran. The same thing has happened to the other fork PRs on this repo, so it does not look specific to this branch.

I re-checked the change locally today on Python 3.13:

• the 8 new tests plus the existing test_json.py pass, 12 passed
• with json.py put back to main and the tests kept, 4 fail, the same 4 listed in the description
• the rest of py/autoevals gives an identical 28 failures with and without this change (all of them missing API keys or missing optional modules)
• black and ruff are clean on both changed files

The branch is still level with main, so there is nothing to rebase. Since these runs have already expired, a fresh one would need to be queued before it can be approved. I can push an empty commit to trigger that if it is useful, just say the word.

[Arthi Arumugam (arthi-arumugam-git)]

The problem

Score documents the contract:

If the score is None, the evaluation is considered to be skipped.

JSONDiff.json_diff filters those out before averaging, but the two branches then divide by
different things:

# dictbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /len(base_scores) # count AFTER filtering# listbase_scores= [sforsinbase_scoresifsisnotNone]
returnsum(base_scores) /max(len(o1), len(o2)) # count BEFORE filtering

In an object a skipped key is removed from the numerator and the denominator, so it is
ignored, which matches the documented meaning. In an array it is removed from the numerator
only, so it is averaged in as a zero.

The same skip therefore lands differently depending on the container:

input	scorer skips one value	score on main
{"a": "skip", "b": "same"}	yes	1.0
["skip", "same"]	yes	0.5

Same values, same scorer, same skip.

This is reachable with any scorer that can abstain, which is the case the None score exists
for: an LLM judge that declines to answer, a scorer that cannot parse a field. The array
result is silently lower and nothing raises.

Second, smaller issue

The dict branch divides by len(base_scores) with no guard. An object whose comparisons are
all skipped leaves that list empty and raises ZeroDivisionError. The empty-object case is
handled above it, so this needs a non-empty object rather than an edge case in the inputs.

The change

Both branches now drop skipped comparisons from the denominator, and return None when
nothing is left to average, which propagates the skip upward instead of inventing a number or
raising.

The list denominator stays max(len(o1), len(o2)) minus the skips. That distinction is
deliberate: an element with no counterpart is a genuine difference and must stay counted,
which is what max(...) contributes over the zip. Only the skips come out.

Tests

py/autoevals/test_json_skipped_scores.py, eight tests, four of which fail on main:

FAILED test_skipped_element_in_a_list_is_not_scored_as_zero
FAILED test_list_and_dict_treat_an_identical_skip_the_same_way
FAILED test_a_fully_skipped_object_reports_a_skip_rather_than_raising
FAILED test_a_fully_skipped_list_reports_a_skip_rather_than_raising

The other four pin behaviour that must not move: missing elements are still penalised
(["a"] vs ["a", "b"] is still 0.5), and unskipped lists, dicts and empty containers score
exactly as before.

py/autoevals/test_json.py: 4 passed. Across py/autoevals the results are identical with
and without this diff (21 failed, 39 passed, 8 errors, all needing API keys or litellm).
black and isort clean at line-length 119.

[Arthi Arumugam (arthi-arumugam-git)]

Flagging in case it is not visible from your side: the workflow run on this PR needs maintainer approval before anything executes, so the checks here are empty rather than failing.

Locally the eight added cases pass, four of which fail on main, and the rest of the py/autoevals suite is byte-identical with and without the change (the pre-existing failures there need API keys). Happy for CI to run whenever someone can enable it, and to fix anything it turns up.

[Arthi Arumugam (arthi-arumugam-git)]

The red checks that showed up on this PR today are not test failures. The six runs were created on 2 Aug, sat waiting for maintainer approval, and GitHub expired them at the 30 day mark (created 2026-08-02 02:33 UTC, marked failed 2026-09-01 02:36 UTC). All six have zero jobs and no logs, so nothing actually ran. The same thing has happened to the other fork PRs on this repo, so it does not look specific to this branch.

I re-checked the change locally today on Python 3.13:

• the 8 new tests plus the existing test_json.py pass, 12 passed
• with json.py put back to main and the tests kept, 4 fail, the same 4 listed in the description
• the rest of py/autoevals gives an identical 28 failures with and without this change (all of them missing API keys or missing optional modules)
• black and ruff are clean on both changed files

The branch is still level with main, so there is nothing to rebase. Since these runs have already expired, a fresh one would need to be queued before it can be approved. I can push an empty commit to trigger that if it is useful, just say the word.