Improve pytest parsing for beginner-friendly evaluation feedback

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item
test_program.py F [100%]
=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>
def test_weekly_todo(monkeypatch, capsys):
user_input = iter([
"Laundry",
"Groceries",
"Gym",
"Study",
"Work",
"Hike",
"Rest",
])
run_program_with(monkeypatch, user_input)
> assert capsys.readouterr().out == (
"Weekly to-do list for the 7 days of the week:\n"
"Mon: Laundry\n"
"Tues: Groceries\n"
"Wed Gym\n"
"Thur: Study\n"
"Fri: Work\n"
"Sat: Hike\n"
"Sun: Rest\n"
)
E AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E Skipping 34 identical leading characters in diff, use -v to show
E - f the week:
E + f the weeke:
E ? +
E Mon: Laundry
E Tues: Groceries...
E
E ...Full output truncated (5 lines hidden), use '-vv' to show
test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
state: 'failed',
tests_run: 1,
passed: 0,
failed: 1,
errors: 0,
no_tests_collected: false,
exit_code: 1,
failure_details: [
{
test_case: 'test_weekly_todo',
expected: '',
received: '',
error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
rawout: '============================= test session starts ==============================\n' +
'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
'collected 1 item\n' +
'\n' +
'test_program.py F [100%]\n' +
'\n' +
'=================================== FAILURES ===================================\n' +
'_______________________________ test_weekly_todo _______________________________\n' +
'\n' +
'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
'\n' +
' def test_weekly_todo(monkeypatch, capsys):\n' +
' user_input = iter([\n' +
' "Laundry",\n' +
' "Groceries",\n' +
' "Gym",\n' +
' "Study",\n' +
' "Work",\n' +
' "Hike",\n' +
' "Rest",\n' +
' ])\n' +
' \n' +
' run_program_with(monkeypatch, user_input)\n' +
' \n' +
'> assert capsys.readouterr().out == (\n' +
' "Weekly to-do list for the 7 days of the week:\\n"\n' +
' "Mon: Laundry\\n"\n' +
' "Tues: Groceries\\n"\n' +
' "Wed Gym\\n"\n' +
' "Thur: Study\\n"\n' +
' "Fri: Work\\n"\n' +
' "Sat: Hike\\n"\n' +
' "Sun: Rest\\n"\n' +
' )\n' +
"E AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
'E \n' +
'E Skipping 34 identical leading characters in diff, use -v to show\n' +
'E - f the week:\n' +
'E + f the weeke:\n' +
'E ? +\n' +
'E Mon: Laundry\n' +
'E Tues: Groceries...\n' +
'E \n' +
"E ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
'\n' +
'test_program.py:29: AssertionError\n' +
'=========================== short test summary info ============================\n' +
"FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
'============================== 1 failed in 0.02s ===============================\n'
}
],
compilation_error: '',
runtime_error: '',
execution_time_exceeded: false,
memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junitxml file that contains information about the tests that were run. if that is something you want to look into.

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item
test_program.py F [100%]
=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>
def test_weekly_todo(monkeypatch, capsys):
user_input = iter([
"Laundry",
"Groceries",
"Gym",
"Study",
"Work",
"Hike",
"Rest",
])
run_program_with(monkeypatch, user_input)
> assert capsys.readouterr().out == (
"Weekly to-do list for the 7 days of the week:\n"
"Mon: Laundry\n"
"Tues: Groceries\n"
"Wed Gym\n"
"Thur: Study\n"
"Fri: Work\n"
"Sat: Hike\n"
"Sun: Rest\n"
)
E AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E Skipping 34 identical leading characters in diff, use -v to show
E - f the week:
E + f the weeke:
E ? +
E Mon: Laundry
E Tues: Groceries...
E
E ...Full output truncated (5 lines hidden), use '-vv' to show
test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
state: 'failed',
tests_run: 1,
passed: 0,
failed: 1,
errors: 0,
no_tests_collected: false,
exit_code: 1,
failure_details: [
{
test_case: 'test_weekly_todo',
expected: '',
received: '',
error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
rawout: '============================= test session starts ==============================\n' +
'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
'collected 1 item\n' +
'\n' +
'test_program.py F [100%]\n' +
'\n' +
'=================================== FAILURES ===================================\n' +
'_______________________________ test_weekly_todo _______________________________\n' +
'\n' +
'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
'\n' +
' def test_weekly_todo(monkeypatch, capsys):\n' +
' user_input = iter([\n' +
' "Laundry",\n' +
' "Groceries",\n' +
' "Gym",\n' +
' "Study",\n' +
' "Work",\n' +
' "Hike",\n' +
' "Rest",\n' +
' ])\n' +
' \n' +
' run_program_with(monkeypatch, user_input)\n' +
' \n' +
'> assert capsys.readouterr().out == (\n' +
' "Weekly to-do list for the 7 days of the week:\\n"\n' +
' "Mon: Laundry\\n"\n' +
' "Tues: Groceries\\n"\n' +
' "Wed Gym\\n"\n' +
' "Thur: Study\\n"\n' +
' "Fri: Work\\n"\n' +
' "Sat: Hike\\n"\n' +
' "Sun: Rest\\n"\n' +
' )\n' +
"E AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
'E \n' +
'E Skipping 34 identical leading characters in diff, use -v to show\n' +
'E - f the week:\n' +
'E + f the weeke:\n' +
'E ? +\n' +
'E Mon: Laundry\n' +
'E Tues: Groceries...\n' +
'E \n' +
"E ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
'\n' +
'test_program.py:29: AssertionError\n' +
'=========================== short test summary info ============================\n' +
"FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
'============================== 1 failed in 0.02s ===============================\n'
}
],
compilation_error: '',
runtime_error: '',
execution_time_exceeded: false,
memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junitxml file that contains information about the tests that were run. if that is something you want to look into.

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item
test_program.py F [100%]
=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>
def test_weekly_todo(monkeypatch, capsys):
user_input = iter([
"Laundry",
"Groceries",
"Gym",
"Study",
"Work",
"Hike",
"Rest",
])
run_program_with(monkeypatch, user_input)
> assert capsys.readouterr().out == (
"Weekly to-do list for the 7 days of the week:\n"
"Mon: Laundry\n"
"Tues: Groceries\n"
"Wed Gym\n"
"Thur: Study\n"
"Fri: Work\n"
"Sat: Hike\n"
"Sun: Rest\n"
)
E AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E Skipping 34 identical leading characters in diff, use -v to show
E - f the week:
E + f the weeke:
E ? +
E Mon: Laundry
E Tues: Groceries...
E
E ...Full output truncated (5 lines hidden), use '-vv' to show
test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
state: 'failed',
tests_run: 1,
passed: 0,
failed: 1,
errors: 0,
no_tests_collected: false,
exit_code: 1,
failure_details: [
{
test_case: 'test_weekly_todo',
expected: '',
received: '',
error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
rawout: '============================= test session starts ==============================\n' +
'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
'collected 1 item\n' +
'\n' +
'test_program.py F [100%]\n' +
'\n' +
'=================================== FAILURES ===================================\n' +
'_______________________________ test_weekly_todo _______________________________\n' +
'\n' +
'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
'\n' +
' def test_weekly_todo(monkeypatch, capsys):\n' +
' user_input = iter([\n' +
' "Laundry",\n' +
' "Groceries",\n' +
' "Gym",\n' +
' "Study",\n' +
' "Work",\n' +
' "Hike",\n' +
' "Rest",\n' +
' ])\n' +
' \n' +
' run_program_with(monkeypatch, user_input)\n' +
' \n' +
'> assert capsys.readouterr().out == (\n' +
' "Weekly to-do list for the 7 days of the week:\\n"\n' +
' "Mon: Laundry\\n"\n' +
' "Tues: Groceries\\n"\n' +
' "Wed Gym\\n"\n' +
' "Thur: Study\\n"\n' +
' "Fri: Work\\n"\n' +
' "Sat: Hike\\n"\n' +
' "Sun: Rest\\n"\n' +
' )\n' +
"E AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
'E \n' +
'E Skipping 34 identical leading characters in diff, use -v to show\n' +
'E - f the week:\n' +
'E + f the weeke:\n' +
'E ? +\n' +
'E Mon: Laundry\n' +
'E Tues: Groceries...\n' +
'E \n' +
"E ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
'\n' +
'test_program.py:29: AssertionError\n' +
'=========================== short test summary info ============================\n' +
"FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
'============================== 1 failed in 0.02s ===============================\n'
}
],
compilation_error: '',
runtime_error: '',
execution_time_exceeded: false,
memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junitxml file that contains information about the tests that were run. if that is something you want to look into.

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item
test_program.py F [100%]
=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>
def test_weekly_todo(monkeypatch, capsys):
user_input = iter([
"Laundry",
"Groceries",
"Gym",
"Study",
"Work",
"Hike",
"Rest",
])
run_program_with(monkeypatch, user_input)
> assert capsys.readouterr().out == (
"Weekly to-do list for the 7 days of the week:\n"
"Mon: Laundry\n"
"Tues: Groceries\n"
"Wed Gym\n"
"Thur: Study\n"
"Fri: Work\n"
"Sat: Hike\n"
"Sun: Rest\n"
)
E AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E Skipping 34 identical leading characters in diff, use -v to show
E - f the week:
E + f the weeke:
E ? +
E Mon: Laundry
E Tues: Groceries...
E
E ...Full output truncated (5 lines hidden), use '-vv' to show
test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
state: 'failed',
tests_run: 1,
passed: 0,
failed: 1,
errors: 0,
no_tests_collected: false,
exit_code: 1,
failure_details: [
{
test_case: 'test_weekly_todo',
expected: '',
received: '',
error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
rawout: '============================= test session starts ==============================\n' +
'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
'collected 1 item\n' +
'\n' +
'test_program.py F [100%]\n' +
'\n' +
'=================================== FAILURES ===================================\n' +
'_______________________________ test_weekly_todo _______________________________\n' +
'\n' +
'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
'\n' +
' def test_weekly_todo(monkeypatch, capsys):\n' +
' user_input = iter([\n' +
' "Laundry",\n' +
' "Groceries",\n' +
' "Gym",\n' +
' "Study",\n' +
' "Work",\n' +
' "Hike",\n' +
' "Rest",\n' +
' ])\n' +
' \n' +
' run_program_with(monkeypatch, user_input)\n' +
' \n' +
'> assert capsys.readouterr().out == (\n' +
' "Weekly to-do list for the 7 days of the week:\\n"\n' +
' "Mon: Laundry\\n"\n' +
' "Tues: Groceries\\n"\n' +
' "Wed Gym\\n"\n' +
' "Thur: Study\\n"\n' +
' "Fri: Work\\n"\n' +
' "Sat: Hike\\n"\n' +
' "Sun: Rest\\n"\n' +
' )\n' +
"E AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
'E \n' +
'E Skipping 34 identical leading characters in diff, use -v to show\n' +
'E - f the week:\n' +
'E + f the weeke:\n' +
'E ? +\n' +
'E Mon: Laundry\n' +
'E Tues: Groceries...\n' +
'E \n' +
"E ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
'\n' +
'test_program.py:29: AssertionError\n' +
'=========================== short test summary info ============================\n' +
"FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
'============================== 1 failed in 0.02s ===============================\n'
}
],
compilation_error: '',
runtime_error: '',
execution_time_exceeded: false,
memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junitxml file that contains information about the tests that were run. if that is something you want to look into.

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item
test_program.py F [100%]
=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>
def test_weekly_todo(monkeypatch, capsys):
user_input = iter([
"Laundry",
"Groceries",
"Gym",
"Study",
"Work",
"Hike",
"Rest",
])
run_program_with(monkeypatch, user_input)
> assert capsys.readouterr().out == (
"Weekly to-do list for the 7 days of the week:\n"
"Mon: Laundry\n"
"Tues: Groceries\n"
"Wed Gym\n"
"Thur: Study\n"
"Fri: Work\n"
"Sat: Hike\n"
"Sun: Rest\n"
)
E AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E Skipping 34 identical leading characters in diff, use -v to show
E - f the week:
E + f the weeke:
E ? +
E Mon: Laundry
E Tues: Groceries...
E
E ...Full output truncated (5 lines hidden), use '-vv' to show
test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
state: 'failed',
tests_run: 1,
passed: 0,
failed: 1,
errors: 0,
no_tests_collected: false,
exit_code: 1,
failure_details: [
{
test_case: 'test_weekly_todo',
expected: '',
received: '',
error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
rawout: '============================= test session starts ==============================\n' +
'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
'collected 1 item\n' +
'\n' +
'test_program.py F [100%]\n' +
'\n' +
'=================================== FAILURES ===================================\n' +
'_______________________________ test_weekly_todo _______________________________\n' +
'\n' +
'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
'\n' +
' def test_weekly_todo(monkeypatch, capsys):\n' +
' user_input = iter([\n' +
' "Laundry",\n' +
' "Groceries",\n' +
' "Gym",\n' +
' "Study",\n' +
' "Work",\n' +
' "Hike",\n' +
' "Rest",\n' +
' ])\n' +
' \n' +
' run_program_with(monkeypatch, user_input)\n' +
' \n' +
'> assert capsys.readouterr().out == (\n' +
' "Weekly to-do list for the 7 days of the week:\\n"\n' +
' "Mon: Laundry\\n"\n' +
' "Tues: Groceries\\n"\n' +
' "Wed Gym\\n"\n' +
' "Thur: Study\\n"\n' +
' "Fri: Work\\n"\n' +
' "Sat: Hike\\n"\n' +
' "Sun: Rest\\n"\n' +
' )\n' +
"E AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
'E \n' +
'E Skipping 34 identical leading characters in diff, use -v to show\n' +
'E - f the week:\n' +
'E + f the weeke:\n' +
'E ? +\n' +
'E Mon: Laundry\n' +
'E Tues: Groceries...\n' +
'E \n' +
"E ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
'\n' +
'test_program.py:29: AssertionError\n' +
'=========================== short test summary info ============================\n' +
"FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
'============================== 1 failed in 0.02s ===============================\n'
}
],
compilation_error: '',
runtime_error: '',
execution_time_exceeded: false,
memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junitxml file that contains information about the tests that were run. if that is something you want to look into.

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item
test_program.py F [100%]
=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>
def test_weekly_todo(monkeypatch, capsys):
user_input = iter([
"Laundry",
"Groceries",
"Gym",
"Study",
"Work",
"Hike",
"Rest",
])
run_program_with(monkeypatch, user_input)
> assert capsys.readouterr().out == (
"Weekly to-do list for the 7 days of the week:\n"
"Mon: Laundry\n"
"Tues: Groceries\n"
"Wed Gym\n"
"Thur: Study\n"
"Fri: Work\n"
"Sat: Hike\n"
"Sun: Rest\n"
)
E AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E Skipping 34 identical leading characters in diff, use -v to show
E - f the week:
E + f the weeke:
E ? +
E Mon: Laundry
E Tues: Groceries...
E
E ...Full output truncated (5 lines hidden), use '-vv' to show
test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
state: 'failed',
tests_run: 1,
passed: 0,
failed: 1,
errors: 0,
no_tests_collected: false,
exit_code: 1,
failure_details: [
{
test_case: 'test_weekly_todo',
expected: '',
received: '',
error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
rawout: '============================= test session starts ==============================\n' +
'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
'collected 1 item\n' +
'\n' +
'test_program.py F [100%]\n' +
'\n' +
'=================================== FAILURES ===================================\n' +
'_______________________________ test_weekly_todo _______________________________\n' +
'\n' +
'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
'\n' +
' def test_weekly_todo(monkeypatch, capsys):\n' +
' user_input = iter([\n' +
' "Laundry",\n' +
' "Groceries",\n' +
' "Gym",\n' +
' "Study",\n' +
' "Work",\n' +
' "Hike",\n' +
' "Rest",\n' +
' ])\n' +
' \n' +
' run_program_with(monkeypatch, user_input)\n' +
' \n' +
'> assert capsys.readouterr().out == (\n' +
' "Weekly to-do list for the 7 days of the week:\\n"\n' +
' "Mon: Laundry\\n"\n' +
' "Tues: Groceries\\n"\n' +
' "Wed Gym\\n"\n' +
' "Thur: Study\\n"\n' +
' "Fri: Work\\n"\n' +
' "Sat: Hike\\n"\n' +
' "Sun: Rest\\n"\n' +
' )\n' +
"E AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
'E \n' +
'E Skipping 34 identical leading characters in diff, use -v to show\n' +
'E - f the week:\n' +
'E + f the weeke:\n' +
'E ? +\n' +
'E Mon: Laundry\n' +
'E Tues: Groceries...\n' +
'E \n' +
"E ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
'\n' +
'test_program.py:29: AssertionError\n' +
'=========================== short test summary info ============================\n' +
"FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
'============================== 1 failed in 0.02s ===============================\n'
}
],
compilation_error: '',
runtime_error: '',
execution_time_exceeded: false,
memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junitxml file that contains information about the tests that were run. if that is something you want to look into.

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item
test_program.py F [100%]
=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>
def test_weekly_todo(monkeypatch, capsys):
user_input = iter([
"Laundry",
"Groceries",
"Gym",
"Study",
"Work",
"Hike",
"Rest",
])
run_program_with(monkeypatch, user_input)
> assert capsys.readouterr().out == (
"Weekly to-do list for the 7 days of the week:\n"
"Mon: Laundry\n"
"Tues: Groceries\n"
"Wed Gym\n"
"Thur: Study\n"
"Fri: Work\n"
"Sat: Hike\n"
"Sun: Rest\n"
)
E AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E Skipping 34 identical leading characters in diff, use -v to show
E - f the week:
E + f the weeke:
E ? +
E Mon: Laundry
E Tues: Groceries...
E
E ...Full output truncated (5 lines hidden), use '-vv' to show
test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
state: 'failed',
tests_run: 1,
passed: 0,
failed: 1,
errors: 0,
no_tests_collected: false,
exit_code: 1,
failure_details: [
{
test_case: 'test_weekly_todo',
expected: '',
received: '',
error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
rawout: '============================= test session starts ==============================\n' +
'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
'collected 1 item\n' +
'\n' +
'test_program.py F [100%]\n' +
'\n' +
'=================================== FAILURES ===================================\n' +
'_______________________________ test_weekly_todo _______________________________\n' +
'\n' +
'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
'\n' +
' def test_weekly_todo(monkeypatch, capsys):\n' +
' user_input = iter([\n' +
' "Laundry",\n' +
' "Groceries",\n' +
' "Gym",\n' +
' "Study",\n' +
' "Work",\n' +
' "Hike",\n' +
' "Rest",\n' +
' ])\n' +
' \n' +
' run_program_with(monkeypatch, user_input)\n' +
' \n' +
'> assert capsys.readouterr().out == (\n' +
' "Weekly to-do list for the 7 days of the week:\\n"\n' +
' "Mon: Laundry\\n"\n' +
' "Tues: Groceries\\n"\n' +
' "Wed Gym\\n"\n' +
' "Thur: Study\\n"\n' +
' "Fri: Work\\n"\n' +
' "Sat: Hike\\n"\n' +
' "Sun: Rest\\n"\n' +
' )\n' +
"E AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
'E \n' +
'E Skipping 34 identical leading characters in diff, use -v to show\n' +
'E - f the week:\n' +
'E + f the weeke:\n' +
'E ? +\n' +
'E Mon: Laundry\n' +
'E Tues: Groceries...\n' +
'E \n' +
"E ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
'\n' +
'test_program.py:29: AssertionError\n' +
'=========================== short test summary info ============================\n' +
"FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
'============================== 1 failed in 0.02s ===============================\n'
}
],
compilation_error: '',
runtime_error: '',
execution_time_exceeded: false,
memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junitxml file that contains information about the tests that were run. if that is something you want to look into.

[NickK21]

Summary

• extracts pytest parsing into a dedicated parser
• preserves real pytest assertion, collection, and runtime messages instead of collapsing them into generic output
• normalizes failure targets and test names
• captures expected, received, and rawout in a more structured way
• codewit.us now depends on stable structured failure data to generate beginner-friendly Outcome messages
• fixes the old problem where useful pytest details were getting buried, truncated, or reduced to noisy output

[DAC098]

where did you get the output for your tests?

[DAC098]

there is a particular test case that I have that I would expect to have the expected and received fields set but does not show them as having been found.

============================= test session starts ==============================
platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0
rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1
collected 1 item
test_program.py F [100%]
=================================== FAILURES ===================================
_______________________________ test_weekly_todo _______________________________
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>
capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>
def test_weekly_todo(monkeypatch, capsys):
user_input = iter([
"Laundry",
"Groceries",
"Gym",
"Study",
"Work",
"Hike",
"Rest",
])
run_program_with(monkeypatch, user_input)
> assert capsys.readouterr().out == (
"Weekly to-do list for the 7 days of the week:\n"
"Mon: Laundry\n"
"Tues: Groceries\n"
"Wed Gym\n"
"Thur: Study\n"
"Fri: Work\n"
"Sat: Hike\n"
"Sun: Rest\n"
)
E AssertionError: assert 'Weekly to-do...\nSun: Rest\n' == 'Weekly to-do...\nSun: Rest\n'
E
E Skipping 34 identical leading characters in diff, use -v to show
E - f the week:
E + f the weeke:
E ? +
E Mon: Laundry
E Tues: Groceries...
E
E ...Full output truncated (5 lines hidden), use '-vv' to show
test_program.py:29: AssertionError
=========================== short test summary info ============================
FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...
============================== 1 failed in 0.02s ===============================

the returned JSON shows this as the output:

{
state: 'failed',
tests_run: 1,
passed: 0,
failed: 1,
errors: 0,
no_tests_collected: false,
exit_code: 1,
failure_details: [
{
test_case: 'test_weekly_todo',
expected: '',
received: '',
error_message: "AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'",
rawout: '============================= test session starts ==============================\n' +
'platform linux -- Python 3.11.12, pytest-9.1.1, pluggy-1.6.0\n' +
'rootdir: /usr/src/app/temp/3ea2aaa1-801a-4609-9b96-3fcb01fd57d1\n' +
'collected 1 item\n' +
'\n' +
'test_program.py F [100%]\n' +
'\n' +
'=================================== FAILURES ===================================\n' +
'_______________________________ test_weekly_todo _______________________________\n' +
'\n' +
'monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7fb9248ae5d0>\n' +
'capsys = <_pytest.capture.CaptureFixture object at 0x7fb9248af010>\n' +
'\n' +
' def test_weekly_todo(monkeypatch, capsys):\n' +
' user_input = iter([\n' +
' "Laundry",\n' +
' "Groceries",\n' +
' "Gym",\n' +
' "Study",\n' +
' "Work",\n' +
' "Hike",\n' +
' "Rest",\n' +
' ])\n' +
' \n' +
' run_program_with(monkeypatch, user_input)\n' +
' \n' +
'> assert capsys.readouterr().out == (\n' +
' "Weekly to-do list for the 7 days of the week:\\n"\n' +
' "Mon: Laundry\\n"\n' +
' "Tues: Groceries\\n"\n' +
' "Wed Gym\\n"\n' +
' "Thur: Study\\n"\n' +
' "Fri: Work\\n"\n' +
' "Sat: Hike\\n"\n' +
' "Sun: Rest\\n"\n' +
' )\n' +
"E AssertionError: assert 'Weekly to-do...\\nSun: Rest\\n' == 'Weekly to-do...\\nSun: Rest\\n'\n" +
'E \n' +
'E Skipping 34 identical leading characters in diff, use -v to show\n' +
'E - f the week:\n' +
'E + f the weeke:\n' +
'E ? +\n' +
'E Mon: Laundry\n' +
'E Tues: Groceries...\n' +
'E \n' +
"E ...Full output truncated (5 lines hidden), use '-vv' to show\n" +
'\n' +
'test_program.py:29: AssertionError\n' +
'=========================== short test summary info ============================\n' +
"FAILED test_program.py::test_weekly_todo - AssertionError: assert 'Weekly to-...\n" +
'============================== 1 failed in 0.02s ===============================\n'
}
],
compilation_error: '',
runtime_error: '',
execution_time_exceeded: false,
memory_exceeded: false
}

or is this expected behavior.

side note, I made a comment about this in the PR that Kevin did that pytest offers the ability to output structured data in the form of a junitxml file that contains information about the tests that were run. if that is something you want to look into.