stdout comparison by KRRT7 · Pull Request #18 · codeflash-ai/codeflash

KRRT7 · 2025-02-18T08:59:57Z

Update test_results.py
Update parse_test_output.py
Update equivalence.py

misrasaurabh1

This needs thorough unit testing

misrasaurabh1 · 2025-03-03T21:41:23Z

why does this have \n whereas all others have proper newlines?

was debugging something, forgot to revert.

Update test_results.py Update parse_test_output.py Update equivalence.py

Update pyproject.toml Update test_instrument_tests.py Update test_instrument_all_and_run.py missing test

## Problem When test files were inside the project root (common case), Codeflash did NOT create the runtime config that disables globalSetup/globalTeardown. This caused Jest to use the project's original config WITH globalSetup, leading to failures when globalSetup required unavailable infrastructure (Docker, databases, etc.): Error: Jest: Got error running globalSetup - /workspace/target/globalSetup.ts, reason: Command failed: docker context ls --format json /bin/sh: 1: docker: not found ## Root Cause Runtime config was only created when tests were OUTSIDE project root: if any(not Path(d).is_relative_to(resolved_root) for d in test_dirs): jest_config = _create_runtime_jest_config(...) But globalSetup should be disabled for ALL Codeflash test runs. ## Solution Always create runtime config when `jest_config` and `test_files` exist: if test_files and jest_config: test_dirs = {str(Path(f).resolve().parent) for f in test_files} jest_config = _create_runtime_jest_config(jest_config, effective_cwd, test_dirs) ## Impact - Affected: ALL projects with globalSetup/globalTeardown in Jest config - Reproducibility: 100% systematic - Example trace: 04dc4dcf-ca9f-449e-aed5-7a82f28c5e23 ## Changes - test_runner.py: Updated 3 functions (behavioral, benchmarking, line profiling) - New test file: test_globalsetup_invocation_bug.py (3 test cases) - All existing tests still pass Fixes#18 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

KRRT7 force-pushed the stdout_comparison_ branch from ace7938 to b265fa2Compare February 18, 2025 11:25

KRRT7 marked this pull request as ready for review February 18, 2025 11:32

KRRT7 requested a review from misrasaurabh1 February 18, 2025 11:32

misrasaurabh1 requested changes Feb 20, 2025
View reviewed changes

KRRT7 requested a review from misrasaurabh1 February 21, 2025 02:32

misrasaurabh1 reviewed Feb 21, 2025
View reviewed changes

Comment threadtests/test_instrument_all_and_run.py Outdated

misrasaurabh1 reviewed Feb 21, 2025
View reviewed changes

Comment threadtests/test_instrument_tests.py Outdated

misrasaurabh1 reviewed Feb 21, 2025
View reviewed changes

Comment threadcode_to_optimize/bubble_sort_method.py

KRRT7 requested a review from misrasaurabh1 February 21, 2025 08:19

codeflash-ai deleted a comment from CLAassistant Feb 24, 2025