[FEAT] mcp server for correctness verification and benchmarking

[mohammedahmed18]

Example e2e workflow (on bubble_sort.py)

============================================================
MCP WORKFLOW TEST
=======================================================================================================================**********python**********===========================================================[1] Running behavioral tests (baseline)...
/home/mohammed/.codeflash/mcp_results.db
run_id: e2e-test-a8e59cfe-0f33-4ebf-a52f-c9d422611088
total_tests: 3
passed: 3
failed: 0
runtime_ns: 1201283130
[2] Running benchmark (baseline, original code)...
run_id: e2e-test-a8e59cfe-0f33-4ebf-a52f-c9d422611088-bench-baseline
total_runtime: 1174053263ns
loops_executed: 3
[3] Applying optimization to bubble_sort.py...
module: /home/mohammed/Work/codeflash/code_to_optimize/bubble_sort.py (optimized in-place)
[4] Running behavioral tests (candidate)...
run_id: e2e-test-a8e59cfe-0f33-4ebf-a52f-c9d422611088-candidate
total_tests: 3
passed: 3
failed: 0
runtime_ns: 231647
[5] Comparing baseline vs candidate...
equivalent: True
total_compared: 3
[6] Running benchmark (candidate, optimized code)...
run_id: e2e-test-a8e59cfe-0f33-4ebf-a52f-c9d422611088-bench-candidate
total_runtime: 197408ns
loops_executed: 10
speedup_x: 5947.344x
speedup_pct: 594634.4%
baseline_ns: 1174053263
candidate_ns: 197408
============================================================
RESULTS SUMMARY
============================================================
Baseline: 3/3 passed
Candidate: 3/3 passed
Equivalent: True
Benchmark baseline: 1174.05ms
Benchmark candidate: 0.20ms
Speedup: 5947.344x
Overall: PASS
===========================================================**********javascript**********===========================================================[1] Running behavioral tests (baseline)...
run_id: e2e-test-3a9a8e98-3410-483a-9544-78e01ee47073
total_tests: 9
passed: 9
failed: 0
runtime_ns: 0
[2] Running benchmark (baseline, original code)...
run_id: e2e-test-3a9a8e98-3410-483a-9544-78e01ee47073-bench-baseline
total_runtime: 171800ns
loops_executed: 23
[3] Applying optimization to bubble_sort.py...
module: /home/mohammed/Work/codeflash/code_to_optimize/js/code_to_optimize_js/bubble_sort.js (optimized in-place)
[4] Running behavioral tests (candidate)...
run_id: e2e-test-3a9a8e98-3410-483a-9544-78e01ee47073-candidate
total_tests: 9
passed: 9
failed: 0
runtime_ns: 0
[5] Comparing baseline vs candidate...
equivalent: True
total_compared: 9
[6] Running benchmark (candidate, optimized code)...
run_id: e2e-test-3a9a8e98-3410-483a-9544-78e01ee47073-bench-candidate
total_runtime: 25911ns
loops_executed: 24
speedup_x: 6.630x
speedup_pct: 563.0%
baseline_ns: 171800
candidate_ns: 25911
============================================================
RESULTS SUMMARY
============================================================
Baseline: 9/9 passed
Candidate: 9/9 passed
Equivalent: True
Benchmark baseline: 0.17ms
Benchmark candidate: 0.03ms
Speedup: 6.630x
Overall: PASS
===========================================================**********java**********===========================================================[1] Running behavioral tests (baseline)...
INFO codeflash-runtime dependency already up to date in pom.xml
INFO Injected validation skip properties and plugin overrides into /home/mohammed/Work/codeflash/code_to_optimize/java/pom.xml
run_id: e2e-test-0ea4f62d-96a9-417c-a021-47917918dc09
total_tests: 8
passed: 8
failed: 0
runtime_ns: 68276385
[2] Running benchmark (baseline, original code)...
INFO codeflash-runtime dependency already up to date in pom.xml
run_id: e2e-test-0ea4f62d-96a9-417c-a021-47917918dc09-bench-baseline
total_runtime: 43233675ns
loops_executed: 39
[3] Applying optimization to bubble_sort.py...
module: /home/mohammed/Work/codeflash/code_to_optimize/java/src/main/java/com/example/BubbleSort.java (optimized in-place)
[4] Running behavioral tests (candidate)...
INFO codeflash-runtime dependency already up to date in pom.xml
run_id: e2e-test-0ea4f62d-96a9-417c-a021-47917918dc09-candidate
total_tests: 8
passed: 8
failed: 0
runtime_ns: 3071838
[5] Comparing baseline vs candidate...
equivalent: True
total_compared: 8
[6] Running benchmark (candidate, optimized code)...
INFO codeflash-runtime dependency already up to date in pom.xml
run_id: e2e-test-0ea4f62d-96a9-417c-a021-47917918dc09-bench-candidate
total_runtime: 682116ns
loops_executed: 59
speedup_x: 63.382x
speedup_pct: 6238.2%
baseline_ns: 43233675
candidate_ns: 682116
============================================================
RESULTS SUMMARY
============================================================
Baseline: 8/8 passed
Candidate: 8/8 passed
Equivalent: True
Benchmark baseline: 43.23ms
Benchmark candidate: 0.68ms
Speedup: 63.382x
Overall: PASS

planning to integrate this with a coding agent (like claude-code), the workflow should look like this:

1- the agent decides to optimize a function
2- it generates a test file(s)
3- calls codeflash-mcp to run the test file(s) and the mcp will store the results into a sqlite db (path is passed via env)
4- calls codeflash-mcp to benchmark the original code by the agent generated tests
5- the agent then applies the optimization (replaces the code into the main file(s))
6- calls the behavioural test mcp tool again, then calls the comparator tool to compare test results (it knows what to compare based on the run-id (agent can provide it or the mcp will generate a random uuid and return it back to the agent))
7- agent keeps on fixing the optimized code until step 6 passes
8- calls codeflash-mcp benchmarking tool on the new code and make sure it's faster
keeps on iterating