RepoExec is a novel benchmark designed to evaluate code generation at the repository level with a focus on executability and correctness. This benchmark addresses the gaps in existing systems by emphasizing real-world applicability and providing a comprehensive assessment of code functionality. It aims to provide a comprehensive evaluation of code functionality and alignment with developer intent, paving the way for more reliable and applicable CodeLLMs in real-world scenarios.
RepoExec is available at Huggingface datasets. The instruction-tuning data used in our work is presented in RepoExec-Instruct.
fromdatasetsimportload_dataset# RepoExec contains 3 subsets, corresponding to the detail of the context level# full_context || medium_context || small_contextdataset=load_dataset("Fsoft-AIC/RepoExec")Examples:
# full contextimportbase64importrandomimportunicodedataimportzlibfromtypingimportUnionfromuuidimportuuid4from ._regeximport*from .errorsimportInvalidInputErrorfrom .validationimportis_snake_case, is_full_string, is_camel_case, is_integer, is_stringCAMEL_CASE_REPLACE_RE=re.compile(r'([a-z]|[A-Z]+)(?=[A-Z])')
classInvalidInputError(TypeError):
""" Custom error raised when received object is not a string as expected. """def__init__(self, input_data: Any):
""" :param input_data: Any received object """type_name=type(input_data).__name__msg='Expected "str", received "{}"'.format(type_name)
super().__init__(msg)
defis_string(obj: Any) ->bool:
""" Checks if an object is a string. *Example:* >>> is_string('foo') # returns true >>> is_string(b'foo') # returns false :param obj: Object to test. :return: True if string, false otherwise. """returnisinstance(obj, str)
defis_camel_case(input_string: Any) ->bool:
""" Checks if a string is formatted as camel case. A string is considered camel case when: - it's composed only by letters ([a-zA-Z]) and optionally numbers ([0-9]) - it contains both lowercase and uppercase letters - it does not start with a number *Examples:* >>> is_camel_case('MyString') # returns true >>> is_camel_case('mystring') # returns false :param input_string: String to test. :type input_string: str :return: True for a camel case string, false otherwise. """returnis_full_string(input_string) andCAMEL_CASE_TEST_RE.match(input_string) isnotNonedefcamel_case_to_snake(input_string, separator='_'):
""" Convert a camel case string into a snake case one. (The original string is returned if is not a valid camel case string) *Example:* >>> camel_case_to_snake('ThisIsACamelStringTest') # returns 'this_is_a_camel_case_string_test' :param input_string: String to convert. :type input_string: str :param separator: Sign to use as separator. :type separator: str :return: Converted string. """# medium contextimportbase64importrandomimportunicodedataimportzlibfromtypingimportUnionfromuuidimportuuid4from ._regeximport*from .errorsimportInvalidInputErrorfrom .validationimportis_snake_case, is_full_string, is_camel_case, is_integer, is_stringCAMEL_CASE_REPLACE_RE=re.compile(r'([a-z]|[A-Z]+)(?=[A-Z])')
classInvalidInputError(TypeError):
""" Custom error raised when received object is not a string as expected. """def__init__(self, input_data: Any):
""" :param input_data: Any received object """defis_string(obj: Any) ->bool:
""" Checks if an object is a string. *Example:* >>> is_string('foo') # returns true >>> is_string(b'foo') # returns false :param obj: Object to test. :return: True if string, false otherwise. """defis_camel_case(input_string: Any) ->bool:
""" Checks if a string is formatted as camel case. A string is considered camel case when: - it's composed only by letters ([a-zA-Z]) and optionally numbers ([0-9]) - it contains both lowercase and uppercase letters - it does not start with a number *Examples:* >>> is_camel_case('MyString') # returns true >>> is_camel_case('mystring') # returns false :param input_string: String to test. :type input_string: str :return: True for a camel case string, false otherwise. """defcamel_case_to_snake(input_string, separator='_'):
""" Convert a camel case string into a snake case one. (The original string is returned if is not a valid camel case string) *Example:* >>> camel_case_to_snake('ThisIsACamelStringTest') # returns 'this_is_a_camel_case_string_test' :param input_string: String to convert. :type input_string: str :param separator: Sign to use as separator. :type separator: str :return: Converted string. """# small contextimportbase64importrandomimportunicodedataimportzlibfromtypingimportUnionfromuuidimportuuid4from ._regeximport*from .errorsimportInvalidInputErrorfrom .validationimportis_snake_case, is_full_string, is_camel_case, is_integer, is_stringCAMEL_CASE_REPLACE_RE=re.compile(r'([a-z]|[A-Z]+)(?=[A-Z])')
classInvalidInputError(TypeError):
def__init__(self, input_data: Any):
defis_string(obj: Any) ->bool:
defis_camel_case(input_string: Any) ->bool:
defcamel_case_to_snake(input_string, separator='_'):
""" Convert a camel case string into a snake case one. (The original string is returned if is not a valid camel case string) *Example:* >>> camel_case_to_snake('ThisIsACamelStringTest') # returns 'this_is_a_camel_case_string_test' :param input_string: String to convert. :type input_string: str :param separator: Sign to use as separator. :type separator: str :return: Converted string. """git clone https://github.com/FSoft-AI4Code/RepoExec.git cd RepoExec
unzip test-apps.zip
pip install -r requirement
cd RepoExec/bigcode-eval-repoexec
pip install -e .
cd RepoExec/execution-code-eval
(sudo) docker build -t codeeval-runner -f Dockerfile --platform linux/amd64 .
Script examples to run evaluation are contained in scripts
cd RepoExec/bigcode-eval-repoexec
pip install -e .
Example scripts are in phi-2-generation
There are 2 kinds of prompts: BasePrompt and InstructPrompt:
- To use BasePrompt, specify the
--tasksargument torepoexec-{full|medium|small}-context. - To use InstructPrompt, specify the
--tasksargument toinstruct-repoexec-{full|medium|small}-contextandpromptargument to use the template specific for each model (e.g.--prompt codellamafor CodeLlama series).
After running the generation script, generation result will be a nested list of prediction for each problem in the dataset and is saved to a generations.json file. See the example in phi-2 prediction
Example:
[[pred_11, pred_12, pred13], [pred_21, pred_22, pred_23], ...]Note: If you're using a closed-source model (such as gpt-4o-turbo), please follow these scripts api_llms_generate.py and api_llms_infer.sh to run the process, and don't forget to export your API key.
Process to acquire the target function from prediction and save to json file.
python3 process_result.py \
--subset medium_context \
--prediction_dir ../results/examples/predictions/repoexec-full-context/BasePrompt-phi-2 \
Note: For closed-source models that require API calls, after collecting the generated responses, run process_result with the additional --is_api flag to process the results properly.
Execute the generated function of the model to obtain the execution output.
python3execute.py--subsetfull_context \
--prediction_dir ../results/examples/predictions/repoexec-full-context/BasePrompt-phi-2 \
--execution_dir ../results/examples/execution_rs/repoexec-full-context/BasePrompt-phi-2 \python3passk.py--execution_dir ../results/examples/execution_rs/repoexec-full-context/BasePrompt-phi-2python3get_dir.py--execution_dir ../results/examples/execution_rs/repoexec-full-context/BasePrompt-phi-2Please see this repo for tool usage.
More details can be found in our paper.
If you're using RepoExec, please cite using this BibTeX:
@article{nam2024repoexec,
title={RepoExec: Evaluate Code Generation with a Repository-Level Executable Benchmark},
author={Hai, Nam Le and Manh, Dung Nguyen and Bui, Nghi DQ},
journal={arXiv preprint arXiv:2406.11927v1},
year={2024}
}This codebase is adapted from:
If you have any questions, comments or suggestions, please do not hesitate to contact us.
- Website: fpt-aicenter
- Email: support.ailab@fpt.com
- Corresponding Author: namlh@soict.hust.edu.vn