We want to create a deep Neural Network that can automatically generate comments for code snippets passed to it. The motivation behind this is that in software development and maintenance, developers spend around 59% of their time on program comprehension activities. Having comments that are generated automatically will hopefully cut this time down. In order to do this we will combine the recent paper, Code2Vec: Learning Distributed Representations of Code by Alon et al. with the paper Deep Code Comment Generation by X. Hu et.al, so as to make a better performing model using the newer Code2Vec encoding that was not used in the Deep Code Comment Generation paper.
In this project, 2 experiments were conducted. For the first one, we used the Code2Seq code to create model which will generate comments for code snippets (functions) of Java instead of the function names. For the second one, we repeated the procedure followed in the first experiment with modified ASTs. In particular, we added the precific name of each variable to the AST in order to make the comments generated more descriptive. The evaluation of the experiments were conducted in terms of BLEU-4 score.
The performance of the first experiment was poor (BLEU-4 score 6.08), while with the novelty introduced in the second experiment we achieved an important improvement (BLEU-4 score 10.08). However, the performance achieved was much worse than the one achieved by X. HU (BLEU-4 score 38.17 ) in Deep Code Comment Generation paper, because our model was not able to produce long comments. Nevertheless, it predicted succesfully the shorter comments, as well as a part of the long ones. The reason for this behavior is that the Code2Seq was built to produce function names which are short, and not long sequences.
All in all, an important main conclusion we can draw regarding our best model, is that with the novelty introduced with the variable names in the AST, it is capable of understanding the syntactic and semantic meaning of Java code regarding the automatic comment generation. However suffers from the incapability to generate longer comments and complete.
The dataset that we used, is the same dataset used by the Deep Code Comment Generation paper, this is a dataset of more than 500,000 code snippets including comments. This also gave us a baseline against which to compare. The dataset can be found here.
The pipeline of the system is:
- Extract the ASTs from the code snippets-comment pairs.
- Use the extracted ASTs to train the model.
- Test the trained model on the test data.
The high-level pipeline is shown in the following image:
The Encoder-Decoder architecture of this project is shown in the image below and is influenced by the work of U. Alon et al.
The BLEU-4 score achieved in the test dataset is presented below:
| Approaches | BLEU-4 |
|---|---|
| DeepCom | 38.17 |
| Method-1 | 6.08 |
| Method-2 | 10.02 |
For more information about the results and a detailed description of the 2 methods used, please feel free to take a look at our project report that is included on this repository.
The structure of the project is:
JavaExtractorThis directory contains the necessary code for exctracting the ASTs from the dataset.code2seq_masterThis directory contains the original Code2Seq code.datahere you can find a small portion of the data we used. We couldn't upload the whole dataset because of its size.preprocConatins all the necessary ptython files and neccesary scripts for the preprocessing and the proper execution of the AST extraction.reportContains the report for this project and its latex code.scriptsContains all the extra scripts used, like the perl script for the BLEU score extraction.bleu.pyExtracts the BLEU-4 score for a reference and a prediction file.