CodeDistillery is a framework aimed at facilitating the mining of source code changes from version control systems.
The thing that makes CodeDistillery a framework more than a tool, is its support for pluggable source code mining mechanisms, while providing the underlying infrastructure to efficiently apply these mechanisms on entire revision histories of numerous software repositories.
While one could get away with mining a single repository with 100 or so commits without considering scale, it is no longer the case when this task involves dozens and more of repositories, each having thousands and more of revisions. In fact, the latter is quite standard for current studies in empirical software engineering, so we thought that the tool we had built for this task could make life easier on other people trying to do similar stuff.
We utilize Spark as the distributed compute engine, and JGit as the data access layer to make the mining workload a highly parallel one. Then, we use spark to distribute and process it.
In light of the above, CodeDistillery would not have been possible (or at the very least, it would have been much much harder to build) without awesome people building awesome open source software, and in particular the OS projects we extensively built upon: Apache Spark, JGit and ChangeDistiller.
git clone https://github.com/staslev/CodeDistillery cd CodeDistillery mvn clean install The project was tested and developed using JDK 8.
If you have multiple JDKs installed, make sure maven uses JDK 8 when executing mvn clean install.
This can be done using the following command:
export JAVA_HOME=/my_jdk1.8/Contents/Home && mvn clean install
<dependencies> <dependency> <groupId>com.staslev.codedistillery</groupId> <artifactId>distillery-core</artifactId>
<version>0.5-SNAPSHOT</version>
</dependency>
<dependency> <groupId>com.staslev.codedistillery</groupId>
<artifactId>change-distiller-uzh</artifactId>
<version>0.5-SNAPSHOT</version>
</dependency>
</dependencies> We demonstrate CodeDistillery by providing an out-of-the-box support for mining Java fine-grained source code changes from Git repositories.
objectMain { defmain(args: Array[String]):Unit= { valcodeDistillery=newCodeDistillery(
vcsFactory =GitRepo.apply, distillerFactory =UzhSourceCodeChangeDistiller.apply, encoderFactory = () =>UzhSourceCodeChangeCSVEncoder) withCrossRepoRevisionParallelismvalrepoPath=Paths.get("/path/to/my/repo") valoutput=Paths.get("/path/to/write/output") valbranch="master"importLocalSparkParallelism.spark
codeDistillery.distill(Set((repoPath, branch)), output) } }The output is a CSV file with a # delimiter, consisting of the following fields (in respective order):
- Project name
- Commit hash
- Author name
- Author email
- Fine-grained change type
- Unique name of changed entity
- Significance level
- Parent entity type
- Unique name of parent entity
- Root entity type
- Unique name of root entity
- Commit message
- Filename
The output from the previous stage is a dataset of raw fine-grained source code changes as distilled from a software repository. It is often useful to aggregate this raw dataset into commit level statistics. A commit level dataset can be obtained by performing the following:
valinput1:: input2 :: output ::Nil=List("/path/to/input1", "/path/to/input2", "/path/to/output") .map(Paths.get(_))
importLocalSparkParallelism.sparkPerCommit.aggregate(Set(input1, input2), output)The output is a CSV file with a # delimiter, consisting of the following fields (in respective order):
Project name
Commit hash
Author name
Author mail
Date
Non test versatility
Commit message
Test cases added
Test cases removed
Test cases changed
Test suites added
Test suites removed
Test suites affected
Has issue ref
Non test files in commit
Total files in commit
Commit message length
++ { fine-grained source code change type frequencies } Which is a lexicographically sorted list of fine-grained source code change types.
The complete list of columns (a.k.a. header line) can be obtained using: PerCommit.headerLine.