BBCTester

Black-Box Compiler Tester

A simple library for black-box testing of compilers from the Compiler Design and Implementation course at UChile.

Dependencies

• dune (>= 2.9)
• ocaml (>= 4.08.0)
• alcotest (>= 1.2.2)
• containers (>= 3.0.1)

Installation

Download the sources as a zip archive, unzip and install the package

$ unzip BBCTester-main.zip
Archive: BBCTester-main.zip
002d4d44e78e9655eff48580f1820961fd2ec520
creating: BBCTester-main/
inflating: BBCTester-main/.gitignore inflating: BBCTester-main/Makefile inflating: BBCTester-main/README.md inflating: BBCTester-main/dune inflating: BBCTester-main/dune-project inflating: BBCTester-main/test.ml inflating: BBCTester-main/test.mli
$ cd BBCTester-main
$ make install
dune build
dune install Installing ...

Alternatively, you can clone the repository and install

$ git clone https://github.com/pleiad/BBCTester.git
Cloning into 'BBCTester'...
remote: Enumerating objects: 19, done.
remote: Counting objects: 100% (19/19), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 19 (delta 3), reused 12 (delta 3), pack-reused 0
Receiving objects: 100% (19/19), 7.10 KiB | 7.10 MiB/s, done.
Resolving deltas: 100% (3/3), done.
$ cd BBCTester
$ make install
dune build
dune install Installing ...

Usage

Entrypoint

This package contains a few helper functions to parse test files (see below for the format) and generate unit-tests for alcotest in a single module Test. The main entrypoint of the library is the following function (from test.mli).

(* Given the path of a C runtime file [runtime], a [compiler] and the path [dir] of a directory containing tests files, produces unit tests for each test files in [dir]. [compile_flags] are passed to the C compiler (clang), defaults to "-g". *)valtests_from_dir :
?compile_flags:string ->
runtime:string->
compiler:compiler->
?oracle:(string -> status*string) ->
string -> (string*unitAlcotest.test_caselist) list

Tests files (*.bbc)

A BBCTester test file contains both a source program to be fed to the compiler and various metadata to help testing the compiler. Here is how it looks like:

NAME: add1
DESCRIPTION: increments a number
SRC:
(add1 20)
EXPECTED:
21

It uses the extension .bbc and is composed of a few sections that appear in the following order:

• NAME: [optional, default empty] : the name of the test
• DESCRIPTION: [optional, default empty] : a longer description of the content of the test
• PARAMS: [optional, default empty] : a ,-separated list of pairs VAR=VAL that are added to the environment variables of the compiled executable
• STATUS: [optional, default No error] : either CT error (compile time error), RT error (runtime error) or No error/ Needs to be set to the appropriate error if the program is expected to fail either at compile time or at runtime. In that case the content of EXPECTED: is interpreted as a pattern (see Str) matched against the output of the failing phase.
• SRC: : the source of the program fed to the compiler
• EXPECTED: : the expected result of the program (note that debugging messages starting by | are ignored and shouldn't be part of the expected result). If the expected result ends with the message |ORACLE then the expected result is obtained by substituting |ORACLE with the result of a provided oracle called on the source code.