Renderly is a .NET library and library for automated comparison of images. It is intended for use in environments where you want to periodically check generated images against a reference image to see if any changes have caused the images to render differently or incorrectly.
The code is broken up into a few projects, described below.
This is the core library which provides the basic interfaces for building an application.
This is a reference application built on top of Renderly that takes a CSV file with test cases, runs image comparisons against the test cases, and generates a report of results.
This project has all the unit tests for the Renderly library.
Read the wiki!
You will need Visual Studio 2012 or Visual Studio 2013 to build. I assume the code should build fine in Visual Studio 2010, but it has not been tested with that version.
I have not provided a build script yet, so right now you should open Renderly and RenderlyApp in a solution and rebuild.
This applies to if you're using the included RenderlyApp. Running the RenderlyApp.exe without any arguments should produce the list of supported commands:
> .\RenderlyApp.exe -h
Available commands are:
batchadd - Add test cases as a batch
deletetest - Delete test cases (and their reference images) from a model
run - Run a rendering job
updateref - Update reference images for provides test cases.
help <name> - For help with one of the above commands
The batchadd command allows you to create a new test case file, or append test cases to an existing one. Note that if you append, it does not perform an in-place modification of the input file.
The command takes the following arguments:
> .\RenderlyApp.exe help batchadd
'batchadd' - Add test cases as a batch
Expected usage: RenderlyApp.exe batchadd <options>
<options> available:
-f, --file=VALUE CSV file with the test cases to generate.
-o, --out=VALUE CSV file to output with test cases.
-a, --append=VALUE Test Case file to append to
The file you pass to the -f argument is a CSV file that should start with the following header:
Type,Reference Location,Reference Save Directory,Release,Description
- The
Typecolumn is the type of test this is. - The
Reference Locationcolumn is a URI to the image where the image should be copied from. This image becomes the test case's reference image, and the location is where the test will obtain the image from in the future. Reference Save Directoryis the directory where the reference image will be saved.Releaseis what software/version release this test was originally created for, or what version it should apply to. This is mostly a reference field in case you are curious about what version/release you were testing an image in.Descriptionis a short comment about the test case. Only used for your reference.
The deletetest command allows you to delete a test case from a file.
The command takes the following arguments:
> .\RenderlyApp.exe help deletetest
'deletetest' - Delete test cases (and their reference images) from a model
Expected usage: RenderlyApp.exe deletetest <options>
<options> available:
-f, --file=VALUE Model file to delete test cases from.
-d, --dates=VALUE Comma-separated list of dates to delete test
cases for. MM-DD-YYYY format.
-r, --releases=VALUE Comma-separated list of releases to delete test
cases for
-t, --testids=VALUE Comma-separated list of test IDs to delete
-o, --outfile=VALUE File to save test cases to. If no file is
specified, the input file is overwritten.
Options should be rather self-explanatory. You can specify multiple different parameters to erase from a test case file. Any reference images will also be deleted from their saved location, if possible (may not be possible due to permissions or other accessibility issues).
The updateref command lets you update test cases, making the application redownload reference images. This may be useful in case your images or rendering changes, and you want to update the reference image a test case uses.
The command takes the following arguments:
> .\RenderlyApp.exe help updateref
'updateref' - Update reference images for provides test cases.
Expected usage: RenderlyApp.exe updateref <options>
<options> available:
-f, --file=VALUE Model file to update test cases in.
-t, --testids=VALUE Comma-separated list of test IDs to redownload
reference images for.
The run command lets you run a rendering and comparison test, using an input CSV file. A report will be generated detailing the results of the run.
The command takes the following arguments:
> .\RenderlyApp.exe help run
'run' - Run a rendering job
Expected usage: RenderlyApp.exe run <options>
<options> available:
-f, --file=VALUE The model to get test cases from.
-n, --name=VALUE The name of the report to generate
-o, --outdir=VALUE The directory to generate the report in
-m, --templatedir=VALUE The directory to get templates for report
generation
--threshold=VALUE Threshold value to configure how aggressive
image comparison is (0-100). 100 is exact match.
Defaults to 100.
--showall Show all results in report (including successes-
). By default, only failures are shown.
--copyref Copy reference images locally to report
directory. Default false.
-t, --testids=VALUE Comma-separated list of test IDs to run
-r, --releases=VALUE Comma-separated list of releases to run
-d, --dates=VALUE Comma-separated list of dates to run. MM-DD-YYYY
format. Runs for Date Added.
-y, --types=VALUE Comma-separated list of types to run.
The CSV file you pass to the -f option should start with the header:
Test Id,Type,Source Location,Reference Location,Date Added,Date Modified,Release,Description
Test Idis the number of this test. It should be a unique identifying number.Typeis the type of test this is.Source Locationis the URI to obtain the image for comparison.Reference Locationis the URI to the base image that is used for comparisons.Date Addedis the date this test was added to the model.Date Modifiedis the date this test was last changed/updated.Releaseis the software release/version this test was originally created against.Descriptionis a short comment about this test case.
A valid CSV file can be obtained by using the batchadd command.