Quick Start Videos

https://www.youtube.com/playlist?list=PLB0Zha5q-7wJp3TLH782J7ZDQ2RwPS_hQ

Contributing

We encourage contribution from the open source community to make DataGenerator better. Please refer to the development page for more information on how to contribute to this project.

Maven Dependency

For the core

<dependency><groupId>org.finra.datagenerator</groupId><artifactId>dg-core</artifactId><version>2.2</version></dependency>

For the commons library

<dependency><groupId>org.finra.datagenerator</groupId><artifactId>dg-common<artifactId><version>2.2</version></dependency>

Building

DataGenerator uses Maven for build. Please install Maven by downloading it from here.

# Clone DataGenerator git repo
git clone git://github.com/FINRAOS/DataGenerator.git
cd DataGenerator
# Checkout master branch
git checkout master
# Run package to compile and create jar (also runs unit tests)
mvn package
# Compile and run unit tests only
mvn test

License

The DataGenerator project is licensed under Apache License Version 2.0

Overview

Data Generator generates pattern using two pieces of user provided information:

1. An SCXML state chart representing interactions between different states, and setting values to output variables
2. A user Transformer that formats the variables and stores them.

The user can optionally provide their own distributor that distributes the search of bigger problems on systems like hadoop. By default, DataGenerator will use a multithreaded distributor.

Quick start

For the full compilable code please see the default example

First step, define an SCXML model:

<scxmlxmlns="http://www.w3.org/2005/07/scxml"xmlns:cs="http://commons.apache.org/scxml"version="1.0"initial="start">
<stateid="start">
<transitionevent="SETV1"target="SETV1"/>
</state>
<stateid="SETV1">
<onentry>
<assignname="var_out_V1_1"expr="set:{A1,B1,C1}"/>
<assignname="var_out_V1_2"expr="set:{A2,B2,C2}"/>
<assignname="var_out_V1_3"expr="77"/>
</onentry>
<transitionevent="SETV2"target="SETV2"/>
</state>
<stateid="SETV2">
<onentry>
<assignname="var_out_V2"expr="set:{1,2,3}"/>
<assignname="var_out_V3"expr="#{customplaceholder}"/>
</onentry>
<transitionevent="end"target="end"/>
</state>
<stateid="end">
<!-- We're done -->
</state>
</scxml>

This model contains five variables controlled by two states. The transition between those states is unconditional. One of those variables is always constant ( var_out_V1_3 ). Three will acquire every value from a set ( var_out_V1_1, var_out_V1_2 and var_out_V2 ). var_out_V3 will be set to a holder value that will be replaced by the user at a later point.

The second step will be to write a Transformer. The code is here

publicclassSampleMachineTransformerimplementsDataTransformer {
privatestaticfinalLoggerlog = Logger.getLogger(SampleMachineTransformer.class);
privatefinalRandomrand = newRandom(System.currentTimeMillis());
/** * The transform method for this DataTransformer * @param cr a reference to DataPipe from which to read the current map */publicvoidtransform(DataPipecr) {
for (Map.Entry<String, String> entry : cr.getDataMap().entrySet()) {
Stringvalue = entry.getValue();
if (value.equals("#{customplaceholder}")) {
// Generate a random numberintran = rand.nextInt();
entry.setValue(String.valueOf(ran));
}
}
}
}

The above transformer will intercept every generated row, and convert the place holder "#customplaceholder" with a random number.

The last step will be writing a main function that ties both pieces together. Code is here

publicstaticvoidmain(String[] args) {
Engineengine = newSCXMLEngine();
//will default to samplemachine, but you could specify a different file if you choose toInputStreamis = CmdLine.class.getResourceAsStream("/" + (args.length == 0 ? "samplemachine" : args[0]) + ".xml");
engine.setModelByInputFileStream(is);
// Usually, this should be more than the number of threads you intend to runengine.setBootstrapMin(1);
//Prepare the consumer with the proper writer and transformerDataConsumerconsumer = newDataConsumer();
consumer.addDataTransformer(newSampleMachineTransformer());
consumer.addDataWriter(newDefaultWriter(System.out,
newString[]{"var_out_V1_1", "var_out_V1_2", "var_out_V1_3", "var_out_V2", "var_out_V3"}));
//Prepare the distributorDefaultDistributordefaultDistributor = newDefaultDistributor();
defaultDistributor.setThreadCount(1);
defaultDistributor.setDataConsumer(consumer);
Logger.getLogger("org.apache").setLevel(Level.WARN);
engine.process(defaultDistributor);
}

The first few lines will open an input stream on the SCXML file and pass the stream to the engine. Calling setBootStrapMin will attempt to split the graph generated from the state chart to at least the given number of splits. Here we passed 1 but in case you will execute the same code over hadoop or use a multithreaded version, you will need to increase that number to be at least the number of threads or mappers you wish to run. The rest of the code will set our transformer to the engine and create a writer based on the DefaultWriter. The function of the writer is to write the output to the user's desired destination.

The final piece sets the number of threads and called engine.process.