data-pipeline

This library is a toolkit for describing data transformation pipelines by compositing simple reusable components.

A typical data-pipeline use-case can be:

• a system aggregating results from several external services: pipelines are modular, easily rearranged and each individual step can be padded with safety nets and error handling without affecting business logic
• a system performing iterative analysis on an input: the data-pipeline data model retains intermediate results from all steps, and each result is tagged with lineage metadata

On top of its core feature-set, complying to the data-pipeline model comes with rather nice benefits:

• out-of-the-box support for micrometer based metrics (success/failure rates, error tracking, etc.) and tracing
• out-of-the-box support for slf4j log markers (pipeline id, component id, etc.)
• easily pluggable resilience4j based resilience features (retries, time-limiter, etc.)

I. Installation

The library requires Java 17+, in order to use it, add the following in your pom.xml:

<dependency>
<groupId>tech.illuin</groupId>
<artifactId>data-pipeline</artifactId>
<version>0.32</version>
</dependency>

Additionally, some optional extension libraries can be added, at the time of this writing this includes:

• data-pipeline-resilience4j (for the resilience4j integration)
• data-pipeline-admin (for the admin interface)
  • data-pipeline-admin-micronaut (for the micronaut integration to the admin interface)
  • data-pipeline-admin-quarkus (for the quarkus integration to the admin interface)
  • data-pipeline-admin-spring-boot (for the springboot integration to the admin interface)

II. Core Principles

The main goals behind its design were:

1. having a fairly straightforward API and overall design: pipelines are made from compositing user-defined functions executed in a linear fashion
2. leveraging this design to introduce useful features: data-model with lineage features, parallel execution, resilience features (retry, time-limiter, etc.), systematic performance metrics, error tracking, etc.

A simplified high-level view of a data-pipeline pipeline looks like the following diagram:

• the main phase is composed of Step functions, they return a Result and are expected to have no side effect (think of them as almost-pure functions)
• the end phase is composed of Sink functions, they return void and are expected to induce side effects (e.g. database persistence, message queue push, etc.)

flowchart LR
INPUT((input))
OUTPUT((output))
subgraph PIPELINE[Pipeline]
direction LR
subgraph STEP_PHASE[Steps]
direction LR
STEP_1(Step 1):::step
STEP_2(Step 2):::step
STEP_N(...):::step
end
subgraph SINK_PHASE[Sinks]
direction LR
SINK_1(Sink 1):::sink
SINK_N(...):::sink
end
end
INPUT --> PIPELINE --> OUTPUT
STEP_PHASE --> SINK_PHASE STEP_1 --> STEP_2 --> STEP_N
classDef optional stroke-dasharray: 6 6;
classDef step stroke:#0f0;
classDef sink stroke:#f00;

Loading

III. Basic Usage

We'll go through a quick example in order to demonstrate what data-pipeline looks like in action.

The goal of this example is to have a simple pipeline for:

• performing a basic tokenization of a sentence
• performing a basic analysis of said tokens
• recovering results and logging them out

flowchart LR
INPUT((sentence))
subgraph PIPELINE[Pipeline]
direction LR
subgraph STEP_PHASE[Steps]
direction LR
TOKENIZER(Tokenizer):::step
MATCHER(Matcher):::step
end
subgraph SINK_PHASE[Sinks]
direction LR
MATCH_LOGGER(Match Logger):::sink
end
end
INPUT --> PIPELINE
STEP_PHASE --> SINK_PHASE
TOKENIZER --> MATCHER
classDef optional stroke-dasharray: 6 6;
classDef step stroke:#0f0;
classDef sink stroke:#f00;

Loading

Defining steps and sinks

First, we'll design the Tokenizer step, with a basic regex split.

Three things to note here, which will remain true for the following pieces:

• The step's entrypoint is annotated with @StepConfig, it will be identified at the pipeline build time
• Some component inputs have to be annotated in order to narrow down their identity, the pipeline input can be supplied with the @Input annotation
• The step output are expected to be a Result subtype, here we chose to go with a dedicated record

publicclassTokenizer
{
@StepConfig(id = "tokenizer")
publicTokenizedSentencetokenize(@InputStringsentence)
{
returnnewTokenizedSentence(Stream.of(sentence.split("[^\\p{L}]+"))
.map(String::toLowerCase)
.toList()
);
}
publicrecordTokenizedSentence(
List<String> tokens
) implementsResult {}
}

Next up, the Matcher step, with a blacklist specified upon instantiation.

It will recover the tokenizer's output, and produce a Matches record of its findings.

Note the @Current annotation for requesting the currently known value for tokenized sentence. There is more to be said about the semantics of this annotation, which we'll cover in details in the documentation section.

publicclassMatcher
{
privatefinalSet<String> blacklist;
publicMatcher(String... blacklist)
{
this.blacklist = Set.of(blacklist);
}
@StepConfig(id = "matcher")
publicMatchesmatch(@CurrentTokenizedSentencetokenized)
{
longwordCount = tokenized.tokens().stream().distinct().count();
Set<String> matches = tokenized.tokens().stream()
.filter(this.blacklist::contains)
.collect(Collectors.toSet())
;
returnnewMatches(wordCount, matches);
}
publicrecordMatches(
longwordCount,
Set<String> blacklistMatches
) implementsResult {}
}

Finally, our MatchLogger sink works very similarly, except we need the @SinkConfig annotation.

publicclassMatchLogger
{
privatestaticfinalLoggerlogger = LoggerFactory.getLogger(MatchLogger.class);
@SinkConfig(id = "logger")
publicvoidlog(@CurrentTokenizedSentencetokenized, @CurrentMatchesmatches)
{
logger.info("Found {} unique tokens in {}, with {} blacklisted {}", matches.wordCount(), tokenized.tokens(), matches.blacklistMatches().size(), matches.blacklistMatches());
}
}

Setting-up the pipeline

Now that we have all our building blocks, creating a Pipeline is simply a matter of combining them.

The Pipeline interface offers a builder initialization method, we'll start from there.

Pipeline<String> pipeline = Pipeline.<String>of("string-processor")
.registerStep(newTokenizer())
.registerStep(newMatcher("mostly", "relatively"))
.registerSink(newMatchLogger())
.build()
;

Now, calling the pipeline with some sentences:

pipeline.run("This is a relatively short and mostly meaningless sentence.");
pipeline.run("This is a much longer sentence that should go through the blacklist unscathed.");
pipeline.run("Relatively cool objects (temperatures less than several thousand degrees) emit their radiation primarily in the infrared, as described by Planck's law.");
pipeline.run("The principles were deliberately non dogmatic, since the brotherhood wished to emphasise the personal responsibility of individual artists to determine their own ideas and methods of depiction.");
pipeline.run("The Mystical Nativity, a relatively small and very personal painting, perhaps for his own use, appears to be dated to the end of 1500.");

We should get the following output (given a simplelogger or somesuch properly configured):

[main] INFO MatchLogger - Found 9 unique tokens in [this, is, a, relatively, short, and, mostly, meaningless, sentence], with 2 blacklisted [mostly, relatively]
[main] INFO MatchLogger - Found 13 unique tokens in [this, is, a, much, longer, sentence, that, should, go, through, the, blacklist, unscathed], with 0 blacklisted []
[main] INFO MatchLogger - Found 22 unique tokens in [relatively, cool, objects, temperatures, less, than, several, thousand, degrees, emit, their, radiation, primarily, in, the, infrared, as, described, by, planck, s, law], with 1 blacklisted [relatively]
[main] INFO MatchLogger - Found 23 unique tokens in [the, principles, were, deliberately, non, dogmatic, since, the, brotherhood, wished, to, emphasise, the, personal, responsibility, of, individual, artists, to, determine, their, own, ideas, and, methods, of, depiction], with 0 blacklisted []
[main] INFO MatchLogger - Found 21 unique tokens in [the, mystical, nativity, a, relatively, small, and, very, personal, painting, perhaps, for, his, own, use, appears, to, be, dated, to, the, end, of], with 1 blacklisted [relatively]

As pipelines may use resources (notably a ServiceExecutor for the —optional— async sink execution), it is best to close it down when you are done using it (or consider using a try-with pattern):

pipeline.close();

IV. Documentation

• Pipelines
  • Configuration
  • Execution
  • Shutting Down
• Result Data Model
• Initializers
• Steps
• Sinks
• Function Modifiers & Hooks
  • Error Handlers
  • Wrappers
  • UID Generators
  • Author Resolvers
  • Tag Resolvers
• Integrations
  • Micrometer
  • Prometheus
  • Grafana
  • Logback Loki
  • Resilience4j
• Admin
  • Core Admin
  • Spring Boot Admin
  • Micronaut Admin
  • Quarkus Admin

V. Dev Installation

This project will require you to have the following:

• Java 17+
• Git (versioning)
• Maven (dependency resolving, publishing and packaging)