Hpipe

 _______ __
| | |.-----.|__|.-----.-----.
| || _ || || _ | -__|
|___|___|| __||__|| __|_____|
|__| |__|

Hpipe is a workflow engine supporting hybrid workflows with built-in support for Hadoop Streaming and Hive.

It is useful in many production scenarios such as offline data processing of click-through-rate prediction for online advertising.

Features

Integrate various types of jobs into a uniformed workflow.
Job status tracking mechanism: keep job status and auto-resume from breakpoints.
Workflow descriptive language based on Graphviz being easy to edit and visualize.