Dialect map: data diff

About

This repository contains a small CLI application to handle data file diffs coming from Dialect map data repository interactions.

Furthermore, it parses those diffs into JSON messages that are published to a GCP Pub/Sub topic. Thanks to the use of Pub/Sub, these messages can be fanned out into a wide range of data-ingestion jobs.

Environment setup

All dependencies can be installed by running:

npm install

Formatting

All JavaScript files are formatted using Prettier, and the custom properties defined in the .prettierrc.json file. To check for code style inconsistencies:

make check

Testing

Project testing is performed using Mocha. In order to run the tests:

make test

Commands

The package exposes a series of CLI commands to deal with data files differences. Each command has its own set of arguments and options.

Publish

Parses diff files and publish self-container change messages to a Pub/Sub topic.

dm-diff publish <data_file_path><diff_file_path><options>

OPTION	DEFAULT	REQUIRED	DESCRIPTION
--gcp-project	ds3-dialect-map	No	GCP project ID
--gcp-pubsub	data-diff	No	GCP Pub/Sub topic name
--debug-flag	false	No	Flag to indicate a debug run