Skip to content
BranchesTags

Repository files navigation

SQL Runner

Build StatusCoverallsGo Report CardReleaseLicense

Overview

Run playbooks of SQL scripts in series and parallel on Snowflake DB, Amazon Redshift and PostgreSQL.

Used with Snowplow for scheduled SQL-based transformations of event stream data.

Setup GuideUser Guide
i1i2

Quick start

Assuming go, docker and docker-compose are installed:

 host> git clone https://github.com/snowplow/sql-runner
host>cd sql-runner
host> make setup-up # Launches Consul + Postgres for testing
host> make # Builds sql-runner binaries
host> make test# Runs unit tests# DISTRO specifies which binary you want to run integration tests with
host> DISTRO=darwin make integration

Note: You will need to ensure that ~/go/bin is on your PATH for gox to work - the underlying tool that we use for building the binaries.

When you are done with testing simply execute make setup-down to terminate the docker-compose stack.

To reset the testing resources execute make setup-reset which will rebuild the docker containers. This can be useful if the state of these systems gets out of sync with what the tests expect.

To remove all build files:

guest> make clean

To format the golang code in the source directory:

guest> make format

Note: Always run make format before submitting any code.

Note: The make test command also generates a code coverage file which can be found at build/coverage/coverage.html.

How to use?

First either compile the binary from source using the above make command or download the published Binary directly from the GitHub release:

CLI Output

sql-runner version: 0.11.0
Run playbooks of SQL scripts in series and parallel on Redshift and Postgres
Usage:
-checkLock string
Checks whether the lockfile already exists
-consul string
The address of a consul server with playbooks and SQL files stored in KV pairs
-consulOnlyForLock
Will read playbooks locally, but use Consul for locking.
-deleteLock string
Will attempt to delete a lockfile if it exists
-dryRun
Runs through a playbook without executing any of the SQL
-fillTemplates
Will print all queries after templates are filled
-fromStep string
Starts from a given step defined in your playbook
-help
Shows this message
-lock string
Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully
-playbook string
Playbook of SQL scripts to execute
-runQuery string
Will run a single query in the playbook
-showQueryOutput
Will print all output from queries
-softLock string
Optional argument, like '-lock' but the lockfile will be deleted even if the run fails
-sqlroot string
Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")
-var value
Variables to be passed to the playbook, in the key=value format
-version
Shows the program version

Copyright and license

SQL Runner is copyright 2015-2022 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

About

Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake

snowplowanalytics.com

Topics

bigquerypostgresqlredshiftsnowflakesnowplowsql-runner

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

81 stars

Watchers

21 watching

Forks

15 forks
Report repository

Releases

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - snowplow/sql-runner: Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake · GitHub
Skip to content
BranchesTags
Repository files navigation
SQL Runner
Build StatusCoverallsGo Report CardReleaseLicense
Overview
Run playbooks of SQL scripts in series and parallel on Snowflake DB, Amazon Redshift and PostgreSQL.
Used with Snowplow for scheduled SQL-based transformations of event stream data.
Setup GuideUser Guide
i1i2
Quick start
Assuming go, docker and docker-compose are installed:
 host> git clone https://github.com/snowplow/sql-runner
host>cd sql-runner
host> make setup-up # Launches Consul + Postgres for testing
host> make # Builds sql-runner binaries
host> make test# Runs unit tests# DISTRO specifies which binary you want to run integration tests with
host> DISTRO=darwin make integration
Note: You will need to ensure that ~/go/bin is on your PATH for gox to work - the underlying tool that we use for building the binaries.
When you are done with testing simply execute make setup-down to terminate the docker-compose stack.
To reset the testing resources execute make setup-reset which will rebuild the docker containers. This can be useful if the state of these systems gets out of sync with what the tests expect.
To remove all build files:
guest> make clean
To format the golang code in the source directory:
guest> make format
Note: Always run make format before submitting any code.
Note: The make test command also generates a code coverage file which can be found at build/coverage/coverage.html.
How to use?
First either compile the binary from source using the above make command or download the published Binary directly from the GitHub release:
CLI Output
sql-runner version: 0.11.0
Run playbooks of SQL scripts in series and parallel on Redshift and Postgres
Usage:
-checkLock string
Checks whether the lockfile already exists
-consul string
The address of a consul server with playbooks and SQL files stored in KV pairs
-consulOnlyForLock
Will read playbooks locally, but use Consul for locking.
-deleteLock string
Will attempt to delete a lockfile if it exists
-dryRun
Runs through a playbook without executing any of the SQL
-fillTemplates
Will print all queries after templates are filled
-fromStep string
Starts from a given step defined in your playbook
-help
Shows this message
-lock string
Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully
-playbook string
Playbook of SQL scripts to execute
-runQuery string
Will run a single query in the playbook
-showQueryOutput
Will print all output from queries
-softLock string
Optional argument, like '-lock' but the lockfile will be deleted even if the run fails
-sqlroot string
Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")
-var value
Variables to be passed to the playbook, in the key=value format
-version
Shows the program version
Copyright and license
SQL Runner is copyright 2015-2022 Snowplow Analytics Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
About
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
snowplowanalytics.com
Topics
bigquerypostgresqlredshiftsnowflakesnowplowsql-runner
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
81 stars
Watchers
21 watching
Forks
15 forks
Report repository
Releases
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - snowplow/sql-runner: Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake · GitHub
Skip to content
BranchesTags
Repository files navigation
SQL Runner
Build StatusCoverallsGo Report CardReleaseLicense
Overview
Run playbooks of SQL scripts in series and parallel on Snowflake DB, Amazon Redshift and PostgreSQL.
Used with Snowplow for scheduled SQL-based transformations of event stream data.
Setup GuideUser Guide
i1i2
Quick start
Assuming go, docker and docker-compose are installed:
 host> git clone https://github.com/snowplow/sql-runner
host>cd sql-runner
host> make setup-up # Launches Consul + Postgres for testing
host> make # Builds sql-runner binaries
host> make test# Runs unit tests# DISTRO specifies which binary you want to run integration tests with
host> DISTRO=darwin make integration
Note: You will need to ensure that ~/go/bin is on your PATH for gox to work - the underlying tool that we use for building the binaries.
When you are done with testing simply execute make setup-down to terminate the docker-compose stack.
To reset the testing resources execute make setup-reset which will rebuild the docker containers. This can be useful if the state of these systems gets out of sync with what the tests expect.
To remove all build files:
guest> make clean
To format the golang code in the source directory:
guest> make format
Note: Always run make format before submitting any code.
Note: The make test command also generates a code coverage file which can be found at build/coverage/coverage.html.
How to use?
First either compile the binary from source using the above make command or download the published Binary directly from the GitHub release:
CLI Output
sql-runner version: 0.11.0
Run playbooks of SQL scripts in series and parallel on Redshift and Postgres
Usage:
-checkLock string
Checks whether the lockfile already exists
-consul string
The address of a consul server with playbooks and SQL files stored in KV pairs
-consulOnlyForLock
Will read playbooks locally, but use Consul for locking.
-deleteLock string
Will attempt to delete a lockfile if it exists
-dryRun
Runs through a playbook without executing any of the SQL
-fillTemplates
Will print all queries after templates are filled
-fromStep string
Starts from a given step defined in your playbook
-help
Shows this message
-lock string
Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully
-playbook string
Playbook of SQL scripts to execute
-runQuery string
Will run a single query in the playbook
-showQueryOutput
Will print all output from queries
-softLock string
Optional argument, like '-lock' but the lockfile will be deleted even if the run fails
-sqlroot string
Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")
-var value
Variables to be passed to the playbook, in the key=value format
-version
Shows the program version
Copyright and license
SQL Runner is copyright 2015-2022 Snowplow Analytics Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
About
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
snowplowanalytics.com
Topics
bigquerypostgresqlredshiftsnowflakesnowplowsql-runner
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
81 stars
Watchers
21 watching
Forks
15 forks
Report repository
Releases
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - snowplow/sql-runner: Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake · GitHub
Skip to content
BranchesTags
Repository files navigation
SQL Runner
Build StatusCoverallsGo Report CardReleaseLicense
Overview
Run playbooks of SQL scripts in series and parallel on Snowflake DB, Amazon Redshift and PostgreSQL.
Used with Snowplow for scheduled SQL-based transformations of event stream data.
Setup GuideUser Guide
i1i2
Quick start
Assuming go, docker and docker-compose are installed:
 host> git clone https://github.com/snowplow/sql-runner
host>cd sql-runner
host> make setup-up # Launches Consul + Postgres for testing
host> make # Builds sql-runner binaries
host> make test# Runs unit tests# DISTRO specifies which binary you want to run integration tests with
host> DISTRO=darwin make integration
Note: You will need to ensure that ~/go/bin is on your PATH for gox to work - the underlying tool that we use for building the binaries.
When you are done with testing simply execute make setup-down to terminate the docker-compose stack.
To reset the testing resources execute make setup-reset which will rebuild the docker containers. This can be useful if the state of these systems gets out of sync with what the tests expect.
To remove all build files:
guest> make clean
To format the golang code in the source directory:
guest> make format
Note: Always run make format before submitting any code.
Note: The make test command also generates a code coverage file which can be found at build/coverage/coverage.html.
How to use?
First either compile the binary from source using the above make command or download the published Binary directly from the GitHub release:
CLI Output
sql-runner version: 0.11.0
Run playbooks of SQL scripts in series and parallel on Redshift and Postgres
Usage:
-checkLock string
Checks whether the lockfile already exists
-consul string
The address of a consul server with playbooks and SQL files stored in KV pairs
-consulOnlyForLock
Will read playbooks locally, but use Consul for locking.
-deleteLock string
Will attempt to delete a lockfile if it exists
-dryRun
Runs through a playbook without executing any of the SQL
-fillTemplates
Will print all queries after templates are filled
-fromStep string
Starts from a given step defined in your playbook
-help
Shows this message
-lock string
Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully
-playbook string
Playbook of SQL scripts to execute
-runQuery string
Will run a single query in the playbook
-showQueryOutput
Will print all output from queries
-softLock string
Optional argument, like '-lock' but the lockfile will be deleted even if the run fails
-sqlroot string
Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")
-var value
Variables to be passed to the playbook, in the key=value format
-version
Shows the program version
Copyright and license
SQL Runner is copyright 2015-2022 Snowplow Analytics Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
About
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
snowplowanalytics.com
Topics
bigquerypostgresqlredshiftsnowflakesnowplowsql-runner
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
81 stars
Watchers
21 watching
Forks
15 forks
Report repository
Releases
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - snowplow/sql-runner: Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake · GitHub
Skip to content
BranchesTags
Repository files navigation
SQL Runner
Build StatusCoverallsGo Report CardReleaseLicense
Overview
Run playbooks of SQL scripts in series and parallel on Snowflake DB, Amazon Redshift and PostgreSQL.
Used with Snowplow for scheduled SQL-based transformations of event stream data.
Setup GuideUser Guide
i1i2
Quick start
Assuming go, docker and docker-compose are installed:
 host> git clone https://github.com/snowplow/sql-runner
host>cd sql-runner
host> make setup-up # Launches Consul + Postgres for testing
host> make # Builds sql-runner binaries
host> make test# Runs unit tests# DISTRO specifies which binary you want to run integration tests with
host> DISTRO=darwin make integration
Note: You will need to ensure that ~/go/bin is on your PATH for gox to work - the underlying tool that we use for building the binaries.
When you are done with testing simply execute make setup-down to terminate the docker-compose stack.
To reset the testing resources execute make setup-reset which will rebuild the docker containers. This can be useful if the state of these systems gets out of sync with what the tests expect.
To remove all build files:
guest> make clean
To format the golang code in the source directory:
guest> make format
Note: Always run make format before submitting any code.
Note: The make test command also generates a code coverage file which can be found at build/coverage/coverage.html.
How to use?
First either compile the binary from source using the above make command or download the published Binary directly from the GitHub release:
CLI Output
sql-runner version: 0.11.0
Run playbooks of SQL scripts in series and parallel on Redshift and Postgres
Usage:
-checkLock string
Checks whether the lockfile already exists
-consul string
The address of a consul server with playbooks and SQL files stored in KV pairs
-consulOnlyForLock
Will read playbooks locally, but use Consul for locking.
-deleteLock string
Will attempt to delete a lockfile if it exists
-dryRun
Runs through a playbook without executing any of the SQL
-fillTemplates
Will print all queries after templates are filled
-fromStep string
Starts from a given step defined in your playbook
-help
Shows this message
-lock string
Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully
-playbook string
Playbook of SQL scripts to execute
-runQuery string
Will run a single query in the playbook
-showQueryOutput
Will print all output from queries
-softLock string
Optional argument, like '-lock' but the lockfile will be deleted even if the run fails
-sqlroot string
Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")
-var value
Variables to be passed to the playbook, in the key=value format
-version
Shows the program version
Copyright and license
SQL Runner is copyright 2015-2022 Snowplow Analytics Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
About
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
snowplowanalytics.com
Topics
bigquerypostgresqlredshiftsnowflakesnowplowsql-runner
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
81 stars
Watchers
21 watching
Forks
15 forks
Report repository
Releases
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - snowplow/sql-runner: Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake · GitHub
Skip to content
BranchesTags
Repository files navigation
SQL Runner
Build StatusCoverallsGo Report CardReleaseLicense
Overview
Run playbooks of SQL scripts in series and parallel on Snowflake DB, Amazon Redshift and PostgreSQL.
Used with Snowplow for scheduled SQL-based transformations of event stream data.
Setup GuideUser Guide
i1i2
Quick start
Assuming go, docker and docker-compose are installed:
 host> git clone https://github.com/snowplow/sql-runner
host>cd sql-runner
host> make setup-up # Launches Consul + Postgres for testing
host> make # Builds sql-runner binaries
host> make test# Runs unit tests# DISTRO specifies which binary you want to run integration tests with
host> DISTRO=darwin make integration
Note: You will need to ensure that ~/go/bin is on your PATH for gox to work - the underlying tool that we use for building the binaries.
When you are done with testing simply execute make setup-down to terminate the docker-compose stack.
To reset the testing resources execute make setup-reset which will rebuild the docker containers. This can be useful if the state of these systems gets out of sync with what the tests expect.
To remove all build files:
guest> make clean
To format the golang code in the source directory:
guest> make format
Note: Always run make format before submitting any code.
Note: The make test command also generates a code coverage file which can be found at build/coverage/coverage.html.
How to use?
First either compile the binary from source using the above make command or download the published Binary directly from the GitHub release:
CLI Output
sql-runner version: 0.11.0
Run playbooks of SQL scripts in series and parallel on Redshift and Postgres
Usage:
-checkLock string
Checks whether the lockfile already exists
-consul string
The address of a consul server with playbooks and SQL files stored in KV pairs
-consulOnlyForLock
Will read playbooks locally, but use Consul for locking.
-deleteLock string
Will attempt to delete a lockfile if it exists
-dryRun
Runs through a playbook without executing any of the SQL
-fillTemplates
Will print all queries after templates are filled
-fromStep string
Starts from a given step defined in your playbook
-help
Shows this message
-lock string
Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully
-playbook string
Playbook of SQL scripts to execute
-runQuery string
Will run a single query in the playbook
-showQueryOutput
Will print all output from queries
-softLock string
Optional argument, like '-lock' but the lockfile will be deleted even if the run fails
-sqlroot string
Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")
-var value
Variables to be passed to the playbook, in the key=value format
-version
Shows the program version
Copyright and license
SQL Runner is copyright 2015-2022 Snowplow Analytics Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
About
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
snowplowanalytics.com
Topics
bigquerypostgresqlredshiftsnowflakesnowplowsql-runner
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
81 stars
Watchers
21 watching
Forks
15 forks
Report repository
Releases
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
})();
GitHub - snowplow/sql-runner: Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake · GitHub
Skip to content
BranchesTags
Repository files navigation
SQL Runner
Build StatusCoverallsGo Report CardReleaseLicense
Overview
Run playbooks of SQL scripts in series and parallel on Snowflake DB, Amazon Redshift and PostgreSQL.
Used with Snowplow for scheduled SQL-based transformations of event stream data.
Setup GuideUser Guide
i1i2
Quick start
Assuming go, docker and docker-compose are installed:
 host> git clone https://github.com/snowplow/sql-runner
host>cd sql-runner
host> make setup-up # Launches Consul + Postgres for testing
host> make # Builds sql-runner binaries
host> make test# Runs unit tests# DISTRO specifies which binary you want to run integration tests with
host> DISTRO=darwin make integration
Note: You will need to ensure that ~/go/bin is on your PATH for gox to work - the underlying tool that we use for building the binaries.
When you are done with testing simply execute make setup-down to terminate the docker-compose stack.
To reset the testing resources execute make setup-reset which will rebuild the docker containers. This can be useful if the state of these systems gets out of sync with what the tests expect.
To remove all build files:
guest> make clean
To format the golang code in the source directory:
guest> make format
Note: Always run make format before submitting any code.
Note: The make test command also generates a code coverage file which can be found at build/coverage/coverage.html.
How to use?
First either compile the binary from source using the above make command or download the published Binary directly from the GitHub release:
CLI Output
sql-runner version: 0.11.0
Run playbooks of SQL scripts in series and parallel on Redshift and Postgres
Usage:
-checkLock string
Checks whether the lockfile already exists
-consul string
The address of a consul server with playbooks and SQL files stored in KV pairs
-consulOnlyForLock
Will read playbooks locally, but use Consul for locking.
-deleteLock string
Will attempt to delete a lockfile if it exists
-dryRun
Runs through a playbook without executing any of the SQL
-fillTemplates
Will print all queries after templates are filled
-fromStep string
Starts from a given step defined in your playbook
-help
Shows this message
-lock string
Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully
-playbook string
Playbook of SQL scripts to execute
-runQuery string
Will run a single query in the playbook
-showQueryOutput
Will print all output from queries
-softLock string
Optional argument, like '-lock' but the lockfile will be deleted even if the run fails
-sqlroot string
Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILD for those respective paths (default "PLAYBOOK")
-var value
Variables to be passed to the playbook, in the key=value format
-version
Shows the program version
Copyright and license
SQL Runner is copyright 2015-2022 Snowplow Analytics Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
About
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
snowplowanalytics.com
Topics
bigquerypostgresqlredshiftsnowflakesnowplowsql-runner
Resources
Readme
Apache-2.0 license
Activity
Custom properties
Stars
81 stars
Watchers
21 watching
Forks
15 forks
Report repository
Releases
Used by
Contributors
Languages