BranchesTags

Repository files navigation

SArchive

crates.ioTravis Build StatusActions StatusCoverage StatusLicenseLoC.

Archival tool for scheduler job scripts and accompanying files.

Note that the master branch here may be running ahead of the latest release on crates.io. During development, we sometimes rely on dependencies that have not yet released a version with the features we use.

Minimum supported rustc

1.70.0

CI tests run against the following Rust versions:

  • stable
  • nightly

If you do not have Rust, please see Rustup for installation instructions.

Usage

sarchive requires that the path to the scheduler's main spool directory is specified. It also requires a cluster (name) to be set.

sarchive supports multiple schedulers, the one to be used must be specified on the command line. Right now, there is support for Slurm and Torque.

For Slurm, the directory to watch is defined as the StateSaveLocation in the slurm config.

Furthermore, sarchive offers various backends. The basic file backend writes a copy of the job scripts and associated files to a directory on a mounted filesystem. We also have limited support for sending job information to Elasticsearch or produce to a Kafka topic. We briefly discuss these backends below.

File archival

Activated using the file subcommand. Note that we do not support using multiple subcommands (i.e., backends) at this moment.

For file archival, sarchive requires the path to the archive's top directory, i.e., where you want to store the backup scripts and accompanying files.

The archive can be further divided into subdirectories per

  • year: YYYY, by provinding --period=yearly
  • month: YYYYMM, by providing --period=monthly
  • day: YYYYMMDD, by providing --period=daily Each of these directories are also created upon file archival if they do not exist. This allows for easily tarring old(er) directories you still wish to keep around, but probably no longer immediately need for user support.

For example,

sarchive --cluster huppel -s /var/spool/slurm file --archive=/var/backups/slurm/job-archive

Elasticsearch archival (removed)

The Elasticsearch backend will be revamped, as using the elastic crate is subject to a vulnerability through its hyper dependency (https://rustsec.org/advisories/RUSTSEC-2021-0078)

This will be added again once we can move to the official Elastic.co crate.

Kafka archival

You can ship the job scripts as messages to Kafka.

For example,

./sarchive --cluster huppel -l /var/log/sarchive.log -s /var/spool/slurm/ kafka --brokers mykafka.mydomain:9092 --topic slurm-job-archival

Support for SSL and SASL is available, through the --ssl and --sasl options. Both of these expect a comma-separated list of options to pass to the underlying kafka library.

Features

  • Multithreaded, watching one dir per thread, so no need for hierarchical watching.
  • Separate processing thread to ensure swift draining of the inotify event queues.
  • Clean log rotation when SIGHUP is received.
  • Experimental support for clean termination on receipt of SIGTERM or SIGINT, where job events that have already been seen are processed, to minimise potential loss when restarting the service.
  • Output to a file in a hierarchical directory structure
  • Output to Elasticsearch
  • Output to Kafka

RPMs

We provide a build script to generate an RPM using the cargo-rpm tool. You may tailor the spec file (listed under the .rpm directory) to fit your needs. The RPM includes a unit file so sarchive can be started as a service by systemd. This file should also be changed to fit your requirements and local configuration.

About

Slurm job script archival

Topics

archivalhpcschedulerslurmtorque

Resources

Readme
MIT license

Code of conduct

Code of conduct
Activity

Stars

12 stars

Watchers

2 watching

Forks

3 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
SArchive
crates.ioTravis Build StatusActions StatusCoverage StatusLicenseLoC.
Archival tool for scheduler job scripts and accompanying files.
Note that the master branch here may be running ahead of the latest release
on crates.io. During development, we sometimes rely on dependencies
that have not yet released a version with the features we use.
Minimum supported rustc
1.70.0
CI tests run against the following Rust versions:
  • stable
  • nightly
If you do not have Rust, please see
Rustup for installation instructions.
Usage
sarchive requires that the path to the scheduler's main spool directory is
specified. It also requires a cluster (name) to be set.
sarchive supports multiple schedulers, the one to be used must be specified
on the command line. Right now, there is support for Slurm
and Torque.
For Slurm, the directory to watch is defined as the StateSaveLocation in the slurm config.
Furthermore, sarchive offers various backends. The basic file backend
writes a copy of the job scripts and associated files to a directory on a
mounted filesystem. We also have limited support for sending job information
to Elasticsearch or produce to a
Kafka topic. We briefly discuss these backends
below.
File archival
Activated using the file subcommand. Note that we do not support using
multiple subcommands (i.e., backends) at this moment.
For file archival, sarchive requires the path to the archive's top
directory, i.e., where you want to store the backup scripts and accompanying
files.
The archive can be further divided into subdirectories per
  • year: YYYY, by provinding --period=yearly
  • month: YYYYMM, by providing --period=monthly
  • day: YYYYMMDD, by providing --period=daily
Each of these directories are also created upon file archival if they do
not exist. This allows for easily tarring old(er) directories you still
wish to keep around, but probably no longer immediately need for user support.
For example,
sarchive --cluster huppel -s /var/spool/slurm file --archive=/var/backups/slurm/job-archive
Elasticsearch archival (removed)
The Elasticsearch backend will be revamped, as using the elastic crate is subject to a
vulnerability through its hyper dependency (https://rustsec.org/advisories/RUSTSEC-2021-0078)
This will be added again once we can move to the official Elastic.co crate.
Kafka archival
You can ship the job scripts as messages to Kafka.
For example,
./sarchive --cluster huppel -l /var/log/sarchive.log -s /var/spool/slurm/ kafka --brokers mykafka.mydomain:9092 --topic slurm-job-archival
Support for SSL and SASL is available, through the --ssl and --sasl options. Both of these expect a comma-separated
list of options to pass to the underlying kafka library.
Features
  • Multithreaded, watching one dir per thread, so no need for hierarchical watching.
  • Separate processing thread to ensure swift draining of the inotify event queues.
  • Clean log rotation when SIGHUP is received.
  • Experimental support for clean termination on receipt of SIGTERM or SIGINT, where
job events that have already been seen are processed, to minimise potential loss
when restarting the service.
  • Output to a file in a hierarchical directory structure
  • Output to Elasticsearch
  • Output to Kafka
RPMs
We provide a build script to generate an RPM using the cargo-rpm tool. You may tailor the spec
file (listed under the .rpm directory) to fit your needs. The RPM includes a unit file so
sarchive can be started as a service by systemd. This file should also be changed to fit your
requirements and local configuration.
About
Slurm job script archival
Topics
archivalhpcschedulerslurmtorque
Resources
Readme
MIT license
Code of conduct
Code of conduct
Activity
Stars
12 stars
Watchers
2 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
SArchive
crates.ioTravis Build StatusActions StatusCoverage StatusLicenseLoC.
Archival tool for scheduler job scripts and accompanying files.
Note that the master branch here may be running ahead of the latest release
on crates.io. During development, we sometimes rely on dependencies
that have not yet released a version with the features we use.
Minimum supported rustc
1.70.0
CI tests run against the following Rust versions:
  • stable
  • nightly
If you do not have Rust, please see
Rustup for installation instructions.
Usage
sarchive requires that the path to the scheduler's main spool directory is
specified. It also requires a cluster (name) to be set.
sarchive supports multiple schedulers, the one to be used must be specified
on the command line. Right now, there is support for Slurm
and Torque.
For Slurm, the directory to watch is defined as the StateSaveLocation in the slurm config.
Furthermore, sarchive offers various backends. The basic file backend
writes a copy of the job scripts and associated files to a directory on a
mounted filesystem. We also have limited support for sending job information
to Elasticsearch or produce to a
Kafka topic. We briefly discuss these backends
below.
File archival
Activated using the file subcommand. Note that we do not support using
multiple subcommands (i.e., backends) at this moment.
For file archival, sarchive requires the path to the archive's top
directory, i.e., where you want to store the backup scripts and accompanying
files.
The archive can be further divided into subdirectories per
  • year: YYYY, by provinding --period=yearly
  • month: YYYYMM, by providing --period=monthly
  • day: YYYYMMDD, by providing --period=daily
Each of these directories are also created upon file archival if they do
not exist. This allows for easily tarring old(er) directories you still
wish to keep around, but probably no longer immediately need for user support.
For example,
sarchive --cluster huppel -s /var/spool/slurm file --archive=/var/backups/slurm/job-archive
Elasticsearch archival (removed)
The Elasticsearch backend will be revamped, as using the elastic crate is subject to a
vulnerability through its hyper dependency (https://rustsec.org/advisories/RUSTSEC-2021-0078)
This will be added again once we can move to the official Elastic.co crate.
Kafka archival
You can ship the job scripts as messages to Kafka.
For example,
./sarchive --cluster huppel -l /var/log/sarchive.log -s /var/spool/slurm/ kafka --brokers mykafka.mydomain:9092 --topic slurm-job-archival
Support for SSL and SASL is available, through the --ssl and --sasl options. Both of these expect a comma-separated
list of options to pass to the underlying kafka library.
Features
  • Multithreaded, watching one dir per thread, so no need for hierarchical watching.
  • Separate processing thread to ensure swift draining of the inotify event queues.
  • Clean log rotation when SIGHUP is received.
  • Experimental support for clean termination on receipt of SIGTERM or SIGINT, where
job events that have already been seen are processed, to minimise potential loss
when restarting the service.
  • Output to a file in a hierarchical directory structure
  • Output to Elasticsearch
  • Output to Kafka
RPMs
We provide a build script to generate an RPM using the cargo-rpm tool. You may tailor the spec
file (listed under the .rpm directory) to fit your needs. The RPM includes a unit file so
sarchive can be started as a service by systemd. This file should also be changed to fit your
requirements and local configuration.
About
Slurm job script archival
Topics
archivalhpcschedulerslurmtorque
Resources
Readme
MIT license
Code of conduct
Code of conduct
Activity
Stars
12 stars
Watchers
2 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
SArchive
crates.ioTravis Build StatusActions StatusCoverage StatusLicenseLoC.
Archival tool for scheduler job scripts and accompanying files.
Note that the master branch here may be running ahead of the latest release
on crates.io. During development, we sometimes rely on dependencies
that have not yet released a version with the features we use.
Minimum supported rustc
1.70.0
CI tests run against the following Rust versions:
  • stable
  • nightly
If you do not have Rust, please see
Rustup for installation instructions.
Usage
sarchive requires that the path to the scheduler's main spool directory is
specified. It also requires a cluster (name) to be set.
sarchive supports multiple schedulers, the one to be used must be specified
on the command line. Right now, there is support for Slurm
and Torque.
For Slurm, the directory to watch is defined as the StateSaveLocation in the slurm config.
Furthermore, sarchive offers various backends. The basic file backend
writes a copy of the job scripts and associated files to a directory on a
mounted filesystem. We also have limited support for sending job information
to Elasticsearch or produce to a
Kafka topic. We briefly discuss these backends
below.
File archival
Activated using the file subcommand. Note that we do not support using
multiple subcommands (i.e., backends) at this moment.
For file archival, sarchive requires the path to the archive's top
directory, i.e., where you want to store the backup scripts and accompanying
files.
The archive can be further divided into subdirectories per
  • year: YYYY, by provinding --period=yearly
  • month: YYYYMM, by providing --period=monthly
  • day: YYYYMMDD, by providing --period=daily
Each of these directories are also created upon file archival if they do
not exist. This allows for easily tarring old(er) directories you still
wish to keep around, but probably no longer immediately need for user support.
For example,
sarchive --cluster huppel -s /var/spool/slurm file --archive=/var/backups/slurm/job-archive
Elasticsearch archival (removed)
The Elasticsearch backend will be revamped, as using the elastic crate is subject to a
vulnerability through its hyper dependency (https://rustsec.org/advisories/RUSTSEC-2021-0078)
This will be added again once we can move to the official Elastic.co crate.
Kafka archival
You can ship the job scripts as messages to Kafka.
For example,
./sarchive --cluster huppel -l /var/log/sarchive.log -s /var/spool/slurm/ kafka --brokers mykafka.mydomain:9092 --topic slurm-job-archival
Support for SSL and SASL is available, through the --ssl and --sasl options. Both of these expect a comma-separated
list of options to pass to the underlying kafka library.
Features
  • Multithreaded, watching one dir per thread, so no need for hierarchical watching.
  • Separate processing thread to ensure swift draining of the inotify event queues.
  • Clean log rotation when SIGHUP is received.
  • Experimental support for clean termination on receipt of SIGTERM or SIGINT, where
job events that have already been seen are processed, to minimise potential loss
when restarting the service.
  • Output to a file in a hierarchical directory structure
  • Output to Elasticsearch
  • Output to Kafka
RPMs
We provide a build script to generate an RPM using the cargo-rpm tool. You may tailor the spec
file (listed under the .rpm directory) to fit your needs. The RPM includes a unit file so
sarchive can be started as a service by systemd. This file should also be changed to fit your
requirements and local configuration.
About
Slurm job script archival
Topics
archivalhpcschedulerslurmtorque
Resources
Readme
MIT license
Code of conduct
Code of conduct
Activity
Stars
12 stars
Watchers
2 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
SArchive
crates.ioTravis Build StatusActions StatusCoverage StatusLicenseLoC.
Archival tool for scheduler job scripts and accompanying files.
Note that the master branch here may be running ahead of the latest release
on crates.io. During development, we sometimes rely on dependencies
that have not yet released a version with the features we use.
Minimum supported rustc
1.70.0
CI tests run against the following Rust versions:
  • stable
  • nightly
If you do not have Rust, please see
Rustup for installation instructions.
Usage
sarchive requires that the path to the scheduler's main spool directory is
specified. It also requires a cluster (name) to be set.
sarchive supports multiple schedulers, the one to be used must be specified
on the command line. Right now, there is support for Slurm
and Torque.
For Slurm, the directory to watch is defined as the StateSaveLocation in the slurm config.
Furthermore, sarchive offers various backends. The basic file backend
writes a copy of the job scripts and associated files to a directory on a
mounted filesystem. We also have limited support for sending job information
to Elasticsearch or produce to a
Kafka topic. We briefly discuss these backends
below.
File archival
Activated using the file subcommand. Note that we do not support using
multiple subcommands (i.e., backends) at this moment.
For file archival, sarchive requires the path to the archive's top
directory, i.e., where you want to store the backup scripts and accompanying
files.
The archive can be further divided into subdirectories per
  • year: YYYY, by provinding --period=yearly
  • month: YYYYMM, by providing --period=monthly
  • day: YYYYMMDD, by providing --period=daily
Each of these directories are also created upon file archival if they do
not exist. This allows for easily tarring old(er) directories you still
wish to keep around, but probably no longer immediately need for user support.
For example,
sarchive --cluster huppel -s /var/spool/slurm file --archive=/var/backups/slurm/job-archive
Elasticsearch archival (removed)
The Elasticsearch backend will be revamped, as using the elastic crate is subject to a
vulnerability through its hyper dependency (https://rustsec.org/advisories/RUSTSEC-2021-0078)
This will be added again once we can move to the official Elastic.co crate.
Kafka archival
You can ship the job scripts as messages to Kafka.
For example,
./sarchive --cluster huppel -l /var/log/sarchive.log -s /var/spool/slurm/ kafka --brokers mykafka.mydomain:9092 --topic slurm-job-archival
Support for SSL and SASL is available, through the --ssl and --sasl options. Both of these expect a comma-separated
list of options to pass to the underlying kafka library.
Features
  • Multithreaded, watching one dir per thread, so no need for hierarchical watching.
  • Separate processing thread to ensure swift draining of the inotify event queues.
  • Clean log rotation when SIGHUP is received.
  • Experimental support for clean termination on receipt of SIGTERM or SIGINT, where
job events that have already been seen are processed, to minimise potential loss
when restarting the service.
  • Output to a file in a hierarchical directory structure
  • Output to Elasticsearch
  • Output to Kafka
RPMs
We provide a build script to generate an RPM using the cargo-rpm tool. You may tailor the spec
file (listed under the .rpm directory) to fit your needs. The RPM includes a unit file so
sarchive can be started as a service by systemd. This file should also be changed to fit your
requirements and local configuration.
About
Slurm job script archival
Topics
archivalhpcschedulerslurmtorque
Resources
Readme
MIT license
Code of conduct
Code of conduct
Activity
Stars
12 stars
Watchers
2 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
SArchive
crates.ioTravis Build StatusActions StatusCoverage StatusLicenseLoC.
Archival tool for scheduler job scripts and accompanying files.
Note that the master branch here may be running ahead of the latest release
on crates.io. During development, we sometimes rely on dependencies
that have not yet released a version with the features we use.
Minimum supported rustc
1.70.0
CI tests run against the following Rust versions:
  • stable
  • nightly
If you do not have Rust, please see
Rustup for installation instructions.
Usage
sarchive requires that the path to the scheduler's main spool directory is
specified. It also requires a cluster (name) to be set.
sarchive supports multiple schedulers, the one to be used must be specified
on the command line. Right now, there is support for Slurm
and Torque.
For Slurm, the directory to watch is defined as the StateSaveLocation in the slurm config.
Furthermore, sarchive offers various backends. The basic file backend
writes a copy of the job scripts and associated files to a directory on a
mounted filesystem. We also have limited support for sending job information
to Elasticsearch or produce to a
Kafka topic. We briefly discuss these backends
below.
File archival
Activated using the file subcommand. Note that we do not support using
multiple subcommands (i.e., backends) at this moment.
For file archival, sarchive requires the path to the archive's top
directory, i.e., where you want to store the backup scripts and accompanying
files.
The archive can be further divided into subdirectories per
  • year: YYYY, by provinding --period=yearly
  • month: YYYYMM, by providing --period=monthly
  • day: YYYYMMDD, by providing --period=daily
Each of these directories are also created upon file archival if they do
not exist. This allows for easily tarring old(er) directories you still
wish to keep around, but probably no longer immediately need for user support.
For example,
sarchive --cluster huppel -s /var/spool/slurm file --archive=/var/backups/slurm/job-archive
Elasticsearch archival (removed)
The Elasticsearch backend will be revamped, as using the elastic crate is subject to a
vulnerability through its hyper dependency (https://rustsec.org/advisories/RUSTSEC-2021-0078)
This will be added again once we can move to the official Elastic.co crate.
Kafka archival
You can ship the job scripts as messages to Kafka.
For example,
./sarchive --cluster huppel -l /var/log/sarchive.log -s /var/spool/slurm/ kafka --brokers mykafka.mydomain:9092 --topic slurm-job-archival
Support for SSL and SASL is available, through the --ssl and --sasl options. Both of these expect a comma-separated
list of options to pass to the underlying kafka library.
Features
  • Multithreaded, watching one dir per thread, so no need for hierarchical watching.
  • Separate processing thread to ensure swift draining of the inotify event queues.
  • Clean log rotation when SIGHUP is received.
  • Experimental support for clean termination on receipt of SIGTERM or SIGINT, where
job events that have already been seen are processed, to minimise potential loss
when restarting the service.
  • Output to a file in a hierarchical directory structure
  • Output to Elasticsearch
  • Output to Kafka
RPMs
We provide a build script to generate an RPM using the cargo-rpm tool. You may tailor the spec
file (listed under the .rpm directory) to fit your needs. The RPM includes a unit file so
sarchive can be started as a service by systemd. This file should also be changed to fit your
requirements and local configuration.
About
Slurm job script archival
Topics
archivalhpcschedulerslurmtorque
Resources
Readme
MIT license
Code of conduct
Code of conduct
Activity
Stars
12 stars
Watchers
2 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
SArchive
crates.ioTravis Build StatusActions StatusCoverage StatusLicenseLoC.
Archival tool for scheduler job scripts and accompanying files.
Note that the master branch here may be running ahead of the latest release
on crates.io. During development, we sometimes rely on dependencies
that have not yet released a version with the features we use.
Minimum supported rustc
1.70.0
CI tests run against the following Rust versions:
  • stable
  • nightly
If you do not have Rust, please see
Rustup for installation instructions.
Usage
sarchive requires that the path to the scheduler's main spool directory is
specified. It also requires a cluster (name) to be set.
sarchive supports multiple schedulers, the one to be used must be specified
on the command line. Right now, there is support for Slurm
and Torque.
For Slurm, the directory to watch is defined as the StateSaveLocation in the slurm config.
Furthermore, sarchive offers various backends. The basic file backend
writes a copy of the job scripts and associated files to a directory on a
mounted filesystem. We also have limited support for sending job information
to Elasticsearch or produce to a
Kafka topic. We briefly discuss these backends
below.
File archival
Activated using the file subcommand. Note that we do not support using
multiple subcommands (i.e., backends) at this moment.
For file archival, sarchive requires the path to the archive's top
directory, i.e., where you want to store the backup scripts and accompanying
files.
The archive can be further divided into subdirectories per
  • year: YYYY, by provinding --period=yearly
  • month: YYYYMM, by providing --period=monthly
  • day: YYYYMMDD, by providing --period=daily
Each of these directories are also created upon file archival if they do
not exist. This allows for easily tarring old(er) directories you still
wish to keep around, but probably no longer immediately need for user support.
For example,
sarchive --cluster huppel -s /var/spool/slurm file --archive=/var/backups/slurm/job-archive
Elasticsearch archival (removed)
The Elasticsearch backend will be revamped, as using the elastic crate is subject to a
vulnerability through its hyper dependency (https://rustsec.org/advisories/RUSTSEC-2021-0078)
This will be added again once we can move to the official Elastic.co crate.
Kafka archival
You can ship the job scripts as messages to Kafka.
For example,
./sarchive --cluster huppel -l /var/log/sarchive.log -s /var/spool/slurm/ kafka --brokers mykafka.mydomain:9092 --topic slurm-job-archival
Support for SSL and SASL is available, through the --ssl and --sasl options. Both of these expect a comma-separated
list of options to pass to the underlying kafka library.
Features
  • Multithreaded, watching one dir per thread, so no need for hierarchical watching.
  • Separate processing thread to ensure swift draining of the inotify event queues.
  • Clean log rotation when SIGHUP is received.
  • Experimental support for clean termination on receipt of SIGTERM or SIGINT, where
job events that have already been seen are processed, to minimise potential loss
when restarting the service.
  • Output to a file in a hierarchical directory structure
  • Output to Elasticsearch
  • Output to Kafka
RPMs
We provide a build script to generate an RPM using the cargo-rpm tool. You may tailor the spec
file (listed under the .rpm directory) to fit your needs. The RPM includes a unit file so
sarchive can be started as a service by systemd. This file should also be changed to fit your
requirements and local configuration.
About
Slurm job script archival
Topics
archivalhpcschedulerslurmtorque
Resources
Readme
MIT license
Code of conduct
Code of conduct
Activity
Stars
12 stars
Watchers
2 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
SArchive
crates.ioTravis Build StatusActions StatusCoverage StatusLicenseLoC.
Archival tool for scheduler job scripts and accompanying files.
Note that the master branch here may be running ahead of the latest release
on crates.io. During development, we sometimes rely on dependencies
that have not yet released a version with the features we use.
Minimum supported rustc
1.70.0
CI tests run against the following Rust versions:
  • stable
  • nightly
If you do not have Rust, please see
Rustup for installation instructions.
Usage
sarchive requires that the path to the scheduler's main spool directory is
specified. It also requires a cluster (name) to be set.
sarchive supports multiple schedulers, the one to be used must be specified
on the command line. Right now, there is support for Slurm
and Torque.
For Slurm, the directory to watch is defined as the StateSaveLocation in the slurm config.
Furthermore, sarchive offers various backends. The basic file backend
writes a copy of the job scripts and associated files to a directory on a
mounted filesystem. We also have limited support for sending job information
to Elasticsearch or produce to a
Kafka topic. We briefly discuss these backends
below.
File archival
Activated using the file subcommand. Note that we do not support using
multiple subcommands (i.e., backends) at this moment.
For file archival, sarchive requires the path to the archive's top
directory, i.e., where you want to store the backup scripts and accompanying
files.
The archive can be further divided into subdirectories per
  • year: YYYY, by provinding --period=yearly
  • month: YYYYMM, by providing --period=monthly
  • day: YYYYMMDD, by providing --period=daily
Each of these directories are also created upon file archival if they do
not exist. This allows for easily tarring old(er) directories you still
wish to keep around, but probably no longer immediately need for user support.
For example,
sarchive --cluster huppel -s /var/spool/slurm file --archive=/var/backups/slurm/job-archive
Elasticsearch archival (removed)
The Elasticsearch backend will be revamped, as using the elastic crate is subject to a
vulnerability through its hyper dependency (https://rustsec.org/advisories/RUSTSEC-2021-0078)
This will be added again once we can move to the official Elastic.co crate.
Kafka archival
You can ship the job scripts as messages to Kafka.
For example,
./sarchive --cluster huppel -l /var/log/sarchive.log -s /var/spool/slurm/ kafka --brokers mykafka.mydomain:9092 --topic slurm-job-archival
Support for SSL and SASL is available, through the --ssl and --sasl options. Both of these expect a comma-separated
list of options to pass to the underlying kafka library.
Features
  • Multithreaded, watching one dir per thread, so no need for hierarchical watching.
  • Separate processing thread to ensure swift draining of the inotify event queues.
  • Clean log rotation when SIGHUP is received.
  • Experimental support for clean termination on receipt of SIGTERM or SIGINT, where
job events that have already been seen are processed, to minimise potential loss
when restarting the service.
  • Output to a file in a hierarchical directory structure
  • Output to Elasticsearch
  • Output to Kafka
RPMs
We provide a build script to generate an RPM using the cargo-rpm tool. You may tailor the spec
file (listed under the .rpm directory) to fit your needs. The RPM includes a unit file so
sarchive can be started as a service by systemd. This file should also be changed to fit your
requirements and local configuration.
About
Slurm job script archival
Topics
archivalhpcschedulerslurmtorque
Resources
Readme
MIT license
Code of conduct
Code of conduct
Activity
Stars
12 stars
Watchers
2 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages