Latest commit

History

104 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

OrangeGrid Examples

This repository provides code examples for commonly used applications within the OrangeGrid cluster. In addition to exploring these examples on the web you can use git to download them into your home directory on the cluster which will allow you to run them directly. The command is

git clone http://github.com/SyracuseUniversity/OrangeGridExamples

Start here

  • hostname: A simple example of submitting a job to the cluster, monitoring it, and checking its output.

Getting the most out of OrangeGrid

These examples discuss general techniques for optimizing performance and throughput

  • Checkpointing: Learn how to save work that your jobs are doing, so if they exit for any reason HTCondor can restart them where they left off.
  • Parallelism: Learn how to divide a task into lots of smaller tasks that can run independently so they can spread out over the cluster.
  • File management: Learn how to arrange your data into chunks that are both more efficient for later analyses and perform better on the cluster.
  • multipleJobs: Learn how to run multiple jobs from one submit file.
  • HTCondorDiagnostics: Check the status of the cluster.

Using particular languages and libraries

  • python: Use Python and Python packages with the Conda package manager.
  • uv: Use Python and Python packages with the uv package manager.
  • tensorflow: Use the Tensorflow package.
  • PyTorch: Use the PyTorch package (note, the uv example also covers this in a way that may be easier to use).
  • Ollama: Run an LLM non-interactively, choosing from among numerous models.
  • julia: Use the Julia language and its libraries.
  • octave: Use Octave, a free and open source math-oriented language that is largely compatibe with Matlab.
  • R: Use the R language and its libraries.
  • Apptainer: Simplify the installation of complex packages by pulling containers from Dockerhub or building new containers.
  • blender: Render images and movies.
  • CVMFS: Use software and data distributed through the CERN Virtual File System
  • Grobid: Use GROBID, a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured documents.
  • CUDA 1.3: Use GPUs with code requiring the latest versions of the CUDA library.
  • JAX: Optimize certain mathematical operations on GPUs.

Workflow managers

Single submit files are great for anything from a single job to thousands of jobs where the same command is run on numerous arguments. However research often entails more complex arrangements of jobs, where an initial stage will create files which are needed by a later stage or post processing can only be run once a set of analyses have completed. In general there may be arbitrary dependencies between jobs. While it is always possible to manage these manually, waiting for one set of jobs to complete before running the next, it is much more convenient to have a workflow manager handle the dependencies. There are many such systems, suitable for different situations.

  • DAGMan (Directed Acyclic Graph Manager) is HTCondor's native workflow manager. Using it involves creating a simple text file describing the jobs and their connections (or writing a program to generate such a file). We discuss DAGMan in the section on Parallelism.

  • Pegasus is an extremely powerful, but also fairly complex, workflow management system developed by the Information Sciences Institute at the University of Southern California.

  • SnakeMake is a tool for running workflows specified by the relationships between input and output files of individual processes.

  • NextFlow is framework for creating scientific workflows with an emphasis on how data moves between various stages of processing. It is inspired in part by the Unix Philosophy encapsulated in the way data flows between processes connected by pipes.

Need Help?

Additional how-to documentation, such as connecting to clusters and running jobs, is available in Answers.

If you would like to contact us directly for assistance or requesting access, email researchcomputing@syr.edu.

About

Sample code and submit files for OrangeGrid

Resources

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Latest commit

History

104 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

OrangeGrid Examples

This repository provides code examples for commonly used applications within the OrangeGrid cluster. In addition to exploring these examples on the web you can use git to download them into your home directory on the cluster which will allow you to run them directly. The command is

git clone http://github.com/SyracuseUniversity/OrangeGridExamples

Start here

  • hostname: A simple example of submitting a job to the cluster, monitoring it, and checking its output.

Getting the most out of OrangeGrid

These examples discuss general techniques for optimizing performance and throughput

  • Checkpointing: Learn how to save work that your jobs are doing, so if they exit for any reason HTCondor can restart them where they left off.
  • Parallelism: Learn how to divide a task into lots of smaller tasks that can run independently so they can spread out over the cluster.
  • File management: Learn how to arrange your data into chunks that are both more efficient for later analyses and perform better on the cluster.
  • multipleJobs: Learn how to run multiple jobs from one submit file.
  • HTCondorDiagnostics: Check the status of the cluster.

Using particular languages and libraries

  • python: Use Python and Python packages with the Conda package manager.
  • uv: Use Python and Python packages with the uv package manager.
  • tensorflow: Use the Tensorflow package.
  • PyTorch: Use the PyTorch package (note, the uv example also covers this in a way that may be easier to use).
  • Ollama: Run an LLM non-interactively, choosing from among numerous models.
  • julia: Use the Julia language and its libraries.
  • octave: Use Octave, a free and open source math-oriented language that is largely compatibe with Matlab.
  • R: Use the R language and its libraries.
  • Apptainer: Simplify the installation of complex packages by pulling containers from Dockerhub or building new containers.
  • blender: Render images and movies.
  • CVMFS: Use software and data distributed through the CERN Virtual File System
  • Grobid: Use GROBID, a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured documents.
  • CUDA 1.3: Use GPUs with code requiring the latest versions of the CUDA library.
  • JAX: Optimize certain mathematical operations on GPUs.

Workflow managers

Single submit files are great for anything from a single job to thousands of jobs where the same command is run on numerous arguments. However research often entails more complex arrangements of jobs, where an initial stage will create files which are needed by a later stage or post processing can only be run once a set of analyses have completed. In general there may be arbitrary dependencies between jobs. While it is always possible to manage these manually, waiting for one set of jobs to complete before running the next, it is much more convenient to have a workflow manager handle the dependencies. There are many such systems, suitable for different situations.

  • DAGMan (Directed Acyclic Graph Manager) is HTCondor's native workflow manager. Using it involves creating a simple text file describing the jobs and their connections (or writing a program to generate such a file). We discuss DAGMan in the section on Parallelism.

  • Pegasus is an extremely powerful, but also fairly complex, workflow management system developed by the Information Sciences Institute at the University of Southern California.

  • SnakeMake is a tool for running workflows specified by the relationships between input and output files of individual processes.

  • NextFlow is framework for creating scientific workflows with an emphasis on how data moves between various stages of processing. It is inspired in part by the Unix Philosophy encapsulated in the way data flows between processes connected by pipes.

Need Help?

Additional how-to documentation, such as connecting to clusters and running jobs, is available in Answers.

If you would like to contact us directly for assistance or requesting access, email researchcomputing@syr.edu.

About

Sample code and submit files for OrangeGrid

Resources

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Latest commit

History

104 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

OrangeGrid Examples

This repository provides code examples for commonly used applications within the OrangeGrid cluster. In addition to exploring these examples on the web you can use git to download them into your home directory on the cluster which will allow you to run them directly. The command is

git clone http://github.com/SyracuseUniversity/OrangeGridExamples

Start here

  • hostname: A simple example of submitting a job to the cluster, monitoring it, and checking its output.

Getting the most out of OrangeGrid

These examples discuss general techniques for optimizing performance and throughput

  • Checkpointing: Learn how to save work that your jobs are doing, so if they exit for any reason HTCondor can restart them where they left off.
  • Parallelism: Learn how to divide a task into lots of smaller tasks that can run independently so they can spread out over the cluster.
  • File management: Learn how to arrange your data into chunks that are both more efficient for later analyses and perform better on the cluster.
  • multipleJobs: Learn how to run multiple jobs from one submit file.
  • HTCondorDiagnostics: Check the status of the cluster.

Using particular languages and libraries

  • python: Use Python and Python packages with the Conda package manager.
  • uv: Use Python and Python packages with the uv package manager.
  • tensorflow: Use the Tensorflow package.
  • PyTorch: Use the PyTorch package (note, the uv example also covers this in a way that may be easier to use).
  • Ollama: Run an LLM non-interactively, choosing from among numerous models.
  • julia: Use the Julia language and its libraries.
  • octave: Use Octave, a free and open source math-oriented language that is largely compatibe with Matlab.
  • R: Use the R language and its libraries.
  • Apptainer: Simplify the installation of complex packages by pulling containers from Dockerhub or building new containers.
  • blender: Render images and movies.
  • CVMFS: Use software and data distributed through the CERN Virtual File System
  • Grobid: Use GROBID, a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured documents.
  • CUDA 1.3: Use GPUs with code requiring the latest versions of the CUDA library.
  • JAX: Optimize certain mathematical operations on GPUs.

Workflow managers

Single submit files are great for anything from a single job to thousands of jobs where the same command is run on numerous arguments. However research often entails more complex arrangements of jobs, where an initial stage will create files which are needed by a later stage or post processing can only be run once a set of analyses have completed. In general there may be arbitrary dependencies between jobs. While it is always possible to manage these manually, waiting for one set of jobs to complete before running the next, it is much more convenient to have a workflow manager handle the dependencies. There are many such systems, suitable for different situations.

  • DAGMan (Directed Acyclic Graph Manager) is HTCondor's native workflow manager. Using it involves creating a simple text file describing the jobs and their connections (or writing a program to generate such a file). We discuss DAGMan in the section on Parallelism.

  • Pegasus is an extremely powerful, but also fairly complex, workflow management system developed by the Information Sciences Institute at the University of Southern California.

  • SnakeMake is a tool for running workflows specified by the relationships between input and output files of individual processes.

  • NextFlow is framework for creating scientific workflows with an emphasis on how data moves between various stages of processing. It is inspired in part by the Unix Philosophy encapsulated in the way data flows between processes connected by pipes.

Need Help?

Additional how-to documentation, such as connecting to clusters and running jobs, is available in Answers.

If you would like to contact us directly for assistance or requesting access, email researchcomputing@syr.edu.

About

Sample code and submit files for OrangeGrid

Resources

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Latest commit

History

104 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

OrangeGrid Examples

This repository provides code examples for commonly used applications within the OrangeGrid cluster. In addition to exploring these examples on the web you can use git to download them into your home directory on the cluster which will allow you to run them directly. The command is

git clone http://github.com/SyracuseUniversity/OrangeGridExamples

Start here

  • hostname: A simple example of submitting a job to the cluster, monitoring it, and checking its output.

Getting the most out of OrangeGrid

These examples discuss general techniques for optimizing performance and throughput

  • Checkpointing: Learn how to save work that your jobs are doing, so if they exit for any reason HTCondor can restart them where they left off.
  • Parallelism: Learn how to divide a task into lots of smaller tasks that can run independently so they can spread out over the cluster.
  • File management: Learn how to arrange your data into chunks that are both more efficient for later analyses and perform better on the cluster.
  • multipleJobs: Learn how to run multiple jobs from one submit file.
  • HTCondorDiagnostics: Check the status of the cluster.

Using particular languages and libraries

  • python: Use Python and Python packages with the Conda package manager.
  • uv: Use Python and Python packages with the uv package manager.
  • tensorflow: Use the Tensorflow package.
  • PyTorch: Use the PyTorch package (note, the uv example also covers this in a way that may be easier to use).
  • Ollama: Run an LLM non-interactively, choosing from among numerous models.
  • julia: Use the Julia language and its libraries.
  • octave: Use Octave, a free and open source math-oriented language that is largely compatibe with Matlab.
  • R: Use the R language and its libraries.
  • Apptainer: Simplify the installation of complex packages by pulling containers from Dockerhub or building new containers.
  • blender: Render images and movies.
  • CVMFS: Use software and data distributed through the CERN Virtual File System
  • Grobid: Use GROBID, a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured documents.
  • CUDA 1.3: Use GPUs with code requiring the latest versions of the CUDA library.
  • JAX: Optimize certain mathematical operations on GPUs.

Workflow managers

Single submit files are great for anything from a single job to thousands of jobs where the same command is run on numerous arguments. However research often entails more complex arrangements of jobs, where an initial stage will create files which are needed by a later stage or post processing can only be run once a set of analyses have completed. In general there may be arbitrary dependencies between jobs. While it is always possible to manage these manually, waiting for one set of jobs to complete before running the next, it is much more convenient to have a workflow manager handle the dependencies. There are many such systems, suitable for different situations.

  • DAGMan (Directed Acyclic Graph Manager) is HTCondor's native workflow manager. Using it involves creating a simple text file describing the jobs and their connections (or writing a program to generate such a file). We discuss DAGMan in the section on Parallelism.

  • Pegasus is an extremely powerful, but also fairly complex, workflow management system developed by the Information Sciences Institute at the University of Southern California.

  • SnakeMake is a tool for running workflows specified by the relationships between input and output files of individual processes.

  • NextFlow is framework for creating scientific workflows with an emphasis on how data moves between various stages of processing. It is inspired in part by the Unix Philosophy encapsulated in the way data flows between processes connected by pipes.

Need Help?

Additional how-to documentation, such as connecting to clusters and running jobs, is available in Answers.

If you would like to contact us directly for assistance or requesting access, email researchcomputing@syr.edu.

About

Sample code and submit files for OrangeGrid

Resources

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Latest commit

History

104 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

OrangeGrid Examples

This repository provides code examples for commonly used applications within the OrangeGrid cluster. In addition to exploring these examples on the web you can use git to download them into your home directory on the cluster which will allow you to run them directly. The command is

git clone http://github.com/SyracuseUniversity/OrangeGridExamples

Start here

  • hostname: A simple example of submitting a job to the cluster, monitoring it, and checking its output.

Getting the most out of OrangeGrid

These examples discuss general techniques for optimizing performance and throughput

  • Checkpointing: Learn how to save work that your jobs are doing, so if they exit for any reason HTCondor can restart them where they left off.
  • Parallelism: Learn how to divide a task into lots of smaller tasks that can run independently so they can spread out over the cluster.
  • File management: Learn how to arrange your data into chunks that are both more efficient for later analyses and perform better on the cluster.
  • multipleJobs: Learn how to run multiple jobs from one submit file.
  • HTCondorDiagnostics: Check the status of the cluster.

Using particular languages and libraries

  • python: Use Python and Python packages with the Conda package manager.
  • uv: Use Python and Python packages with the uv package manager.
  • tensorflow: Use the Tensorflow package.
  • PyTorch: Use the PyTorch package (note, the uv example also covers this in a way that may be easier to use).
  • Ollama: Run an LLM non-interactively, choosing from among numerous models.
  • julia: Use the Julia language and its libraries.
  • octave: Use Octave, a free and open source math-oriented language that is largely compatibe with Matlab.
  • R: Use the R language and its libraries.
  • Apptainer: Simplify the installation of complex packages by pulling containers from Dockerhub or building new containers.
  • blender: Render images and movies.
  • CVMFS: Use software and data distributed through the CERN Virtual File System
  • Grobid: Use GROBID, a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured documents.
  • CUDA 1.3: Use GPUs with code requiring the latest versions of the CUDA library.
  • JAX: Optimize certain mathematical operations on GPUs.

Workflow managers

Single submit files are great for anything from a single job to thousands of jobs where the same command is run on numerous arguments. However research often entails more complex arrangements of jobs, where an initial stage will create files which are needed by a later stage or post processing can only be run once a set of analyses have completed. In general there may be arbitrary dependencies between jobs. While it is always possible to manage these manually, waiting for one set of jobs to complete before running the next, it is much more convenient to have a workflow manager handle the dependencies. There are many such systems, suitable for different situations.

  • DAGMan (Directed Acyclic Graph Manager) is HTCondor's native workflow manager. Using it involves creating a simple text file describing the jobs and their connections (or writing a program to generate such a file). We discuss DAGMan in the section on Parallelism.

  • Pegasus is an extremely powerful, but also fairly complex, workflow management system developed by the Information Sciences Institute at the University of Southern California.

  • SnakeMake is a tool for running workflows specified by the relationships between input and output files of individual processes.

  • NextFlow is framework for creating scientific workflows with an emphasis on how data moves between various stages of processing. It is inspired in part by the Unix Philosophy encapsulated in the way data flows between processes connected by pipes.

Need Help?

Additional how-to documentation, such as connecting to clusters and running jobs, is available in Answers.

If you would like to contact us directly for assistance or requesting access, email researchcomputing@syr.edu.

About

Sample code and submit files for OrangeGrid

Resources

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Latest commit

History

104 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

OrangeGrid Examples

This repository provides code examples for commonly used applications within the OrangeGrid cluster. In addition to exploring these examples on the web you can use git to download them into your home directory on the cluster which will allow you to run them directly. The command is

git clone http://github.com/SyracuseUniversity/OrangeGridExamples

Start here

  • hostname: A simple example of submitting a job to the cluster, monitoring it, and checking its output.

Getting the most out of OrangeGrid

These examples discuss general techniques for optimizing performance and throughput

  • Checkpointing: Learn how to save work that your jobs are doing, so if they exit for any reason HTCondor can restart them where they left off.
  • Parallelism: Learn how to divide a task into lots of smaller tasks that can run independently so they can spread out over the cluster.
  • File management: Learn how to arrange your data into chunks that are both more efficient for later analyses and perform better on the cluster.
  • multipleJobs: Learn how to run multiple jobs from one submit file.
  • HTCondorDiagnostics: Check the status of the cluster.

Using particular languages and libraries

  • python: Use Python and Python packages with the Conda package manager.
  • uv: Use Python and Python packages with the uv package manager.
  • tensorflow: Use the Tensorflow package.
  • PyTorch: Use the PyTorch package (note, the uv example also covers this in a way that may be easier to use).
  • Ollama: Run an LLM non-interactively, choosing from among numerous models.
  • julia: Use the Julia language and its libraries.
  • octave: Use Octave, a free and open source math-oriented language that is largely compatibe with Matlab.
  • R: Use the R language and its libraries.
  • Apptainer: Simplify the installation of complex packages by pulling containers from Dockerhub or building new containers.
  • blender: Render images and movies.
  • CVMFS: Use software and data distributed through the CERN Virtual File System
  • Grobid: Use GROBID, a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured documents.
  • CUDA 1.3: Use GPUs with code requiring the latest versions of the CUDA library.
  • JAX: Optimize certain mathematical operations on GPUs.

Workflow managers

Single submit files are great for anything from a single job to thousands of jobs where the same command is run on numerous arguments. However research often entails more complex arrangements of jobs, where an initial stage will create files which are needed by a later stage or post processing can only be run once a set of analyses have completed. In general there may be arbitrary dependencies between jobs. While it is always possible to manage these manually, waiting for one set of jobs to complete before running the next, it is much more convenient to have a workflow manager handle the dependencies. There are many such systems, suitable for different situations.

  • DAGMan (Directed Acyclic Graph Manager) is HTCondor's native workflow manager. Using it involves creating a simple text file describing the jobs and their connections (or writing a program to generate such a file). We discuss DAGMan in the section on Parallelism.

  • Pegasus is an extremely powerful, but also fairly complex, workflow management system developed by the Information Sciences Institute at the University of Southern California.

  • SnakeMake is a tool for running workflows specified by the relationships between input and output files of individual processes.

  • NextFlow is framework for creating scientific workflows with an emphasis on how data moves between various stages of processing. It is inspired in part by the Unix Philosophy encapsulated in the way data flows between processes connected by pipes.

Need Help?

Additional how-to documentation, such as connecting to clusters and running jobs, is available in Answers.

If you would like to contact us directly for assistance or requesting access, email researchcomputing@syr.edu.

About

Sample code and submit files for OrangeGrid

Resources

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Latest commit

History

104 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

OrangeGrid Examples

This repository provides code examples for commonly used applications within the OrangeGrid cluster. In addition to exploring these examples on the web you can use git to download them into your home directory on the cluster which will allow you to run them directly. The command is

git clone http://github.com/SyracuseUniversity/OrangeGridExamples

Start here

  • hostname: A simple example of submitting a job to the cluster, monitoring it, and checking its output.

Getting the most out of OrangeGrid

These examples discuss general techniques for optimizing performance and throughput

  • Checkpointing: Learn how to save work that your jobs are doing, so if they exit for any reason HTCondor can restart them where they left off.
  • Parallelism: Learn how to divide a task into lots of smaller tasks that can run independently so they can spread out over the cluster.
  • File management: Learn how to arrange your data into chunks that are both more efficient for later analyses and perform better on the cluster.
  • multipleJobs: Learn how to run multiple jobs from one submit file.
  • HTCondorDiagnostics: Check the status of the cluster.

Using particular languages and libraries

  • python: Use Python and Python packages with the Conda package manager.
  • uv: Use Python and Python packages with the uv package manager.
  • tensorflow: Use the Tensorflow package.
  • PyTorch: Use the PyTorch package (note, the uv example also covers this in a way that may be easier to use).
  • Ollama: Run an LLM non-interactively, choosing from among numerous models.
  • julia: Use the Julia language and its libraries.
  • octave: Use Octave, a free and open source math-oriented language that is largely compatibe with Matlab.
  • R: Use the R language and its libraries.
  • Apptainer: Simplify the installation of complex packages by pulling containers from Dockerhub or building new containers.
  • blender: Render images and movies.
  • CVMFS: Use software and data distributed through the CERN Virtual File System
  • Grobid: Use GROBID, a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured documents.
  • CUDA 1.3: Use GPUs with code requiring the latest versions of the CUDA library.
  • JAX: Optimize certain mathematical operations on GPUs.

Workflow managers

Single submit files are great for anything from a single job to thousands of jobs where the same command is run on numerous arguments. However research often entails more complex arrangements of jobs, where an initial stage will create files which are needed by a later stage or post processing can only be run once a set of analyses have completed. In general there may be arbitrary dependencies between jobs. While it is always possible to manage these manually, waiting for one set of jobs to complete before running the next, it is much more convenient to have a workflow manager handle the dependencies. There are many such systems, suitable for different situations.

  • DAGMan (Directed Acyclic Graph Manager) is HTCondor's native workflow manager. Using it involves creating a simple text file describing the jobs and their connections (or writing a program to generate such a file). We discuss DAGMan in the section on Parallelism.

  • Pegasus is an extremely powerful, but also fairly complex, workflow management system developed by the Information Sciences Institute at the University of Southern California.

  • SnakeMake is a tool for running workflows specified by the relationships between input and output files of individual processes.

  • NextFlow is framework for creating scientific workflows with an emphasis on how data moves between various stages of processing. It is inspired in part by the Unix Philosophy encapsulated in the way data flows between processes connected by pipes.

Need Help?

Additional how-to documentation, such as connecting to clusters and running jobs, is available in Answers.

If you would like to contact us directly for assistance or requesting access, email researchcomputing@syr.edu.

About

Sample code and submit files for OrangeGrid

Resources

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Latest commit

History

104 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

OrangeGrid Examples

This repository provides code examples for commonly used applications within the OrangeGrid cluster. In addition to exploring these examples on the web you can use git to download them into your home directory on the cluster which will allow you to run them directly. The command is

git clone http://github.com/SyracuseUniversity/OrangeGridExamples

Start here

  • hostname: A simple example of submitting a job to the cluster, monitoring it, and checking its output.

Getting the most out of OrangeGrid

These examples discuss general techniques for optimizing performance and throughput

  • Checkpointing: Learn how to save work that your jobs are doing, so if they exit for any reason HTCondor can restart them where they left off.
  • Parallelism: Learn how to divide a task into lots of smaller tasks that can run independently so they can spread out over the cluster.
  • File management: Learn how to arrange your data into chunks that are both more efficient for later analyses and perform better on the cluster.
  • multipleJobs: Learn how to run multiple jobs from one submit file.
  • HTCondorDiagnostics: Check the status of the cluster.

Using particular languages and libraries

  • python: Use Python and Python packages with the Conda package manager.
  • uv: Use Python and Python packages with the uv package manager.
  • tensorflow: Use the Tensorflow package.
  • PyTorch: Use the PyTorch package (note, the uv example also covers this in a way that may be easier to use).
  • Ollama: Run an LLM non-interactively, choosing from among numerous models.
  • julia: Use the Julia language and its libraries.
  • octave: Use Octave, a free and open source math-oriented language that is largely compatibe with Matlab.
  • R: Use the R language and its libraries.
  • Apptainer: Simplify the installation of complex packages by pulling containers from Dockerhub or building new containers.
  • blender: Render images and movies.
  • CVMFS: Use software and data distributed through the CERN Virtual File System
  • Grobid: Use GROBID, a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured documents.
  • CUDA 1.3: Use GPUs with code requiring the latest versions of the CUDA library.
  • JAX: Optimize certain mathematical operations on GPUs.

Workflow managers

Single submit files are great for anything from a single job to thousands of jobs where the same command is run on numerous arguments. However research often entails more complex arrangements of jobs, where an initial stage will create files which are needed by a later stage or post processing can only be run once a set of analyses have completed. In general there may be arbitrary dependencies between jobs. While it is always possible to manage these manually, waiting for one set of jobs to complete before running the next, it is much more convenient to have a workflow manager handle the dependencies. There are many such systems, suitable for different situations.

  • DAGMan (Directed Acyclic Graph Manager) is HTCondor's native workflow manager. Using it involves creating a simple text file describing the jobs and their connections (or writing a program to generate such a file). We discuss DAGMan in the section on Parallelism.

  • Pegasus is an extremely powerful, but also fairly complex, workflow management system developed by the Information Sciences Institute at the University of Southern California.

  • SnakeMake is a tool for running workflows specified by the relationships between input and output files of individual processes.

  • NextFlow is framework for creating scientific workflows with an emphasis on how data moves between various stages of processing. It is inspired in part by the Unix Philosophy encapsulated in the way data flows between processes connected by pipes.

Need Help?

Additional how-to documentation, such as connecting to clusters and running jobs, is available in Answers.

If you would like to contact us directly for assistance or requesting access, email researchcomputing@syr.edu.

About

Sample code and submit files for OrangeGrid

Resources

Stars

13 stars

Watchers

4 watching

Forks

Releases

Packages

Contributors