BranchesTags

Repository files navigation

apache_spark

Build Status

This cookbook installs and configures Apache Spark.

Overview

This cookbook installs and configures Apache Spark. Currently, only the standalone deployment mode is supported. Future work:

  • YARN and Mesos deployment modes
  • Support installing from Cloudera and HDP Spark packages.

Compatibility

The following platforms are currently tested:

  • Ubuntu 12.04
  • CentOS 6.5

The following platforms are not tested but will probably work (tests coming soon):

  • Fedora 21
  • Ubuntu 14.04

Configuration

  • node['apache_spark']['install_mode']: tarball to install from a downloaded tarball, or package to install from an OS-specific package.
  • node['apache_spark']['download_url']: the URL to download Apache Spark binary distribution tarball in the tarball installation mode.
  • node['apache_spark']['checksum']: SHA256 checksum for the Apache Spark binary distribution tarball.
  • node['apache_spark']['pkg_name']: package name to install in the package installation mode.
  • node['apache_spark']['pkg_version']: package version to install in the package installation mode.
  • node['apache_spark']['install_dir']: target directory to install Spark to in the tarball installation mode. In the package mode, this must be set to the directory that the package installs Spark into.
  • node['apache_spark']['install_base_dir']: in the tarball installation mode, this is where the tarball is actually extracted, and a symlink pointing to the subdirectory containing a specific Spark version is created at node['apache_spark']['install_dir'].
  • node['apache_spark']['user']: UNIX user to create for running Spark.
  • node['apache_spark']['group']: UNIX group to create for running Spark.
  • node['apache_spark']['standalone']['master_host']: Spark standalone-mode workers will connect to this host.
  • node['apache_spark']['standalone']['master_bind_ip']: the IP the master should bind to. This should be set in such a way that workers will be able to connect to the master.
  • node['apache_spark']['standalone']['master_port']: the port for the Spark standalone master to listen on.
  • node['apache_spark']['standalone']['master_webui_port']: Spark standalone master web UI port.
  • node['apache_spark']['standalone']['worker_bind_ip']: the IP address workers bind to. They bind to all network interfaces by default.
  • node['apache_spark']['standalone']['worker_webui_port']: the port for the Spark worker web UI to listen on.
  • node['apache_spark']['standalone']['job_dir_days_retained']: app-... subdirectories of node['apache_spark']['standalone']['worker_work_dir'] older than this number of days will be deleted periodically on worker nodes to prevent unbounded accumulation. These directories contain Spark executor stdout/stderr logs. The directories will still be retained to honor node['apache_spark']['standalone']['job_dir_num_retained'].
  • node['apache_spark']['standalone']['job_dir_num_retained']: the minimum number of Spark executor log directories (app-...) to retain, regardless of creation time.
  • node['apache_spark']['standalone']['worker_dir_cleanup_log']: log file path for the Spark executor log directories cleanup script.
  • node['apache_spark']['standalone']['worker_cores']: the number of "cores" (threads) to allocate on each worker node.
  • node['apache_spark']['standalone']['worker_work_dir']: the directory to store Spark executor logs and Spark job jars.
  • node['apache_spark']['standalone']['worker_memory_mb']: the amount of memory in MB to allocate to each worker (i.e. the maximum total memory used by different applications' executors running on a worker node).
  • node['apache_spark']['standalone']['default_executor_mem_mb']: the default amount of memory to be allocated to a Spark application's executor on each node.
  • node['apache_spark']['standalone']['log_dir']: the log directory for Spark masters and workers.
  • node['apache_spark']['standalone']['daemon_root_logger']: the spark.root.logger property is set to this.
  • node['apache_spark']['standalone']['max_num_open_files']: the maximum number of open files to set using ulimit before launching a worker.
  • node['apache_spark']['standalone']['java_debug_enabled']: whether Java debugging options are to be enabled for Spark processes. Note: currently, this option is not working as intended.
  • node['apache_spark']['standalone']['default_debug_port']: default Java debug port to use. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['master_debug_port']: default Java debug port to use for Spark masters. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['worker_debug_port']: default Java debug port to use for Spark workers. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['executor_debug_port']: default Java debug port to use for Spark standalone executors. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['common_extra_classpath_items']: common classpath items to add to Spark application driver and executors (but not Spark master and worker processes).
  • node['apache_spark']['standalone']['worker_dir']: Set to a non-nil value to tell the spark worker to use an alternate directory for spark scratch space
  • node['apache_spark']['standalone']['worker_opts']: Set to a non-nil value to pass along any additional settings to the spark worker. E.G.: -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400. Ideal for worker options only that you do not want in the default configuration file.
  • node['apache_spark']['conf']['...']: Spark configuration options that go into the default Spark configuration file. See https://spark.apache.org/docs/latest/configuration.html for details.
  • node['apache_spark']['standalone']['local_dirs']: a list of local directories to use on workers. This is where map output files are stored, so these directories should have enough space available.

Testing

ChefSpec

bundle install
bundle exec rspec

Test Kitchen

bundle install
bundle exec kitchen test

Contributing

If you would like to contribute this cookbook's development, please follow the steps below:

  • Fork this repository on GitHub
  • Make your changes
  • Run tests
  • Submit a pull request

License

Apache License 2.0

https://www.apache.org/licenses/LICENSE-2.0

About

A cookbook for installing and configuring Apache Spark

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
BranchesTags
Repository files navigation
apache_spark
Build Status
This cookbook installs and configures Apache Spark.
Overview
This cookbook installs and configures Apache Spark. Currently, only the standalone deployment mode
is supported. Future work:
  • YARN and Mesos deployment modes
  • Support installing from Cloudera and HDP Spark packages.
Compatibility
The following platforms are currently tested:
  • Ubuntu 12.04
  • CentOS 6.5
The following platforms are not tested but will probably work (tests coming soon):
  • Fedora 21
  • Ubuntu 14.04
Configuration
  • node['apache_spark']['install_mode']: tarball to install from a downloaded tarball,
or package to install from an OS-specific package.
  • node['apache_spark']['download_url']: the URL to download Apache Spark binary distribution
tarball in the tarball installation mode.
  • node['apache_spark']['checksum']: SHA256 checksum for the Apache Spark binary distribution
tarball.
  • node['apache_spark']['pkg_name']: package name to install in the package installation mode.
  • node['apache_spark']['pkg_version']: package version to install in the package installation
mode.
  • node['apache_spark']['install_dir']: target directory to install Spark to in the tarball
installation mode. In the package mode, this must be set to the directory that the package
installs Spark into.
  • node['apache_spark']['install_base_dir']: in the tarball installation mode, this is where
the tarball is actually extracted, and a symlink pointing to the subdirectory containing a
specific Spark version is created at node['apache_spark']['install_dir'].
  • node['apache_spark']['user']: UNIX user to create for running Spark.
  • node['apache_spark']['group']: UNIX group to create for running Spark.
  • node['apache_spark']['standalone']['master_host']: Spark standalone-mode workers will connect to
this host.
  • node['apache_spark']['standalone']['master_bind_ip']: the IP the master should bind to. This
should be set in such a way that workers will be able to connect to the master.
  • node['apache_spark']['standalone']['master_port']: the port for the Spark standalone master to
listen on.
  • node['apache_spark']['standalone']['master_webui_port']: Spark standalone master web UI port.
  • node['apache_spark']['standalone']['worker_bind_ip']: the IP address workers bind to.
They bind to all network interfaces by default.
  • node['apache_spark']['standalone']['worker_webui_port']: the port for the Spark worker web UI
to listen on.
  • node['apache_spark']['standalone']['job_dir_days_retained']: app-... subdirectories of
node['apache_spark']['standalone']['worker_work_dir'] older than this number of days will be
deleted periodically on worker nodes to prevent unbounded accumulation. These directories contain
Spark executor stdout/stderr logs. The directories will still be retained to honor
node['apache_spark']['standalone']['job_dir_num_retained'].
  • node['apache_spark']['standalone']['job_dir_num_retained']: the minimum number of Spark
executor log directories (app-...) to retain, regardless of creation time.
  • node['apache_spark']['standalone']['worker_dir_cleanup_log']: log file path for the Spark
executor log directories cleanup script.
  • node['apache_spark']['standalone']['worker_cores']: the number of "cores" (threads) to allocate
on each worker node.
  • node['apache_spark']['standalone']['worker_work_dir']: the directory to store Spark
executor logs and Spark job jars.
  • node['apache_spark']['standalone']['worker_memory_mb']: the amount of memory in MB to allocate
to each worker (i.e. the maximum total memory used by different applications' executors running
on a worker node).
  • node['apache_spark']['standalone']['default_executor_mem_mb']: the default amount of memory
to be allocated to a Spark application's executor on each node.
  • node['apache_spark']['standalone']['log_dir']: the log directory for Spark masters and workers.
  • node['apache_spark']['standalone']['daemon_root_logger']: the spark.root.logger property
is set to this.
  • node['apache_spark']['standalone']['max_num_open_files']: the maximum number of open files to
set using ulimit before launching a worker.
  • node['apache_spark']['standalone']['java_debug_enabled']: whether Java debugging options are
to be enabled for Spark processes. Note: currently, this option is not working as intended.
  • node['apache_spark']['standalone']['default_debug_port']: default Java debug port to use.
A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['master_debug_port']: default Java debug port to use for
Spark masters. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['worker_debug_port']: default Java debug port to use for
Spark workers. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['executor_debug_port']: default Java debug port to use for
Spark standalone executors. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['common_extra_classpath_items']: common classpath items to
add to Spark application driver and executors (but not Spark master and worker processes).
  • node['apache_spark']['standalone']['worker_dir']: Set to a non-nil value to tell the spark worker to use an alternate directory for spark scratch space
  • node['apache_spark']['standalone']['worker_opts']: Set to a non-nil value to pass along any additional settings to the spark worker. E.G.: -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400. Ideal for worker options only that you do not want in the default configuration file.
  • node['apache_spark']['conf']['...']: Spark configuration options that go into the default
Spark configuration file. See https://spark.apache.org/docs/latest/configuration.html for details.
  • node['apache_spark']['standalone']['local_dirs']: a list of local directories to use on workers.
This is where map output files are stored, so these directories should have enough space
available.
Testing
ChefSpec
bundle install
bundle exec rspec

Test Kitchen
bundle install
bundle exec kitchen test

Contributing
If you would like to contribute this cookbook's development, please follow the steps below:
  • Fork this repository on GitHub
  • Make your changes
  • Run tests
  • Submit a pull request
License
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
About
A cookbook for installing and configuring Apache Spark
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
apache_spark
Build Status
This cookbook installs and configures Apache Spark.
Overview
This cookbook installs and configures Apache Spark. Currently, only the standalone deployment mode
is supported. Future work:
  • YARN and Mesos deployment modes
  • Support installing from Cloudera and HDP Spark packages.
Compatibility
The following platforms are currently tested:
  • Ubuntu 12.04
  • CentOS 6.5
The following platforms are not tested but will probably work (tests coming soon):
  • Fedora 21
  • Ubuntu 14.04
Configuration
  • node['apache_spark']['install_mode']: tarball to install from a downloaded tarball,
or package to install from an OS-specific package.
  • node['apache_spark']['download_url']: the URL to download Apache Spark binary distribution
tarball in the tarball installation mode.
  • node['apache_spark']['checksum']: SHA256 checksum for the Apache Spark binary distribution
tarball.
  • node['apache_spark']['pkg_name']: package name to install in the package installation mode.
  • node['apache_spark']['pkg_version']: package version to install in the package installation
mode.
  • node['apache_spark']['install_dir']: target directory to install Spark to in the tarball
installation mode. In the package mode, this must be set to the directory that the package
installs Spark into.
  • node['apache_spark']['install_base_dir']: in the tarball installation mode, this is where
the tarball is actually extracted, and a symlink pointing to the subdirectory containing a
specific Spark version is created at node['apache_spark']['install_dir'].
  • node['apache_spark']['user']: UNIX user to create for running Spark.
  • node['apache_spark']['group']: UNIX group to create for running Spark.
  • node['apache_spark']['standalone']['master_host']: Spark standalone-mode workers will connect to
this host.
  • node['apache_spark']['standalone']['master_bind_ip']: the IP the master should bind to. This
should be set in such a way that workers will be able to connect to the master.
  • node['apache_spark']['standalone']['master_port']: the port for the Spark standalone master to
listen on.
  • node['apache_spark']['standalone']['master_webui_port']: Spark standalone master web UI port.
  • node['apache_spark']['standalone']['worker_bind_ip']: the IP address workers bind to.
They bind to all network interfaces by default.
  • node['apache_spark']['standalone']['worker_webui_port']: the port for the Spark worker web UI
to listen on.
  • node['apache_spark']['standalone']['job_dir_days_retained']: app-... subdirectories of
node['apache_spark']['standalone']['worker_work_dir'] older than this number of days will be
deleted periodically on worker nodes to prevent unbounded accumulation. These directories contain
Spark executor stdout/stderr logs. The directories will still be retained to honor
node['apache_spark']['standalone']['job_dir_num_retained'].
  • node['apache_spark']['standalone']['job_dir_num_retained']: the minimum number of Spark
executor log directories (app-...) to retain, regardless of creation time.
  • node['apache_spark']['standalone']['worker_dir_cleanup_log']: log file path for the Spark
executor log directories cleanup script.
  • node['apache_spark']['standalone']['worker_cores']: the number of "cores" (threads) to allocate
on each worker node.
  • node['apache_spark']['standalone']['worker_work_dir']: the directory to store Spark
executor logs and Spark job jars.
  • node['apache_spark']['standalone']['worker_memory_mb']: the amount of memory in MB to allocate
to each worker (i.e. the maximum total memory used by different applications' executors running
on a worker node).
  • node['apache_spark']['standalone']['default_executor_mem_mb']: the default amount of memory
to be allocated to a Spark application's executor on each node.
  • node['apache_spark']['standalone']['log_dir']: the log directory for Spark masters and workers.
  • node['apache_spark']['standalone']['daemon_root_logger']: the spark.root.logger property
is set to this.
  • node['apache_spark']['standalone']['max_num_open_files']: the maximum number of open files to
set using ulimit before launching a worker.
  • node['apache_spark']['standalone']['java_debug_enabled']: whether Java debugging options are
to be enabled for Spark processes. Note: currently, this option is not working as intended.
  • node['apache_spark']['standalone']['default_debug_port']: default Java debug port to use.
A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['master_debug_port']: default Java debug port to use for
Spark masters. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['worker_debug_port']: default Java debug port to use for
Spark workers. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['executor_debug_port']: default Java debug port to use for
Spark standalone executors. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['common_extra_classpath_items']: common classpath items to
add to Spark application driver and executors (but not Spark master and worker processes).
  • node['apache_spark']['standalone']['worker_dir']: Set to a non-nil value to tell the spark worker to use an alternate directory for spark scratch space
  • node['apache_spark']['standalone']['worker_opts']: Set to a non-nil value to pass along any additional settings to the spark worker. E.G.: -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400. Ideal for worker options only that you do not want in the default configuration file.
  • node['apache_spark']['conf']['...']: Spark configuration options that go into the default
Spark configuration file. See https://spark.apache.org/docs/latest/configuration.html for details.
  • node['apache_spark']['standalone']['local_dirs']: a list of local directories to use on workers.
This is where map output files are stored, so these directories should have enough space
available.
Testing
ChefSpec
bundle install
bundle exec rspec

Test Kitchen
bundle install
bundle exec kitchen test

Contributing
If you would like to contribute this cookbook's development, please follow the steps below:
  • Fork this repository on GitHub
  • Make your changes
  • Run tests
  • Submit a pull request
License
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
About
A cookbook for installing and configuring Apache Spark
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
apache_spark
Build Status
This cookbook installs and configures Apache Spark.
Overview
This cookbook installs and configures Apache Spark. Currently, only the standalone deployment mode
is supported. Future work:
  • YARN and Mesos deployment modes
  • Support installing from Cloudera and HDP Spark packages.
Compatibility
The following platforms are currently tested:
  • Ubuntu 12.04
  • CentOS 6.5
The following platforms are not tested but will probably work (tests coming soon):
  • Fedora 21
  • Ubuntu 14.04
Configuration
  • node['apache_spark']['install_mode']: tarball to install from a downloaded tarball,
or package to install from an OS-specific package.
  • node['apache_spark']['download_url']: the URL to download Apache Spark binary distribution
tarball in the tarball installation mode.
  • node['apache_spark']['checksum']: SHA256 checksum for the Apache Spark binary distribution
tarball.
  • node['apache_spark']['pkg_name']: package name to install in the package installation mode.
  • node['apache_spark']['pkg_version']: package version to install in the package installation
mode.
  • node['apache_spark']['install_dir']: target directory to install Spark to in the tarball
installation mode. In the package mode, this must be set to the directory that the package
installs Spark into.
  • node['apache_spark']['install_base_dir']: in the tarball installation mode, this is where
the tarball is actually extracted, and a symlink pointing to the subdirectory containing a
specific Spark version is created at node['apache_spark']['install_dir'].
  • node['apache_spark']['user']: UNIX user to create for running Spark.
  • node['apache_spark']['group']: UNIX group to create for running Spark.
  • node['apache_spark']['standalone']['master_host']: Spark standalone-mode workers will connect to
this host.
  • node['apache_spark']['standalone']['master_bind_ip']: the IP the master should bind to. This
should be set in such a way that workers will be able to connect to the master.
  • node['apache_spark']['standalone']['master_port']: the port for the Spark standalone master to
listen on.
  • node['apache_spark']['standalone']['master_webui_port']: Spark standalone master web UI port.
  • node['apache_spark']['standalone']['worker_bind_ip']: the IP address workers bind to.
They bind to all network interfaces by default.
  • node['apache_spark']['standalone']['worker_webui_port']: the port for the Spark worker web UI
to listen on.
  • node['apache_spark']['standalone']['job_dir_days_retained']: app-... subdirectories of
node['apache_spark']['standalone']['worker_work_dir'] older than this number of days will be
deleted periodically on worker nodes to prevent unbounded accumulation. These directories contain
Spark executor stdout/stderr logs. The directories will still be retained to honor
node['apache_spark']['standalone']['job_dir_num_retained'].
  • node['apache_spark']['standalone']['job_dir_num_retained']: the minimum number of Spark
executor log directories (app-...) to retain, regardless of creation time.
  • node['apache_spark']['standalone']['worker_dir_cleanup_log']: log file path for the Spark
executor log directories cleanup script.
  • node['apache_spark']['standalone']['worker_cores']: the number of "cores" (threads) to allocate
on each worker node.
  • node['apache_spark']['standalone']['worker_work_dir']: the directory to store Spark
executor logs and Spark job jars.
  • node['apache_spark']['standalone']['worker_memory_mb']: the amount of memory in MB to allocate
to each worker (i.e. the maximum total memory used by different applications' executors running
on a worker node).
  • node['apache_spark']['standalone']['default_executor_mem_mb']: the default amount of memory
to be allocated to a Spark application's executor on each node.
  • node['apache_spark']['standalone']['log_dir']: the log directory for Spark masters and workers.
  • node['apache_spark']['standalone']['daemon_root_logger']: the spark.root.logger property
is set to this.
  • node['apache_spark']['standalone']['max_num_open_files']: the maximum number of open files to
set using ulimit before launching a worker.
  • node['apache_spark']['standalone']['java_debug_enabled']: whether Java debugging options are
to be enabled for Spark processes. Note: currently, this option is not working as intended.
  • node['apache_spark']['standalone']['default_debug_port']: default Java debug port to use.
A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['master_debug_port']: default Java debug port to use for
Spark masters. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['worker_debug_port']: default Java debug port to use for
Spark workers. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['executor_debug_port']: default Java debug port to use for
Spark standalone executors. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['common_extra_classpath_items']: common classpath items to
add to Spark application driver and executors (but not Spark master and worker processes).
  • node['apache_spark']['standalone']['worker_dir']: Set to a non-nil value to tell the spark worker to use an alternate directory for spark scratch space
  • node['apache_spark']['standalone']['worker_opts']: Set to a non-nil value to pass along any additional settings to the spark worker. E.G.: -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400. Ideal for worker options only that you do not want in the default configuration file.
  • node['apache_spark']['conf']['...']: Spark configuration options that go into the default
Spark configuration file. See https://spark.apache.org/docs/latest/configuration.html for details.
  • node['apache_spark']['standalone']['local_dirs']: a list of local directories to use on workers.
This is where map output files are stored, so these directories should have enough space
available.
Testing
ChefSpec
bundle install
bundle exec rspec

Test Kitchen
bundle install
bundle exec kitchen test

Contributing
If you would like to contribute this cookbook's development, please follow the steps below:
  • Fork this repository on GitHub
  • Make your changes
  • Run tests
  • Submit a pull request
License
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
About
A cookbook for installing and configuring Apache Spark
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
BranchesTags
Repository files navigation
apache_spark
Build Status
This cookbook installs and configures Apache Spark.
Overview
This cookbook installs and configures Apache Spark. Currently, only the standalone deployment mode
is supported. Future work:
  • YARN and Mesos deployment modes
  • Support installing from Cloudera and HDP Spark packages.
Compatibility
The following platforms are currently tested:
  • Ubuntu 12.04
  • CentOS 6.5
The following platforms are not tested but will probably work (tests coming soon):
  • Fedora 21
  • Ubuntu 14.04
Configuration
  • node['apache_spark']['install_mode']: tarball to install from a downloaded tarball,
or package to install from an OS-specific package.
  • node['apache_spark']['download_url']: the URL to download Apache Spark binary distribution
tarball in the tarball installation mode.
  • node['apache_spark']['checksum']: SHA256 checksum for the Apache Spark binary distribution
tarball.
  • node['apache_spark']['pkg_name']: package name to install in the package installation mode.
  • node['apache_spark']['pkg_version']: package version to install in the package installation
mode.
  • node['apache_spark']['install_dir']: target directory to install Spark to in the tarball
installation mode. In the package mode, this must be set to the directory that the package
installs Spark into.
  • node['apache_spark']['install_base_dir']: in the tarball installation mode, this is where
the tarball is actually extracted, and a symlink pointing to the subdirectory containing a
specific Spark version is created at node['apache_spark']['install_dir'].
  • node['apache_spark']['user']: UNIX user to create for running Spark.
  • node['apache_spark']['group']: UNIX group to create for running Spark.
  • node['apache_spark']['standalone']['master_host']: Spark standalone-mode workers will connect to
this host.
  • node['apache_spark']['standalone']['master_bind_ip']: the IP the master should bind to. This
should be set in such a way that workers will be able to connect to the master.
  • node['apache_spark']['standalone']['master_port']: the port for the Spark standalone master to
listen on.
  • node['apache_spark']['standalone']['master_webui_port']: Spark standalone master web UI port.
  • node['apache_spark']['standalone']['worker_bind_ip']: the IP address workers bind to.
They bind to all network interfaces by default.
  • node['apache_spark']['standalone']['worker_webui_port']: the port for the Spark worker web UI
to listen on.
  • node['apache_spark']['standalone']['job_dir_days_retained']: app-... subdirectories of
node['apache_spark']['standalone']['worker_work_dir'] older than this number of days will be
deleted periodically on worker nodes to prevent unbounded accumulation. These directories contain
Spark executor stdout/stderr logs. The directories will still be retained to honor
node['apache_spark']['standalone']['job_dir_num_retained'].
  • node['apache_spark']['standalone']['job_dir_num_retained']: the minimum number of Spark
executor log directories (app-...) to retain, regardless of creation time.
  • node['apache_spark']['standalone']['worker_dir_cleanup_log']: log file path for the Spark
executor log directories cleanup script.
  • node['apache_spark']['standalone']['worker_cores']: the number of "cores" (threads) to allocate
on each worker node.
  • node['apache_spark']['standalone']['worker_work_dir']: the directory to store Spark
executor logs and Spark job jars.
  • node['apache_spark']['standalone']['worker_memory_mb']: the amount of memory in MB to allocate
to each worker (i.e. the maximum total memory used by different applications' executors running
on a worker node).
  • node['apache_spark']['standalone']['default_executor_mem_mb']: the default amount of memory
to be allocated to a Spark application's executor on each node.
  • node['apache_spark']['standalone']['log_dir']: the log directory for Spark masters and workers.
  • node['apache_spark']['standalone']['daemon_root_logger']: the spark.root.logger property
is set to this.
  • node['apache_spark']['standalone']['max_num_open_files']: the maximum number of open files to
set using ulimit before launching a worker.
  • node['apache_spark']['standalone']['java_debug_enabled']: whether Java debugging options are
to be enabled for Spark processes. Note: currently, this option is not working as intended.
  • node['apache_spark']['standalone']['default_debug_port']: default Java debug port to use.
A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['master_debug_port']: default Java debug port to use for
Spark masters. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['worker_debug_port']: default Java debug port to use for
Spark workers. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['executor_debug_port']: default Java debug port to use for
Spark standalone executors. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['common_extra_classpath_items']: common classpath items to
add to Spark application driver and executors (but not Spark master and worker processes).
  • node['apache_spark']['standalone']['worker_dir']: Set to a non-nil value to tell the spark worker to use an alternate directory for spark scratch space
  • node['apache_spark']['standalone']['worker_opts']: Set to a non-nil value to pass along any additional settings to the spark worker. E.G.: -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400. Ideal for worker options only that you do not want in the default configuration file.
  • node['apache_spark']['conf']['...']: Spark configuration options that go into the default
Spark configuration file. See https://spark.apache.org/docs/latest/configuration.html for details.
  • node['apache_spark']['standalone']['local_dirs']: a list of local directories to use on workers.
This is where map output files are stored, so these directories should have enough space
available.
Testing
ChefSpec
bundle install
bundle exec rspec

Test Kitchen
bundle install
bundle exec kitchen test

Contributing
If you would like to contribute this cookbook's development, please follow the steps below:
  • Fork this repository on GitHub
  • Make your changes
  • Run tests
  • Submit a pull request
License
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
About
A cookbook for installing and configuring Apache Spark
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
apache_spark
Build Status
This cookbook installs and configures Apache Spark.
Overview
This cookbook installs and configures Apache Spark. Currently, only the standalone deployment mode
is supported. Future work:
  • YARN and Mesos deployment modes
  • Support installing from Cloudera and HDP Spark packages.
Compatibility
The following platforms are currently tested:
  • Ubuntu 12.04
  • CentOS 6.5
The following platforms are not tested but will probably work (tests coming soon):
  • Fedora 21
  • Ubuntu 14.04
Configuration
  • node['apache_spark']['install_mode']: tarball to install from a downloaded tarball,
or package to install from an OS-specific package.
  • node['apache_spark']['download_url']: the URL to download Apache Spark binary distribution
tarball in the tarball installation mode.
  • node['apache_spark']['checksum']: SHA256 checksum for the Apache Spark binary distribution
tarball.
  • node['apache_spark']['pkg_name']: package name to install in the package installation mode.
  • node['apache_spark']['pkg_version']: package version to install in the package installation
mode.
  • node['apache_spark']['install_dir']: target directory to install Spark to in the tarball
installation mode. In the package mode, this must be set to the directory that the package
installs Spark into.
  • node['apache_spark']['install_base_dir']: in the tarball installation mode, this is where
the tarball is actually extracted, and a symlink pointing to the subdirectory containing a
specific Spark version is created at node['apache_spark']['install_dir'].
  • node['apache_spark']['user']: UNIX user to create for running Spark.
  • node['apache_spark']['group']: UNIX group to create for running Spark.
  • node['apache_spark']['standalone']['master_host']: Spark standalone-mode workers will connect to
this host.
  • node['apache_spark']['standalone']['master_bind_ip']: the IP the master should bind to. This
should be set in such a way that workers will be able to connect to the master.
  • node['apache_spark']['standalone']['master_port']: the port for the Spark standalone master to
listen on.
  • node['apache_spark']['standalone']['master_webui_port']: Spark standalone master web UI port.
  • node['apache_spark']['standalone']['worker_bind_ip']: the IP address workers bind to.
They bind to all network interfaces by default.
  • node['apache_spark']['standalone']['worker_webui_port']: the port for the Spark worker web UI
to listen on.
  • node['apache_spark']['standalone']['job_dir_days_retained']: app-... subdirectories of
node['apache_spark']['standalone']['worker_work_dir'] older than this number of days will be
deleted periodically on worker nodes to prevent unbounded accumulation. These directories contain
Spark executor stdout/stderr logs. The directories will still be retained to honor
node['apache_spark']['standalone']['job_dir_num_retained'].
  • node['apache_spark']['standalone']['job_dir_num_retained']: the minimum number of Spark
executor log directories (app-...) to retain, regardless of creation time.
  • node['apache_spark']['standalone']['worker_dir_cleanup_log']: log file path for the Spark
executor log directories cleanup script.
  • node['apache_spark']['standalone']['worker_cores']: the number of "cores" (threads) to allocate
on each worker node.
  • node['apache_spark']['standalone']['worker_work_dir']: the directory to store Spark
executor logs and Spark job jars.
  • node['apache_spark']['standalone']['worker_memory_mb']: the amount of memory in MB to allocate
to each worker (i.e. the maximum total memory used by different applications' executors running
on a worker node).
  • node['apache_spark']['standalone']['default_executor_mem_mb']: the default amount of memory
to be allocated to a Spark application's executor on each node.
  • node['apache_spark']['standalone']['log_dir']: the log directory for Spark masters and workers.
  • node['apache_spark']['standalone']['daemon_root_logger']: the spark.root.logger property
is set to this.
  • node['apache_spark']['standalone']['max_num_open_files']: the maximum number of open files to
set using ulimit before launching a worker.
  • node['apache_spark']['standalone']['java_debug_enabled']: whether Java debugging options are
to be enabled for Spark processes. Note: currently, this option is not working as intended.
  • node['apache_spark']['standalone']['default_debug_port']: default Java debug port to use.
A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['master_debug_port']: default Java debug port to use for
Spark masters. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['worker_debug_port']: default Java debug port to use for
Spark workers. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['executor_debug_port']: default Java debug port to use for
Spark standalone executors. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['common_extra_classpath_items']: common classpath items to
add to Spark application driver and executors (but not Spark master and worker processes).
  • node['apache_spark']['standalone']['worker_dir']: Set to a non-nil value to tell the spark worker to use an alternate directory for spark scratch space
  • node['apache_spark']['standalone']['worker_opts']: Set to a non-nil value to pass along any additional settings to the spark worker. E.G.: -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400. Ideal for worker options only that you do not want in the default configuration file.
  • node['apache_spark']['conf']['...']: Spark configuration options that go into the default
Spark configuration file. See https://spark.apache.org/docs/latest/configuration.html for details.
  • node['apache_spark']['standalone']['local_dirs']: a list of local directories to use on workers.
This is where map output files are stored, so these directories should have enough space
available.
Testing
ChefSpec
bundle install
bundle exec rspec

Test Kitchen
bundle install
bundle exec kitchen test

Contributing
If you would like to contribute this cookbook's development, please follow the steps below:
  • Fork this repository on GitHub
  • Make your changes
  • Run tests
  • Submit a pull request
License
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
About
A cookbook for installing and configuring Apache Spark
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
BranchesTags
Repository files navigation
apache_spark
Build Status
This cookbook installs and configures Apache Spark.
Overview
This cookbook installs and configures Apache Spark. Currently, only the standalone deployment mode
is supported. Future work:
  • YARN and Mesos deployment modes
  • Support installing from Cloudera and HDP Spark packages.
Compatibility
The following platforms are currently tested:
  • Ubuntu 12.04
  • CentOS 6.5
The following platforms are not tested but will probably work (tests coming soon):
  • Fedora 21
  • Ubuntu 14.04
Configuration
  • node['apache_spark']['install_mode']: tarball to install from a downloaded tarball,
or package to install from an OS-specific package.
  • node['apache_spark']['download_url']: the URL to download Apache Spark binary distribution
tarball in the tarball installation mode.
  • node['apache_spark']['checksum']: SHA256 checksum for the Apache Spark binary distribution
tarball.
  • node['apache_spark']['pkg_name']: package name to install in the package installation mode.
  • node['apache_spark']['pkg_version']: package version to install in the package installation
mode.
  • node['apache_spark']['install_dir']: target directory to install Spark to in the tarball
installation mode. In the package mode, this must be set to the directory that the package
installs Spark into.
  • node['apache_spark']['install_base_dir']: in the tarball installation mode, this is where
the tarball is actually extracted, and a symlink pointing to the subdirectory containing a
specific Spark version is created at node['apache_spark']['install_dir'].
  • node['apache_spark']['user']: UNIX user to create for running Spark.
  • node['apache_spark']['group']: UNIX group to create for running Spark.
  • node['apache_spark']['standalone']['master_host']: Spark standalone-mode workers will connect to
this host.
  • node['apache_spark']['standalone']['master_bind_ip']: the IP the master should bind to. This
should be set in such a way that workers will be able to connect to the master.
  • node['apache_spark']['standalone']['master_port']: the port for the Spark standalone master to
listen on.
  • node['apache_spark']['standalone']['master_webui_port']: Spark standalone master web UI port.
  • node['apache_spark']['standalone']['worker_bind_ip']: the IP address workers bind to.
They bind to all network interfaces by default.
  • node['apache_spark']['standalone']['worker_webui_port']: the port for the Spark worker web UI
to listen on.
  • node['apache_spark']['standalone']['job_dir_days_retained']: app-... subdirectories of
node['apache_spark']['standalone']['worker_work_dir'] older than this number of days will be
deleted periodically on worker nodes to prevent unbounded accumulation. These directories contain
Spark executor stdout/stderr logs. The directories will still be retained to honor
node['apache_spark']['standalone']['job_dir_num_retained'].
  • node['apache_spark']['standalone']['job_dir_num_retained']: the minimum number of Spark
executor log directories (app-...) to retain, regardless of creation time.
  • node['apache_spark']['standalone']['worker_dir_cleanup_log']: log file path for the Spark
executor log directories cleanup script.
  • node['apache_spark']['standalone']['worker_cores']: the number of "cores" (threads) to allocate
on each worker node.
  • node['apache_spark']['standalone']['worker_work_dir']: the directory to store Spark
executor logs and Spark job jars.
  • node['apache_spark']['standalone']['worker_memory_mb']: the amount of memory in MB to allocate
to each worker (i.e. the maximum total memory used by different applications' executors running
on a worker node).
  • node['apache_spark']['standalone']['default_executor_mem_mb']: the default amount of memory
to be allocated to a Spark application's executor on each node.
  • node['apache_spark']['standalone']['log_dir']: the log directory for Spark masters and workers.
  • node['apache_spark']['standalone']['daemon_root_logger']: the spark.root.logger property
is set to this.
  • node['apache_spark']['standalone']['max_num_open_files']: the maximum number of open files to
set using ulimit before launching a worker.
  • node['apache_spark']['standalone']['java_debug_enabled']: whether Java debugging options are
to be enabled for Spark processes. Note: currently, this option is not working as intended.
  • node['apache_spark']['standalone']['default_debug_port']: default Java debug port to use.
A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['master_debug_port']: default Java debug port to use for
Spark masters. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['worker_debug_port']: default Java debug port to use for
Spark workers. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['executor_debug_port']: default Java debug port to use for
Spark standalone executors. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['common_extra_classpath_items']: common classpath items to
add to Spark application driver and executors (but not Spark master and worker processes).
  • node['apache_spark']['standalone']['worker_dir']: Set to a non-nil value to tell the spark worker to use an alternate directory for spark scratch space
  • node['apache_spark']['standalone']['worker_opts']: Set to a non-nil value to pass along any additional settings to the spark worker. E.G.: -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400. Ideal for worker options only that you do not want in the default configuration file.
  • node['apache_spark']['conf']['...']: Spark configuration options that go into the default
Spark configuration file. See https://spark.apache.org/docs/latest/configuration.html for details.
  • node['apache_spark']['standalone']['local_dirs']: a list of local directories to use on workers.
This is where map output files are stored, so these directories should have enough space
available.
Testing
ChefSpec
bundle install
bundle exec rspec

Test Kitchen
bundle install
bundle exec kitchen test

Contributing
If you would like to contribute this cookbook's development, please follow the steps below:
  • Fork this repository on GitHub
  • Make your changes
  • Run tests
  • Submit a pull request
License
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
About
A cookbook for installing and configuring Apache Spark
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
BranchesTags
Repository files navigation
apache_spark
Build Status
This cookbook installs and configures Apache Spark.
Overview
This cookbook installs and configures Apache Spark. Currently, only the standalone deployment mode
is supported. Future work:
  • YARN and Mesos deployment modes
  • Support installing from Cloudera and HDP Spark packages.
Compatibility
The following platforms are currently tested:
  • Ubuntu 12.04
  • CentOS 6.5
The following platforms are not tested but will probably work (tests coming soon):
  • Fedora 21
  • Ubuntu 14.04
Configuration
  • node['apache_spark']['install_mode']: tarball to install from a downloaded tarball,
or package to install from an OS-specific package.
  • node['apache_spark']['download_url']: the URL to download Apache Spark binary distribution
tarball in the tarball installation mode.
  • node['apache_spark']['checksum']: SHA256 checksum for the Apache Spark binary distribution
tarball.
  • node['apache_spark']['pkg_name']: package name to install in the package installation mode.
  • node['apache_spark']['pkg_version']: package version to install in the package installation
mode.
  • node['apache_spark']['install_dir']: target directory to install Spark to in the tarball
installation mode. In the package mode, this must be set to the directory that the package
installs Spark into.
  • node['apache_spark']['install_base_dir']: in the tarball installation mode, this is where
the tarball is actually extracted, and a symlink pointing to the subdirectory containing a
specific Spark version is created at node['apache_spark']['install_dir'].
  • node['apache_spark']['user']: UNIX user to create for running Spark.
  • node['apache_spark']['group']: UNIX group to create for running Spark.
  • node['apache_spark']['standalone']['master_host']: Spark standalone-mode workers will connect to
this host.
  • node['apache_spark']['standalone']['master_bind_ip']: the IP the master should bind to. This
should be set in such a way that workers will be able to connect to the master.
  • node['apache_spark']['standalone']['master_port']: the port for the Spark standalone master to
listen on.
  • node['apache_spark']['standalone']['master_webui_port']: Spark standalone master web UI port.
  • node['apache_spark']['standalone']['worker_bind_ip']: the IP address workers bind to.
They bind to all network interfaces by default.
  • node['apache_spark']['standalone']['worker_webui_port']: the port for the Spark worker web UI
to listen on.
  • node['apache_spark']['standalone']['job_dir_days_retained']: app-... subdirectories of
node['apache_spark']['standalone']['worker_work_dir'] older than this number of days will be
deleted periodically on worker nodes to prevent unbounded accumulation. These directories contain
Spark executor stdout/stderr logs. The directories will still be retained to honor
node['apache_spark']['standalone']['job_dir_num_retained'].
  • node['apache_spark']['standalone']['job_dir_num_retained']: the minimum number of Spark
executor log directories (app-...) to retain, regardless of creation time.
  • node['apache_spark']['standalone']['worker_dir_cleanup_log']: log file path for the Spark
executor log directories cleanup script.
  • node['apache_spark']['standalone']['worker_cores']: the number of "cores" (threads) to allocate
on each worker node.
  • node['apache_spark']['standalone']['worker_work_dir']: the directory to store Spark
executor logs and Spark job jars.
  • node['apache_spark']['standalone']['worker_memory_mb']: the amount of memory in MB to allocate
to each worker (i.e. the maximum total memory used by different applications' executors running
on a worker node).
  • node['apache_spark']['standalone']['default_executor_mem_mb']: the default amount of memory
to be allocated to a Spark application's executor on each node.
  • node['apache_spark']['standalone']['log_dir']: the log directory for Spark masters and workers.
  • node['apache_spark']['standalone']['daemon_root_logger']: the spark.root.logger property
is set to this.
  • node['apache_spark']['standalone']['max_num_open_files']: the maximum number of open files to
set using ulimit before launching a worker.
  • node['apache_spark']['standalone']['java_debug_enabled']: whether Java debugging options are
to be enabled for Spark processes. Note: currently, this option is not working as intended.
  • node['apache_spark']['standalone']['default_debug_port']: default Java debug port to use.
A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['master_debug_port']: default Java debug port to use for
Spark masters. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['worker_debug_port']: default Java debug port to use for
Spark workers. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['executor_debug_port']: default Java debug port to use for
Spark standalone executors. A free port is chosen if this port is unavailable.
  • node['apache_spark']['standalone']['common_extra_classpath_items']: common classpath items to
add to Spark application driver and executors (but not Spark master and worker processes).
  • node['apache_spark']['standalone']['worker_dir']: Set to a non-nil value to tell the spark worker to use an alternate directory for spark scratch space
  • node['apache_spark']['standalone']['worker_opts']: Set to a non-nil value to pass along any additional settings to the spark worker. E.G.: -Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=86400. Ideal for worker options only that you do not want in the default configuration file.
  • node['apache_spark']['conf']['...']: Spark configuration options that go into the default
Spark configuration file. See https://spark.apache.org/docs/latest/configuration.html for details.
  • node['apache_spark']['standalone']['local_dirs']: a list of local directories to use on workers.
This is where map output files are stored, so these directories should have enough space
available.
Testing
ChefSpec
bundle install
bundle exec rspec

Test Kitchen
bundle install
bundle exec kitchen test

Contributing
If you would like to contribute this cookbook's development, please follow the steps below:
  • Fork this repository on GitHub
  • Make your changes
  • Run tests
  • Submit a pull request
License
Apache License 2.0
https://www.apache.org/licenses/LICENSE-2.0
About
A cookbook for installing and configuring Apache Spark
Resources
Readme
Activity
Stars
0 stars
Watchers
0 watching
Forks
0 forks
Report repository
Releases
Packages
Contributors
Languages