| Property Name | Default | Meaning | ||
|---|---|---|---|---|
| spark.executor.memory | -512m | +spark.cores.max | +(not set) |
- Amount of memory to use per executor process, in the same format as JVM memory strings (e.g. 512m, 2g).
+ When running on a standalone deploy cluster or a
+ Mesos cluster in "coarse-grained"
+ sharing mode, the maximum amount of CPU cores to request for the application from
+ across the cluster (not from each machine). If not set, the default will be
+ spark.deploy.defaultCores on Spark's standalone cluster manager, or
+ infinite (all available cores) on Mesos.
|
| spark.serializer | -org.apache.spark.serializer. JavaSerializer |
+ spark.eventLog.compress | +false |
- Class to use for serializing objects that will be sent over the network or need to be cached
- in serialized form. The default of Java serialization works with any Serializable Java object but is
- quite slow, so we recommend using org.apache.spark.serializer.KryoSerializer
- and configuring Kryo serialization when speed is necessary. Can be any subclass of
- org.apache.spark.Serializer.
+ Whether to compress logged events, if spark.eventLog.enabled is true.
+ |
+
| spark.eventLog.enabled | +false | ++ Whether to log spark events, useful for reconstructing the Web UI after the application has finished. + | +||
| spark.executor.memory | +512m | +
+ Amount of memory to use per executor process, in the same format as JVM memory strings (e.g. 512m, 2g).
|
||
| spark.cores.max | -(not set) | +spark.serializer | +org.apache.spark.serializer. JavaSerializer |
- When running on a standalone deploy cluster or a
- Mesos cluster in "coarse-grained"
- sharing mode, the maximum amount of CPU cores to request for the application from
- across the cluster (not from each machine). If not set, the default will be
- spark.deploy.defaultCores on Spark's standalone cluster manager, or
- infinite (all available cores) on Mesos.
+ Class to use for serializing objects that will be sent over the network or need to be cached
+ in serialized form. The default of Java serialization works with any Serializable Java object but is
+ quite slow, so we recommend using org.apache.spark.serializer.KryoSerializer
+ and configuring Kryo serialization when speed is necessary. Can be any subclass of
+ org.apache.spark.Serializer.
|
| Property Name | Default | Meaning | ||
|---|---|---|---|---|
| spark.default.parallelism | -8 | +spark.akka.askTimeout | +30 |
- Default number of tasks to use across the cluster for distributed shuffle operations (groupByKey,
- reduceByKey, etc) when not set by user.
+ This parameter defines the timeout to use for Akka ask operations. (in seconds)
|
| spark.storage.memoryFraction | -0.6 | +spark.akka.lookupTimeout | +30 | - Fraction of Java heap to use for Spark's memory cache. This should not be larger than the "old" - generation of objects in the JVM, which by default is given 0.6 of the heap, but you can increase - it if you configure your own old generation size. + This parameter defines the timeout to use for Akka lookup operations. (in seconds) |
| spark.shuffle.memoryFraction | -0.3 | +spark.akka.batchSize | +15 |
- Fraction of Java heap to use for aggregation and cogroups during shuffles, if
- spark.shuffle.spill is true. At any given time, the collective size of
- all in-memory maps used for shuffles is bounded by this limit, beyond which the contents will
- begin to spill to disk. If spills are often, consider increasing this value at the expense of
- spark.storage.memoryFraction.
+ The parameter specifies defines the maximum number of messages to be processed per actor
+ before the thread jumps to the next actor. Set to 1 for as fair as possible.
|
| spark.mesos.coarse | +spark.akka.frameSize | +10 | +
+ Maximum message size to allow in "control plane" communication (for serialized tasks and task
+ results), in MB. Increase this if your tasks need to send back large results to the driver
+ (e.g. using collect() on a large dataset).
+ |
+|
| spark.akka.logAkkaConfig | false | - If set to "true", runs over Mesos clusters in - "coarse-grained" sharing mode, - where Spark acquires one long-lived Mesos task on each machine instead of one Mesos task per Spark task. - This gives lower-latency scheduling for short queries, but leaves resources in use for the whole - duration of the Spark job. + This parameter controls three logging behaviours of Akka: 1) Logging the events including: DisassociatedEvent, + AssociatedEvent, AssociationErrorEvent, RemotingListenEvent, RemotingShutdownEvent, RemotingErrorEvent; 2) Logging + the dead letters during running; 3) Logging the dead letters during the shutdown. | ||
| spark.ui.port | -4040 | +spark.akka.failure-detector.threshold | +300.0 | - Port for your application's dashboard, which shows memory and workload data + This is set to a larger value to disable failure detector that comes inbuilt akka. It can be enabled again, if you plan to use this feature (Not recommended). This maps to akka's `akka.remote.transport-failure-detector.threshold`. Tune this in combination of `spark.akka.heartbeat.pauses` and `spark.akka.heartbeat.interval` if you need to. |
| spark.ui.retainedStages | +spark.akka.heartbeat.interval | 1000 | - How many stages the Spark UI remembers before garbage collecting. + This is set to a larger value to disable failure detector that comes inbuilt akka. It can be enabled again, if you plan to use this feature (Not recommended). A larger interval value in seconds reduces network overhead and a smaller value ( ~ 1 s) might be more informative for akka's failure detector. Tune this in combination of `spark.akka.heartbeat.pauses` and `spark.akka.failure-detector.threshold` if you need to. Only positive use case for using failure detector can be, a sensistive failure detector can help evict rogue executors really quick. However this is usually not the case as gc pauses and network lags are expected in a real spark cluster. Apart from that enabling this leads to a lot of exchanges of heart beats between nodes leading to flooding the network with those. | |
| spark.shuffle.compress | -true | +spark.akka.heartbeat.pauses | +600 | - Whether to compress map output files. Generally a good idea. + This is set to a larger value to disable failure detector that comes inbuilt akka. It can be enabled again, if you plan to use this feature (Not recommended). Acceptable heart beat pause in seconds for akka. This can be used to control sensitivity to gc pauses. Tune this in combination of `spark.akka.heartbeat.interval` and `spark.akka.failure-detector.threshold` if you need to. |
| spark.shuffle.spill.compress | -true | +spark.akka.timeout | +100 | - Whether to compress data spilled during shuffles. + Communication timeout between Spark nodes, in seconds. |
| spark.broadcast.compress | -true | +spark.akka.threads | +4 | - Whether to compress broadcast variables before sending them. Generally a good idea. + Number of actor threads to use for communication. Can be useful to increase on large clusters + when the driver has a lot of CPU cores. |
| spark.rdd.compress | +spark.authenticate | false |
- Whether to compress serialized RDD partitions (e.g. for StorageLevel.MEMORY_ONLY_SER).
- Can save substantial space at the cost of some extra CPU time.
+ Whether spark authenticates its internal connections. See spark.authenticate.secret if not
+ running on Yarn.
|
|
| spark.io.compression.codec | -org.apache.spark.io. LZFCompressionCodec |
+ spark.authenticate.secret | +None |
- The codec used to compress internal data such as RDD partitions and shuffle outputs. By default, Spark provides two
- codecs: org.apache.spark.io.LZFCompressionCodec and org.apache.spark.io.SnappyCompressionCodec.
+ Set the secret key used for Spark to authenticate between components. This needs to be set if
+ not running on Yarn and authentication is enabled.
|
| spark.io.compression.snappy.block.size | -32768 | +spark.broadcast.blockSize | +4096 |
- Block size (in bytes) used in Snappy compression, in the case when Snappy compression codec is used.
+ Size of each piece of a block in kilobytes for TorrentBroadcastFactory.
+ Too large a value decreases parallelism during broadcast (makes it slower); however, if it is too small, BlockManager might take a performance hit.
|
| spark.scheduler.mode | -FIFO | +spark.broadcast.compress | +true |
- The scheduling mode between
- jobs submitted to the same SparkContext. Can be set to FAIR
- to use fair sharing instead of queueing jobs one after another. Useful for
- multi-user services.
+ Whether to compress broadcast variables before sending them. Generally a good idea.
|
| spark.scheduler.revive.interval | -1000 | +spark.broadcast.factory | +org.apache.spark.broadcast. HttpBroadcastFactory |
- The interval length for the scheduler to revive the worker resource offers to run tasks. (in milliseconds) + Which broadcast implementation to use. |
| spark.reducer.maxMbInFlight | -48 | +spark.cleaner.ttl | +(infinite) | - Maximum size (in megabytes) of map outputs to fetch simultaneously from each reduce task. Since - each output requires us to create a buffer to receive it, this represents a fixed memory overhead - per reduce task, so keep it small unless you have a large amount of memory. + Duration (seconds) of how long Spark will remember any metadata (stages generated, tasks generated, etc.). + Periodic cleanups will ensure that metadata older than this duration will be forgetten. This is + useful for running Spark for many hours / days (for example, running 24/7 in case of Spark Streaming + applications). Note that any RDD that persists in memory for more than this duration will be cleared as well. |
| spark.kryo.referenceTracking | -true | +spark.core.connection.auth.wait.timeout | +30 | - Whether to track references to the same object when serializing data with Kryo, which is - necessary if your object graphs have loops and useful for efficiency if they contain multiple - copies of the same object. Can be disabled to improve performance if you know this is not the - case. + Number of seconds for the connection to wait for authentication to occur before timing + out and giving up. |
| spark.kryoserializer.buffer.mb | -2 | +spark.core.connection.handler.threads.min | +20 | - Maximum object size to allow within Kryo (the library needs to create a buffer at least as - large as the largest single object you'll serialize). Increase this if you get a "buffer limit - exceeded" exception inside Kryo. Note that there will be one buffer per core on each worker. + The number of threads to keep in the thread pool handling acceptance of the new connection, even they are + idle. |
| spark.broadcast.factory | -org.apache.spark.broadcast. HttpBroadcastFactory |
+ spark.core.connection.handler.threads.max | +60 | - Which broadcast implementation to use. + The maximum number of threads to allow in the pool handling acceptance of the new connection. |
| spark.locality.wait | -3000 | +spark.core.connection.handler.threads.keepalive | +60 |
- Number of milliseconds to wait to launch a data-local task before giving up and launching it
- on a less-local node. The same wait will be used to step through multiple locality levels
- (process-local, node-local, rack-local and then any). It is also possible to customize the
- waiting time for each level by setting spark.locality.wait.node, etc.
- You should increase this setting if your tasks are long and see poor locality, but the
- default usually works well.
+ When the number of threads in the pool handling acceptance of the new connection is greater than the core,
+ this is the maximum time that excess idle threads will wait for new tasks before terminating.
|
| spark.locality.wait.process | -spark.locality.wait | +spark.core.connection.io.threads.min | +20 | - Customize the locality wait for process locality. This affects tasks that attempt to access - cached data in a particular executor process. + The number of threads to keep in the thread pool handling BufferMessage receive/send, even they are + idle. |
| spark.locality.wait.node | -spark.locality.wait | +spark.core.connection.io.threads.max | +60 | - Customize the locality wait for node locality. For example, you can set this to 0 to skip - node locality and search immediately for rack locality (if your cluster has rack information). + The maximum number of threads to allow in the pool handling BufferMessage receive/send. |
| spark.locality.wait.rack | -spark.locality.wait | +spark.core.connection.io.threads.keepalive | +60 | - Customize the locality wait for rack locality. + When the number of threads in the pool handling BufferMessage receive/send is greater than the core, + this is the maximum time that excess idle threads will wait for new tasks before terminating. |
| spark.worker.timeout | +spark.core.connection.connect.threads.min | +20 | ++ The number of threads to keep in the thread pool handling the establishment of the connection, even they are + idle. + | +|
| spark.core.connection.connect.threads.max | 60 | - Number of seconds after which the standalone deploy master considers a worker lost if it - receives no heartbeats. + The maximum number of threads to allow in the pool handling the establishment of the connection. | ||
| spark.akka.frameSize | -10 | +spark.core.connection.connect.threads.keepalive | +60 |
- Maximum message size to allow in "control plane" communication (for serialized tasks and task
- results), in MB. Increase this if your tasks need to send back large results to the driver
- (e.g. using collect() on a large dataset).
+ When the number of threads in the pool handling the establishment of the connection is greater than the core,
+ this is the maximum time that excess idle threads will wait for new tasks before terminating.
|
| spark.akka.threads | -4 | +spark.dead.worker.persistence | +15 |
- Number of actor threads to use for communication. Can be useful to increase on large clusters
- when the driver has a lot of CPU cores.
+ This parameter controls when Spark will remove the worker information from the UI if it is dead.
+ If it has been longer than (spark.dead.worker.persistence + 1) * spark.worker.timeout
+ seconds since the worker is dead, Spark will remove it from UI.
|
| spark.akka.timeout | -100 | +spark.default.parallelism | +8 |
- Communication timeout between Spark nodes, in seconds.
+ Default number of tasks to use across the cluster for distributed shuffle operations (groupByKey,
+ reduceByKey, etc) when not set by user.
|
| spark.akka.heartbeat.pauses | -600 | +spark.deploy.defaultCores | +(infinite) |
- This is set to a larger value to disable failure detector that comes inbuilt akka. It can be enabled again, if you plan to use this feature (Not recommended). Acceptable heart beat pause in seconds for akka. This can be used to control sensitivity to gc pauses. Tune this in combination of `spark.akka.heartbeat.interval` and `spark.akka.failure-detector.threshold` if you need to.
+ Default number of cores to give to applications in Spark's standalone mode if they don't
+ set spark.cores.max. If not set, applications always get all available
+ cores unless they configure spark.cores.max themselves.
+ Set this lower on a shared cluster to prevent users from grabbing
+ the whole cluster by default. + Note: this setting needs to be configured in the standalone cluster master, not in individual + applications; you can set it through SPARK_JAVA_OPTS in spark-env.sh.
|
| spark.akka.failure-detector.threshold | -300.0 | +spark.deploy.retainedApplications | +200 | - This is set to a larger value to disable failure detector that comes inbuilt akka. It can be enabled again, if you plan to use this feature (Not recommended). This maps to akka's `akka.remote.transport-failure-detector.threshold`. Tune this in combination of `spark.akka.heartbeat.pauses` and `spark.akka.heartbeat.interval` if you need to. + The maximum number of records on completed applications kept in the Master node. When the completed + application number exceeds the threshold, the first Max(spark.deploy.retainedApplications, 1) will be + discarded. |
| spark.akka.heartbeat.interval | -1000 | +spark.deploy.spreadOut | +true | +
+ Whether the standalone cluster manager should spread applications out across nodes or try
+ to consolidate them onto as few nodes as possible. Spreading out is usually better for
+ data locality in HDFS, but consolidating is more efficient for compute-intensive workloads. + Note: this setting needs to be configured in the standalone cluster master, not in individual + applications; you can set it through SPARK_JAVA_OPTS in spark-env.sh.
+ |
+
| spark.diskStore.subDirectories | +64 | - This is set to a larger value to disable failure detector that comes inbuilt akka. It can be enabled again, if you plan to use this feature (Not recommended). A larger interval value in seconds reduces network overhead and a smaller value ( ~ 1 s) might be more informative for akka's failure detector. Tune this in combination of `spark.akka.heartbeat.pauses` and `spark.akka.failure-detector.threshold` if you need to. Only positive use case for using failure detector can be, a sensistive failure detector can help evict rogue executors really quick. However this is usually not the case as gc pauses and network lags are expected in a real spark cluster. Apart from that enabling this leads to a lot of exchanges of heart beats between nodes leading to flooding the network with those. + Specifies how many subdirectories under each local directory. Setting more sub directories under each local directory + can help to avoid the fat inode. | ||
| spark.cleaner.ttl | -(infinite) | +spark.files.fetchTimeout | +false | - Duration (seconds) of how long Spark will remember any metadata (stages generated, tasks generated, etc.). - Periodic cleanups will ensure that metadata older than this duration will be forgetten. This is - useful for running Spark for many hours / days (for example, running 24/7 in case of Spark Streaming - applications). Note that any RDD that persists in memory for more than this duration will be cleared as well. + Communication timeout to use when fetching files added through SparkContext.addFile() from + the driver. |
| spark.streaming.blockInterval | -200 | +spark.files.overwrite | +false | - Duration (milliseconds) of how long to batch new objects coming from network receivers used - in Spark Streaming. + Whether to overwrite files added through SparkContext.addFile() when the target file exists and its contents do not match those of the source. |
| spark.streaming.unpersist | -false | +spark.io.compression.codec | +org.apache.spark.io. LZFCompressionCodec |
- Force RDDs generated and persisted by Spark Streaming to be automatically unpersisted from
- Spark's memory. Setting this to true is likely to reduce Spark's RDD memory usage.
+ The codec used to compress internal data such as RDD partitions and shuffle outputs. By default, Spark provides two
+ codecs: org.apache.spark.io.LZFCompressionCodec and org.apache.spark.io.SnappyCompressionCodec.
|
| spark.task.maxFailures | +spark.io.compression.snappy.block.size | +32768 | ++ Block size (in bytes) used in Snappy compression, in the case when Snappy compression codec is used. + | +|
| spark.jars | +(not set) | ++ The external jars to be propagated to the executors from the driver. Users can specify multiple jars by separating them + with ",". + | +||
| spark.kryo.referenceTracking | +true | ++ Whether to track references to the same object when serializing data with Kryo, which is + necessary if your object graphs have loops and useful for efficiency if they contain multiple + copies of the same object. Can be disabled to improve performance if you know this is not the + case. + | +||
| spark.kryoserializer.buffer.mb | +2 | ++ Maximum object size to allow within Kryo (the library needs to create a buffer at least as + large as the largest single object you'll serialize). Increase this if you get a "buffer limit + exceeded" exception inside Kryo. Note that there will be one buffer per core on each worker. + | +||
| spark.locality.wait.node | +spark.locality.wait | ++ Customize the locality wait for node locality. For example, you can set this to 0 to skip + node locality and search immediately for rack locality (if your cluster has rack information). + | +||
| spark.locality.wait.process | +spark.locality.wait | ++ Customize the locality wait for process locality. This affects tasks that attempt to access + cached data in a particular executor process. + | +||
| spark.locality.wait.rack | +spark.locality.wait | ++ Customize the locality wait for rack locality. + | +||
| spark.locality.wait | +3000 | +
+ Number of milliseconds to wait to launch a data-local task before giving up and launching it
+ on a less-local node. The same wait will be used to step through multiple locality levels
+ (process-local, node-local, rack-local and then any). It is also possible to customize the
+ waiting time for each level by setting spark.locality.wait.node, etc.
+ You should increase this setting if your tasks are long and see poor locality, but the
+ default usually works well.
+ |
+||
| spark.logging.exceptionPrintInterval | +10000 | ++ How frequently the driver reprints duplicate exceptions in full. (in milliseconds) + | +||
| spark.logConf | +false | ++ Whether to log the supplied SparkConf as INFO at start of spark context. + | +||
| spark.mesos.coarse | +false | ++ If set to "true", runs over Mesos clusters in + "coarse-grained" sharing mode, + where Spark acquires one long-lived Mesos task on each machine instead of one Mesos task per Spark task. + This gives lower-latency scheduling for short queries, but leaves resources in use for the whole + duration of the Spark job. + | +||
| spark.mesos.extra.cores | +0 | ++ When using Mesos, the user can specify how many cores each task can use in addition to the cores it gets from + the resource offer. + | +||
| spark.rdd.compress | +false | +
+ Whether to compress serialized RDD partitions (e.g. for StorageLevel.MEMORY_ONLY_SER).
+ Can save substantial space at the cost of some extra CPU time.
+ |
+||
| spark.reducer.maxMbInFlight | +48 | ++ Maximum size (in megabytes) of map outputs to fetch simultaneously from each reduce task. Since + each output requires us to create a buffer to receive it, this represents a fixed memory overhead + per reduce task, so keep it small unless you have a large amount of memory. + | +||
| spark.repl.class.uri | +null | ++ This parameter specifies the URI of the class defined by REPL as user-defined code. + | +||
| spark.resultGetter.threads | 4 | - Number of individual task failures before giving up on the job. - Should be greater than or equal to 1. Number of allowed retries = this value - 1. + Set the size of thread pool in TaskScheduler to deserialize and get the task results from remote end when necessary. + The size of the thread pool is fixed. | ||
| spark.broadcast.blockSize | -4096 | +spark.scheduler.mode | +FIFO |
- Size of each piece of a block in kilobytes for TorrentBroadcastFactory.
- Too large a value decreases parallelism during broadcast (makes it slower); however, if it is too small, BlockManager might take a performance hit.
+ The scheduling mode between
+ jobs submitted to the same SparkContext. Can be set to FAIR
+ to use fair sharing instead of queueing jobs one after another. Useful for
+ multi-user services.
+ |
+
| spark.scheduler.revive.interval | +1000 | ++ The interval length for the scheduler to revive the worker resource offers to run tasks. (in milliseconds) + | +||
| spark.serializer.objectStreamReset | +10000 | ++ When serializing using org.apache.spark.serializer.JavaSerializer, the serializer caches + objects to prevent writing redundant data, however that stops garbage collection of those + objects. By calling 'reset' you flush that info from the serializer, and allow old + objects to be collected. To turn off this periodic reset set it to a value of <= 0. + By default it will reset the serializer every 10,000 objects. + | +||
| spark.shuffle.compress | +true | ++ Whether to compress map output files. Generally a good idea. + | +||
| spark.shuffle.copier.threads | +6 | +
+ When using Netty, this parameter specifies the number of concurrent threads fetching the shuffle results. This parameter
+ is only valid when spark.shuffle.use.netty is set to true.
|
||
| spark.shuffle.consolidateFiles | false | @@ -411,6 +604,32 @@ Apart from these, the following properties are also available, and may be useful reduce the number of disk seeks and system calls made in creating intermediate shuffle files.|||
| spark.shuffle.memoryFraction | +0.3 | +
+ Fraction of Java heap to use for aggregation and cogroups during shuffles, if
+ spark.shuffle.spill is true. At any given time, the collective size of
+ all in-memory maps used for shuffles is bounded by this limit, beyond which the contents will
+ begin to spill to disk. If spills are often, consider increasing this value at the expense of
+ spark.storage.memoryFraction.
+ |
+||
| spark.shuffle.netty.connect.timeout | +60000 | ++ This parameter sets the timeout threshold for the Netty connection when shuffling. (in milliseconds) + | +||
| spark.shuffle.sender.port | +0 | +
+ When using Netty to send result for shuffle, this parameter specifies the port number for data transfer.
+ This parameter is only valid when spark.shuffle.use.netty is set to true.
+ |
+||
| spark.shuffle.spill | true | @@ -419,6 +638,27 @@ Apart from these, the following properties are also available, and may be useful threshold is specified by|||
| spark.shuffle.spill.compress | +true | ++ Whether to compress data spilled during shuffles. + | +||
| spark.shuffle.sync | +false | ++ If this parameter is set to true, Spark will force all outstanding writing for the Map outputs be flushed to the disk when closing the writers. + | +||
| spark.shuffle.use.netty | +false | ++ If use Netty to fetch map output in shuffle. + | +||
| spark.speculation | false | @@ -433,6 +673,13 @@ Apart from these, the following properties are also available, and may be useful How often Spark will check for tasks to speculate, in milliseconds.|||
| spark.speculation.multiplier | +1.5 | ++ How many times slower a task is than the median to be considered for speculation. + | +||
| spark.speculation.quantile | 0.75 | @@ -441,48 +688,117 @@ Apart from these, the following properties are also available, and may be useful|||
| spark.speculation.multiplier | -1.5 | +spark.starvation.timeout | +15000 | - How many times slower a task is than the median to be considered for speculation. + Threshold above which we warn user initial TaskSet may be starved. (in milliseconds) |
| spark.logConf | +spark.storage.blockManagerSlaveTimeoutMs | +spark.storage.blockManagerTimeoutIntervalMs * 3 |
+ + Number of milliseconds after which the BlockManagerMasterActor considers a BlockManager running on the executor lost if it receives no heartbeats. + | +|
| spark.storage.blockManagerTimeoutIntervalMs | +60000 | +
+ This parameter controls two values: First, the interval length for the BlockManagers running on the executors to send hearbeat
+ to the BlockManager running on the driver. Please note that in Spark the actual frequency is the value set with this parameter divided by 4.
+ Second, this value also controls the default timeout threshold of the slave node, which is spark.storage.blockManagerTimeoutIntervalMs * 3.
+ (In milliseconds)
+ |
+||
| spark.storage.memoryFraction | +0.6 | ++ Fraction of Java heap to use for Spark's memory cache. This should not be larger than the "old" + generation of objects in the JVM, which by default is given 0.6 of the heap, but you can increase + it if you configure your own old generation size. + | +||
| spark.streaming.blockInterval | +200 | ++ Duration (milliseconds) of how long to batch new objects coming from network receivers used + in Spark Streaming. + | +||
| spark.streaming.unpersist | false | - Log the supplied SparkConf as INFO at start of spark context. + Force RDDs generated and persisted by Spark Streaming to be automatically unpersisted from + Spark's memory. Setting this to true is likely to reduce Spark's RDD memory usage. | ||
| spark.deploy.spreadOut | -true | +spark.task.cpus | +1 |
- Whether the standalone cluster manager should spread applications out across nodes or try
- to consolidate them onto as few nodes as possible. Spreading out is usually better for
- data locality in HDFS, but consolidating is more efficient for compute-intensive workloads. - Note: this setting needs to be configured in the standalone cluster master, not in individual - applications; you can set it through SPARK_JAVA_OPTS in spark-env.sh.
+ Number of cores to allocate for each task.
|
| spark.deploy.defaultCores | -(infinite) | +spark.task.maxFailures | +4 |
- Default number of cores to give to applications in Spark's standalone mode if they don't
- set spark.cores.max. If not set, applications always get all available
- cores unless they configure spark.cores.max themselves.
- Set this lower on a shared cluster to prevent users from grabbing
- the whole cluster by default. - Note: this setting needs to be configured in the standalone cluster master, not in individual - applications; you can set it through SPARK_JAVA_OPTS in spark-env.sh.
- |
+ Number of individual task failures before giving up on the job.
+ Should be greater than or equal to 1. Number of allowed retries = this value - 1.
+
| spark.files.overwrite | +spark.ui.acls.enable | false |
- Whether to overwrite files added through SparkContext.addFile() when the target file exists and its contents do not match those of the source.
+ Whether spark web ui acls should are enabled. If enabled, this checks to see if the user has
+ access permissions to view the web ui. See spark.ui.view.acls for more details.
+ Also note this requires the user to be known, if the user comes across as null no checks
+ are done. Filters can be used to authenticate and set the user.
+ |
+spark.ui.filters | +None | +
+ Comma separated list of filter class names to apply to the Spark web ui. The filter should be a
+ standard javax servlet Filter. Parameters to each filter can also be specified by setting a
+ java system property of spark. |
+
+
| spark.ui.port | +4040 | ++ Port for your application's dashboard, which shows memory and workload data + | +||
| spark.ui.retainedStages | +1000 | ++ How many stages the Spark UI remembers before garbage collecting. + | +||
| spark.ui.view.acls | +Empty | ++ Comma separated list of users that have view access to the spark web ui. By default only the + user that started the Spark job has view access. + | +||
| spark.worker.timeout | +60 | ++ Number of seconds after which the standalone deploy master considers a worker lost if it + receives no heartbeats. |