Uh oh!
There was an error while loading. Please reload this page.
update proportion of memory - #66
Closed
CrazyJvm wants to merge 1 commit into
Closed
Conversation
the default value of "spark.storage.memoryFraction" has been change from 0.66 to 0.6 . So it should be 60% of the memory to cache while 40% used for task execution.
AmplabJenkins
commented
Mar 3, 2014
Can one of the admins verify this patch? |
rxin
commented
Mar 3, 2014
Contributor
Thanks. I've merged this. |
wli600 pushed a commit
to wli600/spark
that referenced
this pull request
Jul 29, 2015
SKIPME merging Apache branch-1.4 bug fixes
JasonMWhite pushed a commit
to JasonMWhite/spark
that referenced
this pull request
Dec 2, 2015
add spark streaming requirements to pomfile
marcosdotps pushed a commit
to marcosdotps/spark
that referenced
this pull request
Sep 21, 2017
* revert change hosts * Update Jenkinsfile
cenyuhai added a commit
to cenyuhai/spark
that referenced
this pull request
Oct 8, 2017
[SPARK-21414] Refine SlidingWindowFunctionFrame to avoid OOM Refine SlidingWindowFunctionFrame to avoid OOM resolveapache#66 See merge request !59
ashangit added a commit
to ashangit/spark
that referenced
this pull request
Feb 22, 2018
[SPARK-22683][CORE] Allow tuning the number of dynamically allocated executors
jamesrgrinter pushed a commit
to jamesrgrinter/spark
that referenced
this pull request
Apr 22, 2018
Signed-off-by: Rostyslav Sotnychenko <rsotnychenko@maprtech.com> (cherry picked from commit de237dc)
Igosuki pushed a commit
to Adikteev/spark
that referenced
this pull request
Jul 31, 2018
[SPARK-300] fix docs for HDFS
clems4ever pushed a commit
to clems4ever/spark
that referenced
this pull request
Feb 11, 2019
[SPARK-22683][CORE] Allow tuning the number of dynamically allocated executors
bzhaoopenstack pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
UT: dims/openstack-cloud-controller-manager + devstack [in vm]
yuexingri pushed a commit
to yuexingri/spark
that referenced
this pull request
Dec 9, 2019
apache#39 performance issue in fuction getAliasedConstraints of LogicalPlan
arjunshroff pushed a commit
to arjunshroff/spark
that referenced
this pull request
Nov 24, 2020
cloud-fan pushed a commit
that referenced
this pull request
Jan 14, 2021
…join can be planned as broadcast join
### What changes were proposed in this pull request?
Should not pushdown LeftSemi/LeftAnti over Aggregate for some cases.
```scala
spark.range(50000000L).selectExpr("id % 10000 as a", "id % 10000 as b").write.saveAsTable("t1")
spark.range(40000000L).selectExpr("id % 8000 as c", "id % 8000 as d").write.saveAsTable("t2")
spark.sql("SELECT distinct a, b FROM t1 INTERSECT SELECT distinct c, d FROM t2").explain
```
Before this pr:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- Exchange hashpartitioning(a#16L, b#17L, 5), ENSURE_REQUIREMENTS, [id=#72]
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- SortMergeJoin [coalesce(a#16L, 0), isnull(a#16L), coalesce(b#17L, 0), isnull(b#17L)], [coalesce(c#18L, 0), isnull(c#18L), coalesce(d#19L, 0), isnull(d#19L)], LeftSemi
:- Sort [coalesce(a#16L, 0) ASC NULLS FIRST, isnull(a#16L) ASC NULLS FIRST, coalesce(b#17L, 0) ASC NULLS FIRST, isnull(b#17L) ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(coalesce(a#16L, 0), isnull(a#16L), coalesce(b#17L, 0), isnull(b#17L), 5), ENSURE_REQUIREMENTS, [id=#65]
: +- FileScan parquet default.t1[a#16L,b#17L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/spark-warehouse/org.apache.spark.sql.Data..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint>
+- Sort [coalesce(c#18L, 0) ASC NULLS FIRST, isnull(c#18L) ASC NULLS FIRST, coalesce(d#19L, 0) ASC NULLS FIRST, isnull(d#19L) ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(coalesce(c#18L, 0), isnull(c#18L), coalesce(d#19L, 0), isnull(d#19L), 5), ENSURE_REQUIREMENTS, [id=#66]
+- HashAggregate(keys=[c#18L, d#19L], functions=[])
+- Exchange hashpartitioning(c#18L, d#19L, 5), ENSURE_REQUIREMENTS, [id=#61]
+- HashAggregate(keys=[c#18L, d#19L], functions=[])
+- FileScan parquet default.t2[c#18L,d#19L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/spark-warehouse/org.apache.spark.sql.Data..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c:bigint,d:bigint>
```
After this pr:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- Exchange hashpartitioning(a#16L, b#17L, 5), ENSURE_REQUIREMENTS, [id=#74]
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- SortMergeJoin [coalesce(a#16L, 0), isnull(a#16L), coalesce(b#17L, 0), isnull(b#17L)], [coalesce(c#18L, 0), isnull(c#18L), coalesce(d#19L, 0), isnull(d#19L)], LeftSemi
:- Sort [coalesce(a#16L, 0) ASC NULLS FIRST, isnull(a#16L) ASC NULLS FIRST, coalesce(b#17L, 0) ASC NULLS FIRST, isnull(b#17L) ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(coalesce(a#16L, 0), isnull(a#16L), coalesce(b#17L, 0), isnull(b#17L), 5), ENSURE_REQUIREMENTS, [id=#67]
: +- HashAggregate(keys=[a#16L, b#17L], functions=[])
: +- Exchange hashpartitioning(a#16L, b#17L, 5), ENSURE_REQUIREMENTS, [id=#61]
: +- HashAggregate(keys=[a#16L, b#17L], functions=[])
: +- FileScan parquet default.t1[a#16L,b#17L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/spark-warehouse/org.apache.spark.sql.Data..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint>
+- Sort [coalesce(c#18L, 0) ASC NULLS FIRST, isnull(c#18L) ASC NULLS FIRST, coalesce(d#19L, 0) ASC NULLS FIRST, isnull(d#19L) ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(coalesce(c#18L, 0), isnull(c#18L), coalesce(d#19L, 0), isnull(d#19L), 5), ENSURE_REQUIREMENTS, [id=#68]
+- HashAggregate(keys=[c#18L, d#19L], functions=[])
+- Exchange hashpartitioning(c#18L, d#19L, 5), ENSURE_REQUIREMENTS, [id=#63]
+- HashAggregate(keys=[c#18L, d#19L], functions=[])
+- FileScan parquet default.t2[c#18L,d#19L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/spark-warehouse/org.apache.spark.sql.Data..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c:bigint,d:bigint>
```
### Why are the changes needed?
1. Pushdown LeftSemi/LeftAnti over Aggregate will affect performance.
2. It will remove user added DISTINCT operator, e.g.: [q38](https://github.com/apache/spark/blob/master/sql/core/src/test/resources/tpcds/q38.sql), [q87](https://github.com/apache/spark/blob/master/sql/core/src/test/resources/tpcds/q87.sql).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and benchmark test.
SQL | Before this PR(Seconds) | After this PR(Seconds)
-- | -- | --
q14a | 660 | 594
q14b | 660 | 600
q38 | 55 | 29
q87 | 66 | 35
Before this pr:
After this pr:
Closes#31145 from wangyum/SPARK-34081.
Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
dongjoon-hyun pushed a commit
that referenced
this pull request
Sep 12, 2024
…r `postgreSQL/float4.sql` and `postgreSQL/int8.sql` ### What changes were proposed in this pull request? This pr regenerate Java 21 golden file for `postgreSQL/float4.sql` and `postgreSQL/int8.sql` to fix Java 21 daily test. ### Why are the changes needed? Fix Java 21 daily test: - https://github.com/apache/spark/actions/runs/10823897095/job/30030200710 ``` [info] - postgreSQL/float4.sql *** FAILED *** (1 second, 100 milliseconds) [info] postgreSQL/float4.sql [info] Expected "...arameters" : { [info] "[ansiConfig" : "\"spark.sql.ansi.enabled\"", [info] "]expression" : "'N A ...", but got "...arameters" : { [info] "[]expression" : "'N A ..." Result did not match for query #11 [info] SELECT float('N A N') (SQLQueryTestSuite.scala:663) ... [info] - postgreSQL/int8.sql *** FAILED *** (2 seconds, 474 milliseconds) [info] postgreSQL/int8.sql [info] Expected "...arameters" : { [info] "[ansiConfig" : "\"spark.sql.ansi.enabled\"", [info] "]sourceType" : "\"BIG...", but got "...arameters" : { [info] "[]sourceType" : "\"BIG..." Result did not match for query #66 [info] SELECT CAST(q1 AS int) FROM int8_tbl WHERE q2 <> 456 (SQLQueryTestSuite.scala:663) ... [info] *** 2 TESTS FAILED *** [error] Failed: Total 3559, Failed 2, Errors 0, Passed 3557, Ignored 4 [error] Failed tests: [error] org.apache.spark.sql.SQLQueryTestSuite [error] (sql / Test / test) sbt.TestsFailedException: Tests unsuccessful ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - Pass Github Acitons - Manual checked: `build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite" with Java 21, all test passed ` ### Was this patch authored or co-authored using generative AI tooling? No Closes#48089 from LuciferYang/SPARK-49578-FOLLOWUP. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
MaxGekk added a commit
to MaxGekk/spark
that referenced
this pull request
Sep 1, 2026
…rs at AVX-512 The previous commit declined task 32 on 225.8 M rows/s against 430.7 for the four independently emitted nodes. That number does not measure sharing. ChronoVectorOps factored its decomposition into a computeFields helper returning a record of four IntVectors. javap puts that helper at 376 bytecode bytes and C2's FreqInlineSize is 325, so it never inlined into the loop; with no compilation unit holding both the allocation and its consumers, escape analysis could not scalar-replace the record, and the kernel really allocated five objects per lane group. VarkaLoopEmitter.emitChrono, the path the kernel exists to model, emits no call boundary at all in its lane path. Rewritten with the whole lane path written out by hand. javap now reports zero invokestatic to any ChronoVectorOps method, and -XX:+PrintInlining (narrowed to this method with -XX:CompileCommand=option) reports one non-inlined callee, fourFieldsEpilogue at bytecode 919 - past the loop, once per batch, which is the same split the emitter's loop/epilogue method pair has. Two omissions the review of PR apache#66 found are fixed with it, because a ceiling has to charge both arms the same things: the kernel now emits the narrow-range guard (once, where the four-node baseline pays it four times) and returns a VarkaFusedKernel-shaped status, and it writes four destination validity buffers rather than one. The benchmark's hand-copied chunk loop is replaced by the same eachChunk walk every other case in the section uses. Measured, five iterations over two-second windows on an idle machine: AVX-512 four nodes 450.4, 448.8 shared 692.4, 678.8 1.54x, 1.51x 128-bit four nodes 154.1 to 157.6 shared 165.6 to 167.0, once 236.1 So sharing is worth about 1.5x at the native width, reproducibly, and is a wash at 128-bit - four runs of five at 1.06x, one at 1.50x, with zero stdev inside each run and 42% between them, which is a compilation the JVM either finds or does not rather than noise. C1 declines the 936-byte body outright at both widths ("out of virtual registers in linear scan"). Task 17's register-pressure finding is visible here as a width-dependent ceiling on the size of the win, not as a reversal of its sign. PLAN_TASK_32.md replans the task around this: step B builds emitter-side fragment sharing keyed on (fragment, child node) behind a VarkaEmitOptions switch, with no IR change and with the default decided at both widths. Section 2.9, the task table row and the debt register entry are rewritten; SKILLS.md's lesson is replaced with the one that actually holds. Numbers requoted from the regenerated parity file, VarkaLoopEmitter's GROUP_BUDGET javadoc included. Also from the review: the quarter test oracle is now IsoFields.QUARTER_OF_YEAR rather than (month + 2) / 3, which restated the implementation; the private isBitSet copy is dropped for VarkaVectorSupport.isBitSet; SIZES regains 100. The "task 32 may move the plumbing" paragraph in PLAN_TASK_34.md and PLAN_TASK_35.md is deliberately kept - under this plan it is live guidance, not a stale contingency. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3VxmqDKhWHkvQ4Jrp6coG
MaxGekk added a commit
to MaxGekk/spark
that referenced
this pull request
Sep 1, 2026
Task 32 (apache#66) landed and both branches add cases to the same benchmark section and lessons to the same file. The benchmark takes both sets of cases in a sensible order: master's two shared decomposition cases beside the four-field case they are measured against, then this branch's scalar baseline and the four auto-vectorization probes. The import takes both kernels. The results file could not be resolved to either side, since both regenerated it and the merged benchmark has eleven cases neither version carried. It is regenerated here instead, which is the only correct resolution, and every quote of it is updated with it - SKILLS.md, PLAN_MILESTONE_4.md, PLAN_TASK_32.md, ChronoVectorOps's class doc and VarkaLoopEmitter's GROUP_BUDGET javadoc, whose task-17 figures move to 4119.9 against 2928.2. The two run-record tables are treated as history rather than rewritten: task 32's AVX-512 comparison gains a fourth run (445.7 against 679.0, ratio 1.52x, in line with the previous three) and the "committed file" label moves to it, rather than the earlier runs being restated as something they were not. Green: engine suite at both widths (ChronoVectorOpsTest 4, ChronoScalarOpsTest 3, each twice), catalyst 98 at both widths, dev/lint-java and dev/scalastyle clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V3VxmqDKhWHkvQ4Jrp6coG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default value of "spark.storage.memoryFraction" has been changed from 0.66 to 0.6 . So it should be 60% of the memory to cache while 40% used for task execution.