Uh oh!
There was an error while loading. Please reload this page.
SPARK-1178: missing document of spark.scheduler.revive.interval - #74
Closed
CodingCat wants to merge 1 commit into
Closed
SPARK-1178: missing document of spark.scheduler.revive.interval#74CodingCat wants to merge 1 commit into
CodingCat wants to merge 1 commit into
Conversation
AmplabJenkins
commented
Mar 4, 2014
Merged build triggered. |
AmplabJenkins
commented
Mar 4, 2014
Merged build started. |
AmplabJenkins
commented
Mar 4, 2014
Merged build finished. |
AmplabJenkins
commented
Mar 4, 2014
All automated tests passed. |
rxin
commented
Mar 4, 2014
Contributor
Thanks. Merged. |
jhartlaub referenced
this pull request
in jhartlaub/spark
May 27, 2014
Job cancellation via job group id.
This PR adds a simple API to group together a set of jobs belonging to a thread and threads spawned from it. It also allows the cancellation of all jobs in this group.
An example:
sc.setJobDescription("this_is_the_group_id", "some job description")
sc.parallelize(1 to 10000, 2).map { i => Thread.sleep(10); i }.count()
In a separate thread:
sc.cancelJobGroup("this_is_the_group_id")
(cherry picked from commit 599dcb0)
Signed-off-by: Reynold Xin <rxin@apache.org>clockfly added a commit
to clockfly/spark
that referenced
this pull request
Sep 22, 2016
…for implementing percentile_approx This is cherry-pick of open source master branch (hash: cc33460) ## What changes were proposed in this pull request? This is a sub-task of [SPARK-16283](https://issues.apache.org/jira/browse/SPARK-16283) (Implement percentile_approx SQL function), which moves class QuantileSummaries to project catalyst so that it can be reused when implementing aggregation function `percentile_approx`. This PR only does class relocation, class implementation is not changed. Author: Sean Zhong <seanzhongdatabricks.com> Author: Sean Zhong <seanzhong@databricks.com> Closesapache#74 from clockfly/move_quantile_summaries.
robert3005 pushed a commit
to robert3005/spark
that referenced
this pull request
Jan 12, 2017
cenyuhai added a commit
to cenyuhai/spark
that referenced
this pull request
Oct 8, 2017
[SPARK-20865] Structured streaming dataframe cache、unpersist报错 structured streaming dataset cache 会报错，应当给个log告警忽略cache,unpersist等操作。 resolveapache#74 See merge request !65
jlopezmalla pushed a commit
to jlopezmalla/spark
that referenced
this pull request
Nov 3, 2017
ashangit added a commit
to ashangit/spark
that referenced
this pull request
Jul 18, 2018
Bump spark criteo-2.2 to last branch-2.2 commits
cloud-fan pushed a commit
that referenced
this pull request
Jan 14, 2021
…join can be planned as broadcast join
### What changes were proposed in this pull request?
Should not pushdown LeftSemi/LeftAnti over Aggregate for some cases.
```scala
spark.range(50000000L).selectExpr("id % 10000 as a", "id % 10000 as b").write.saveAsTable("t1")
spark.range(40000000L).selectExpr("id % 8000 as c", "id % 8000 as d").write.saveAsTable("t2")
spark.sql("SELECT distinct a, b FROM t1 INTERSECT SELECT distinct c, d FROM t2").explain
```
Before this pr:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- Exchange hashpartitioning(a#16L, b#17L, 5), ENSURE_REQUIREMENTS, [id=#72]
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- SortMergeJoin [coalesce(a#16L, 0), isnull(a#16L), coalesce(b#17L, 0), isnull(b#17L)], [coalesce(c#18L, 0), isnull(c#18L), coalesce(d#19L, 0), isnull(d#19L)], LeftSemi
:- Sort [coalesce(a#16L, 0) ASC NULLS FIRST, isnull(a#16L) ASC NULLS FIRST, coalesce(b#17L, 0) ASC NULLS FIRST, isnull(b#17L) ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(coalesce(a#16L, 0), isnull(a#16L), coalesce(b#17L, 0), isnull(b#17L), 5), ENSURE_REQUIREMENTS, [id=#65]
: +- FileScan parquet default.t1[a#16L,b#17L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/spark-warehouse/org.apache.spark.sql.Data..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint>
+- Sort [coalesce(c#18L, 0) ASC NULLS FIRST, isnull(c#18L) ASC NULLS FIRST, coalesce(d#19L, 0) ASC NULLS FIRST, isnull(d#19L) ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(coalesce(c#18L, 0), isnull(c#18L), coalesce(d#19L, 0), isnull(d#19L), 5), ENSURE_REQUIREMENTS, [id=#66]
+- HashAggregate(keys=[c#18L, d#19L], functions=[])
+- Exchange hashpartitioning(c#18L, d#19L, 5), ENSURE_REQUIREMENTS, [id=#61]
+- HashAggregate(keys=[c#18L, d#19L], functions=[])
+- FileScan parquet default.t2[c#18L,d#19L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/spark-warehouse/org.apache.spark.sql.Data..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c:bigint,d:bigint>
```
After this pr:
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- Exchange hashpartitioning(a#16L, b#17L, 5), ENSURE_REQUIREMENTS, [id=#74]
+- HashAggregate(keys=[a#16L, b#17L], functions=[])
+- SortMergeJoin [coalesce(a#16L, 0), isnull(a#16L), coalesce(b#17L, 0), isnull(b#17L)], [coalesce(c#18L, 0), isnull(c#18L), coalesce(d#19L, 0), isnull(d#19L)], LeftSemi
:- Sort [coalesce(a#16L, 0) ASC NULLS FIRST, isnull(a#16L) ASC NULLS FIRST, coalesce(b#17L, 0) ASC NULLS FIRST, isnull(b#17L) ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(coalesce(a#16L, 0), isnull(a#16L), coalesce(b#17L, 0), isnull(b#17L), 5), ENSURE_REQUIREMENTS, [id=#67]
: +- HashAggregate(keys=[a#16L, b#17L], functions=[])
: +- Exchange hashpartitioning(a#16L, b#17L, 5), ENSURE_REQUIREMENTS, [id=#61]
: +- HashAggregate(keys=[a#16L, b#17L], functions=[])
: +- FileScan parquet default.t1[a#16L,b#17L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/spark-warehouse/org.apache.spark.sql.Data..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<a:bigint,b:bigint>
+- Sort [coalesce(c#18L, 0) ASC NULLS FIRST, isnull(c#18L) ASC NULLS FIRST, coalesce(d#19L, 0) ASC NULLS FIRST, isnull(d#19L) ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(coalesce(c#18L, 0), isnull(c#18L), coalesce(d#19L, 0), isnull(d#19L), 5), ENSURE_REQUIREMENTS, [id=#68]
+- HashAggregate(keys=[c#18L, d#19L], functions=[])
+- Exchange hashpartitioning(c#18L, d#19L, 5), ENSURE_REQUIREMENTS, [id=#63]
+- HashAggregate(keys=[c#18L, d#19L], functions=[])
+- FileScan parquet default.t2[c#18L,d#19L] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/Users/yumwang/spark/spark-warehouse/org.apache.spark.sql.Data..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<c:bigint,d:bigint>
```
### Why are the changes needed?
1. Pushdown LeftSemi/LeftAnti over Aggregate will affect performance.
2. It will remove user added DISTINCT operator, e.g.: [q38](https://github.com/apache/spark/blob/master/sql/core/src/test/resources/tpcds/q38.sql), [q87](https://github.com/apache/spark/blob/master/sql/core/src/test/resources/tpcds/q87.sql).
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test and benchmark test.
SQL | Before this PR(Seconds) | After this PR(Seconds)
-- | -- | --
q14a | 660 | 594
q14b | 660 | 600
q38 | 55 | 29
q87 | 66 | 35
Before this pr:
After this pr:
Closes#31145 from wangyum/SPARK-34081.
Authored-by: Yuming Wang <yumwang@ebay.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
MaxGekk added a commit
to MaxGekk/spark
that referenced
this pull request
Sep 2, 2026
… budget (apache#74) ### What changes were proposed in this pull request? A plan only, no code: section 10 of `PLAN_TASK_32.md`, plus a pointer from `PLAN_MILESTONE_4.md`'s task-32 row and section 2.9. Step B2 is the grouping change that turns step B1's fragment sharing into a hot-loop win. Today every calendar output still gets its own loop method (`CHRONO_WEIGHT` 50 against `GROUP_BUDGET` 16), so the shared civil-from-days prefix only ever fires in the epilogue. The gate measured B2's shape at 1.29x/1.57x/1.80x for two/three/four fields at AVX-512 and nearly the same at 128-bit (`PLAN_TASK_32.md` 7.2, 7.4), and measured the compile cliff away (7.5) - but it did so by forcing `groupBudget` to 200, which is the measurement rig, not a shippable rule: a wide budget also merges plain chains, and task 17 measured that losing (4436 vs 3150 M rows/s in the committed parity file). The plan replaces that with a rule. `addOps` counts a fragment's prefix once per group and reports what an output saved by reusing one; `groupOutputs` gains a second join clause that admits an output past `GROUP_BUDGET` only when `saved > 0`, bounded by a new `fusedCeiling` emit option. That is section 3.2's original sketch with one correction: 3.2 wrote the test as `marginal <= budget`, which would have re-merged task 17's case (marginal 6); `saved > 0` keeps it split. Section 10.2 works the rule through the shapes this task has measured, including one recorded limitation (greedy output order can leave `month(d)` in a group without `year(d)`; correct, no worse than today, parked in the debt register). The rest: weights re-counted from emitted instructions because they now bound a method (10.3); a ladder past four fields - 6, 8, 12 outputs over one date via `add_months(d, k)` literals - that sets `fusedCeiling` on throughput and `-XX:+PrintCompilation` time at both widths (10.4); which tests flip and which guard is added (10.5, including a byte-identity check over non-calendar shapes so clause 2 cannot leak past fragments); six registered predictions, one of them for task 45 to score (10.6); risks (10.7); a four-commit sequence after apache#73 and apache#64 have landed (10.8); and what stays out (10.9). ### Why are the changes needed? The gate for B2 has cleared at both widths and the remaining work is a design choice with a measured wrong answer available (the wide budget). Writing the rule down before building it is the project's standing practice, and this one has a specific trap - 3.2's own sketch re-merges task 17's case - that is cheaper to catch on paper than in a parity regeneration diff. ### Does this PR introduce _any_ user-facing change? No. Documentation only. ### How was this patch tested? Not applicable - plan files only. ASCII checked; the prose width matches the file's existing convention. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5.1)
MaxGekk added a commit
to MaxGekk/spark
that referenced
this pull request
Sep 4, 2026
…ne 5 for the other lanes; tasks 56-62 ### What changes were proposed in this pull request? Plan files only, in four commits, each standing on its own. **1. The task table audited against master.** Every row of `PLAN_MILESTONE_4.md` without a DONE marker was checked against the code, its plan file and the merged pull requests. One row was done and unmarked: 41, `unix_date`/`date_from_unix_date`, shipped as apache#63. Row 32 now says what master shows: steps A and B1 done (apache#72), step B2 planned (`PLAN_TASK_32.md` 10, apache#74) and not built, since `groupOutputs` is unchanged and the emitter suite pins one loop method per calendar output. Rows 25, 44, 46 and 47 say "not started" with what moved under them; 39, 42 and 49 say "planned" with their dependencies; 49's plan is located in section 2.19 rather than a task file. `PLAN_TASK_43.md` carried two "8. Outcome" headings; the template one is gone. Rows 34, 35, 38 and 52 are left to apache#115 and apache#116, which close them. **2. Milestone 4 re-scoped to the date family and the emitter; the other lanes become milestone 5.** The owner's decision: milestone 4 is `DateType` plus the emitter and evaluator infrastructure, and every task whose subject is another lane or output type moves out. Six tasks move, text and numbers unchanged, into a new `PLAN_MILESTONE_5.md`: 27 boolean outputs, 28 lane-width conversion, 29 int64 lanes, 30 ANSI integer arithmetic, 39 `date - date` and 49 civil-from-days in long lanes, with their design sections, table rows, verification, risks, the two open questions they own, and the catalogue items about other lanes and the follow-on ladder (1-5, 7-10), item numbers preserved. Milestone 4 keeps stubs under the old section numbers so citations resolve, a dated re-scope note at the top because a plan is a record, and its own catalogue items 6, 11, 12 and 13. The coverage milestone that was called 5 is now 6: `SCOPE_MILESTONE_5.md` is renamed to `SCOPE_MILESTONE_6.md` with a renumbering note, and every reference that meant the coverage milestone (README, VISION, AGENTS, SKILLS, milestone 3's plan, tasks 19, 21 and 22, the quote allowlist) says 6. README gains a milestone 5 entry. **3. Six date-lane tasks from a coverage survey, 56-61.** A survey of Spark's date surface (the function registry and `datetimeExpressions.scala`) against the compiler's arms, taken after the re-scope, left six gaps a date lane can close, each with a design section (2.23 to 2.28) and a row: 56 `date + INTERVAL n DAY` as a compiler rewrite of the int-cast form onto task 38's column-offset node, the stored int64 interval column staying out by decision; 57 `extract(DAYOFWEEK_ISO)` as one narrow arm to a new node, not general int arithmetic; 58 `extract(YEAROFWEEK)` as `year` over task 37's Thursday shift; 59 `next_day` with a weekday column, introducing the derived int32 leaf (an evaluator pre-pass through the row engine's own parser) that 61 reuses; 60 `add_months` with a month-count column, its compile-time bound moved to task 52's producer guard; 61 `trunc` with a format column as a node computing all four levels and selecting per lane. **4. Task 62, the closing measurement.** Every committed number so far compares two Varka lowerings on a laptop whose AVX-512 runs through a 256-bit datapath, against this fork with Varka off. The milestone closes with the number a reader wants: every covered date expression, in the projection and filter shapes, against stock Apache Spark on JDK 17 (its default) and on JDK 25, with Varka on JDK 25, on a runner with a real 512-bit datapath - one Java driver submitted to the three distributions in one dispatch of the benchmark workflow with `expected-cpu` pinned to a full-width Xeon, provenance including the 256-to-512 op-count ladder that proves the datapath, and README's benchmark section rewritten from the three files with a reproduction guide an outside reader can follow. Section 2.29 and row 62. ### Why are the changes needed? The table had drifted from master (one finished task unmarked, several rows with no state), the milestone's scope had grown past its title, and the coverage survey found date-lane gaps no row named. Milestone 4 now says what it is, what is left in it, and how it ends. ### Does this PR introduce _any_ user-facing change? No. Plan and documentation files only; no code. ### How was this patch tested? `dev/varka_quote_check.py` (every quoted figure traces to a committed results file; the four probe figures followed their paragraph to `PLAN_MILESTONE_5.md` in the allowlist) and `dev/varka_precommit.sh --working-tree` on every commit. Every cross-reference to the renamed scope document was found by `git grep` and repointed; the milestone 4 stubs keep every old section and item number resolvable. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5.1)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://spark-project.atlassian.net/browse/SPARK-1178
The configuration on spark.scheduler.revive.interval is undocumented but actually used
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/cluster/CoarseGrainedSchedulerBackend.scala#L64