Uh oh!
There was an error while loading. Please reload this page.
SPARK-964 Fix for -java-home note. - #71
Closed
ScrapCodes wants to merge 1 commit into
Closed
Conversation
ScrapCodes
commented
Mar 4, 2014
MemberAuthor
@pwendell Hey Patrick, It might be good to have jenkins not test the PRs which start with [WIP] or WIP. Or something like that ? |
AmplabJenkins
commented
Mar 4, 2014
Build triggered. |
AmplabJenkins
commented
Mar 4, 2014
Build started. |
AmplabJenkins
commented
Mar 4, 2014
Merged build triggered. |
AmplabJenkins
commented
Mar 4, 2014
Merged build started. |
ScrapCodes
commented
Mar 4, 2014
MemberAuthor
It does not cover the case if JAVA_HOME points to invalid directory, it will simply take alternate path instead of failing nicely. |
AmplabJenkins
commented
Mar 4, 2014
Merged build finished. |
AmplabJenkins
commented
Mar 4, 2014
One or more automated tests failed |
AmplabJenkins
commented
Mar 4, 2014
Build finished. |
AmplabJenkins
commented
Mar 4, 2014
All automated tests passed. |
AmplabJenkins
commented
Mar 4, 2014
Merged build triggered. |
AmplabJenkins
commented
Mar 4, 2014
Merged build started. |
AmplabJenkins
commented
Mar 4, 2014
Merged build finished. |
AmplabJenkins
commented
Mar 4, 2014
All automated tests passed. |
jhartlaub referenced
this pull request
in jhartlaub/spark
May 27, 2014
Fix mesos urls This was a bug I introduced in https://github.com/apache/incubator-spark/pull/71. Previously, we explicitly removed the mesos:// part; with alteryx#71, this no longer occurs. (cherry picked from commit 39d2e9b) Signed-off-by: Reynold Xin <rxin@apache.org>
jhartlaub referenced
this pull request
in jhartlaub/spark
May 27, 2014
Fix mesos urls This was a bug I introduced in https://github.com/apache/incubator-spark/pull/71. Previously, we explicitly removed the mesos:// part; with alteryx#71, this no longer occurs. (cherry picked from commit 39d2e9b) Signed-off-by: Reynold Xin <rxin@apache.org>
JasonMWhite pushed a commit
to JasonMWhite/spark
that referenced
this pull request
Dec 2, 2015
clockfly added a commit
to clockfly/spark
that referenced
this pull request
Sep 22, 2016
…s internal aggregation buffer object ## What changes were proposed in this pull request? (This PR cherry-picks PR apache#14753 to Databricks Spark branch-2.0.) This PR introduces an abstract class `TypedImperativeAggregate` so that an aggregation function of TypedImperativeAggregate can use **arbitrary** user-defined Java object as intermediate aggregation buffer object. **This has advantages like:** 1. It now can support larger category of aggregation functions. For example, it will be much easier to implement aggregation function `percentile_approx`, which has a complex aggregation buffer definition. 2. It can be used to avoid doing serialization/de-serialization for every call of `update` or `merge` when converting domain specific aggregation object to internal Spark-Sql storage format. 3. It is easier to integrate with other existing monoid libraries like algebird, and supports more aggregation functions with high performance. Please see `org.apache.spark.sql.TypedImperativeAggregateSuite.TypedMaxAggregate` to find an example of how to defined a `TypedImperativeAggregate` aggregation function. Please see Java doc of `TypedImperativeAggregate` and Jira ticket SPARK-17187 for more information. ## How was this patch tested? Unit tests. Author: Sean Zhong <seanzhongdatabricks.com> Author: Yin Huai <yhuaidatabricks.com> Closesapache#14753 from clockfly/object_aggregation_buffer_try_2. Author: Sean Zhong <seanzhong@databricks.com> Closesapache#71 from liancheng/typed-imperative-agg-db-2.0.
ash211 pushed a commit
to ash211/spark
that referenced
this pull request
Feb 2, 2017
* Allow adding arbitrary files * Address comments and add documentation
lins05 pushed a commit
to lins05/spark
that referenced
this pull request
Apr 23, 2017
* Allow adding arbitrary files * Address comments and add documentation
erikerlandson pushed a commit
to erikerlandson/spark
that referenced
this pull request
Jul 28, 2017
* Allow adding arbitrary files * Address comments and add documentation
jamesrgrinter pushed a commit
to jamesrgrinter/spark
that referenced
this pull request
Apr 22, 2018
Signed-off-by: Rostyslav Sotnychenko <rsotnychenko@maprtech.com> (cherry picked from commit e338b71)
ashangit pushed a commit
to ashangit/spark
that referenced
this pull request
Jul 13, 2018
LAKECOMP-390 Fix race condition for fileToAppInfo field
Igosuki pushed a commit
to Adikteev/spark
that referenced
this pull request
Jul 31, 2018
Show spark log even after the task has completed
cloud-fan pushed a commit
that referenced
this pull request
Nov 10, 2020
### What changes were proposed in this pull request?
Push down filter through expand. For case below:
```
create table t1(pid int, uid int, sid int, dt date, suid int) using parquet;
create table t2(pid int, vs int, uid int, csid int) using parquet;
SELECT
years,
appversion,
SUM(uusers) AS users
FROM (SELECT
Date_trunc('year', dt) AS years,
CASE
WHEN h.pid = 3 THEN 'iOS'
WHEN h.pid = 4 THEN 'Android'
ELSE 'Other'
END AS viewport,
h.vs AS appversion,
Count(DISTINCT u.uid) AS uusers
,Count(DISTINCT u.suid) AS srcusers
FROM t1 u
join t2 h
ON h.uid = u.uid
GROUP BY 1,
2,
3) AS a
WHERE viewport = 'iOS'
GROUP BY 1,
2
```
Plan. before this pr:
```
== Physical Plan ==
*(5) HashAggregate(keys=[years#30, appversion#32], functions=[sum(uusers#33L)])
+- Exchange hashpartitioning(years#30, appversion#32, 200), true, [id=#251]
+- *(4) HashAggregate(keys=[years#30, appversion#32], functions=[partial_sum(uusers#33L)])
+- *(4) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12], functions=[count(if ((gid#44 = 1)) u.`uid`#47 else null)])
+- Exchange hashpartitioning(date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, 200), true, [id=#246]
+- *(3) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12], functions=[partial_count(if ((gid#44 = 1)) u.`uid`#47 else null)])
+- *(3) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44], functions=[])
+- Exchange hashpartitioning(date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44, 200), true, [id=#241]
+- *(2) HashAggregate(keys=[date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44], functions=[])
+- *(2) Filter (CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46 = iOS)
+- *(2) Expand [ArrayBuffer(date_trunc(year, cast(dt#9 as timestamp), Some(Etc/GMT+7)), CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END, vs#12, uid#7, null, 1), ArrayBuffer(date_trunc(year, cast(dt#9 as timestamp), Some(Etc/GMT+7)), CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END, vs#12, null, suid#10, 2)], [date_trunc('year', CAST(u.`dt` AS TIMESTAMP))#45, CASE WHEN (h.`pid` = 3) THEN 'iOS' WHEN (h.`pid` = 4) THEN 'Android' ELSE 'Other' END#46, vs#12, u.`uid`#47, u.`suid`#48, gid#44]
+- *(2) Project [uid#7, dt#9, suid#10, pid#11, vs#12]
+- *(2) BroadcastHashJoin [uid#7], [uid#13], Inner, BuildRight
:- *(2) Project [uid#7, dt#9, suid#10]
: +- *(2) Filter isnotnull(uid#7)
: +- *(2) ColumnarToRow
: +- FileScan parquet default.t1[uid#7,dt#9,suid#10] Batched: true, DataFilters: [isnotnull(uid#7)], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.0-bin-hadoop3.2/spark-warehouse/t1], PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<uid:int,dt:date,suid:int>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[2, int, true] as bigint))), [id=#233]
+- *(1) Project [pid#11, vs#12, uid#13]
+- *(1) Filter isnotnull(uid#13)
+- *(1) ColumnarToRow
+- FileScan parquet default.t2[pid#11,vs#12,uid#13] Batched: true, DataFilters: [isnotnull(uid#13)], Format: Parquet, Location: InMemoryFileIndex[file:/root/spark-3.0.0-bin-hadoop3.2/spark-warehouse/t2], PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<pid:int,vs:int,uid:int>
```
Plan. after. this pr. :
```
== Physical Plan ==
AdaptiveSparkPlan isFinalPlan=false
+- HashAggregate(keys=[years#0, appversion#2], functions=[sum(uusers#3L)], output=[years#0, appversion#2, users#5L])
+- Exchange hashpartitioning(years#0, appversion#2, 5), true, [id=#71]
+- HashAggregate(keys=[years#0, appversion#2], functions=[partial_sum(uusers#3L)], output=[years#0, appversion#2, sum#22L])
+- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12], functions=[count(distinct uid#7)], output=[years#0, appversion#2, uusers#3L])
+- Exchange hashpartitioning(date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, 5), true, [id=#67]
+- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12], functions=[partial_count(distinct uid#7)], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, count#27L])
+- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7], functions=[], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7])
+- Exchange hashpartitioning(date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7, 5), true, [id=#63]
+- HashAggregate(keys=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles)) AS date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END AS CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7], functions=[], output=[date_trunc(year, cast(dt#9 as timestamp), Some(America/Los_Angeles))#23, CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END#24, vs#12, uid#7])
+- Project [uid#7, dt#9, pid#11, vs#12]
+- BroadcastHashJoin [uid#7], [uid#13], Inner, BuildRight, false
:- Filter isnotnull(uid#7)
: +- FileScan parquet default.t1[uid#7,dt#9] Batched: true, DataFilters: [isnotnull(uid#7)], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/4l/7_c5c97s1_gb0d9_d6shygx00000gn/T/warehouse-c069d87..., PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<uid:int,dt:date>
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[2, int, false] as bigint)),false), [id=#58]
+- Filter ((CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END = iOS) AND isnotnull(uid#13))
+- FileScan parquet default.t2[pid#11,vs#12,uid#13] Batched: true, DataFilters: [(CASE WHEN (pid#11 = 3) THEN iOS WHEN (pid#11 = 4) THEN Android ELSE Other END = iOS), isnotnull..., Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/4l/7_c5c97s1_gb0d9_d6shygx00000gn/T/warehouse-c069d87..., PartitionFilters: [], PushedFilters: [IsNotNull(uid)], ReadSchema: struct<pid:int,vs:int,uid:int>
```
### Why are the changes needed?
Improve performance, filter more data.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added UT
Closes#30278 from AngersZhuuuu/SPARK-33302.
Authored-by: angerszhu <angers.zhu@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>arjunshroff pushed a commit
to arjunshroff/spark
that referenced
this pull request
Nov 24, 2020
VandanaSah pushed a commit
to VandanaSah/spark
that referenced
this pull request
Apr 13, 2026
MaxGekk added a commit
to MaxGekk/spark
that referenced
this pull request
Sep 1, 2026
…k the resample (apache#71) ### What changes were proposed in this pull request? Adds task 50 to the milestone-4 plan, and records in the debt register a related idea that is deliberately **not** being built. Plan only - no code. A companion investigation found that one of the engine's kernels runs 30-40% slower on some JVM starts than others, at random, on narrow vector hardware. The cause is the JIT's register allocator: given identical input it sometimes produces a good assignment and sometimes one that spills heavily to the stack. The kernel sits right at the edge of the available registers on that hardware, and which side of the edge it lands on is a coin flip taken once per JVM. The structural fix belongs to the task that created the situation - don't ask for more registers than the hardware has. **This task is the other half: noticing when it happens.** Today a badly-compiled kernel is completely invisible; it just makes queries slower and nothing reports it. It turns out to be observable using ordinary public Java APIs. The JVM publishes an event when it compiles a method, and that event includes the size of the compiled code. The good and bad versions of this kernel differ by roughly a factor of two in that number, so the problem is plainly visible in a field anyone can read, with no agent, no special flags and no debug JVM. The design point worth the write-up is what to compare against. The obvious approach - a checked-in table of expected sizes - is the wrong one, because the table has to come from somewhere and goes stale every time the code generator changes. The engine already identifies each generated kernel by a fingerprint, and the same fingerprint always produces byte-identical code, so the comparison can simply be *between compilations of the same fingerprint*. Nothing to maintain, no drift, and it gets more accurate the longer a process runs. Scope is kept deliberately small: off unless switched on, a metric and a log line rather than any action, and explicitly no attempt to react. **The idea being parked.** Because every kernel is generated as a fresh class, regenerating one gives the allocator another roll of the dice - so once the problem is detectable, automatically retrying until you get a good compilation becomes possible with no new machinery. The debt register records why that is not scheduled, here or next milestone: each retry costs a class, a compilation and a warm-up, against a kernel that a short query runs only a handful of times, so it loses money exactly where it would matter most; unbounded retrying is a loop and bounded retrying is a slot machine; class churn is already a watched concern in this engine; and it treats a symptom whose cause can be removed outright at no runtime cost. It is worth revisiting only for a long-running kernel, and only once this task has produced numbers on how often the bad outcome actually occurs. ### Why are the changes needed? A 30-40% slowdown that strikes at random and reports nothing is the kind of problem that gets misattributed for a long time - this project spent six failed hypotheses on it before finding the cause, and could only do so with a purpose-built debug JVM. Making it visible in production costs almost nothing and turns it from a mystery into a fact. Writing down the tempting fix and why it is not being taken is the other half of the value. ### Does this PR introduce any user-facing change? No. Planning document only. ### How was this patch tested? Not applicable - no code. The measurements behind it are in the companion pull request. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I just did a manual testing of this.
with -java-home "jdk", setting just JAVA_HOME and both. Hope it covers all cases.