Uh oh!
There was an error while loading. Please reload this page.
Fixed link for latest pyspark api guide - #83
Closed
jyotiska wants to merge 1 commit into
Closed
Conversation
AmplabJenkins
commented
Mar 5, 2014
Merged build triggered. |
AmplabJenkins
commented
Mar 5, 2014
Merged build started. |
AmplabJenkins
commented
Mar 5, 2014
Merged build finished. |
AmplabJenkins
commented
Mar 5, 2014
All automated tests passed. |
Contributor
There was a problem hiding this comment.
It would be better to make this a relative URL... that way for people hosting copies of the docs won't break.
Contributor
There was a problem hiding this comment.
I don't think the current link is actually broken (try the link in the current docs). If you use a relative URL that contains latest, then you will break links for older versions of the docs by linking to API docs from newer versions.
Contributor
There was a problem hiding this comment.
ah - so then is this fix in general just not needed?
jhartlaub referenced
this pull request
in jhartlaub/spark
May 27, 2014
Add an add() method to pyspark accumulators. Add a regular method for adding a term to accumulators in pyspark. Currently if you have a non-global accumulator, adding to it is awkward. The += operator can't be used for non-global accumulators captured via closure because it's involves an assignment. The only way to do it is using __iadd__ directly. Adding this method lets you write code like this: def main(): sc = SparkContext() accum = sc.accumulator(0) rdd = sc.parallelize([1,2,3]) def f(x): accum.add(x) rdd.foreach(f) print accum.value where using accum += x instead would have caused UnboundLocalError exceptions in workers. Currently it would have to be written as accum.__iadd__(x). (cherry picked from commit 747f538) Signed-off-by: Reynold Xin <rxin@apache.org>
cenyuhai added a commit
to cenyuhai/spark
that referenced
this pull request
Oct 8, 2017
[SPARK-20464] Structured streaming web ui增加job描述信息 spark job ui增加description  resolveapache#83 See merge request !61
Igosuki pushed a commit
to Adikteev/spark
that referenced
this pull request
Jul 31, 2018
[SPARK-379] Update to dcos==0.4.14
weixiuli pushed a commit
to weixiuli/spark
that referenced
this pull request
Jun 18, 2019
Backport PR#80 to spark2.3.0 . Merging
bzhaoopenstack pushed a commit
to bzhaoopenstack/spark
that referenced
this pull request
Sep 11, 2019
…-ccm Remove commas between service names
MaxGekk added a commit
to MaxGekk/spark
that referenced
this pull request
Sep 2, 2026
…p table (apache#86) ### What changes were proposed in this pull request? Adds **Item 10, "The calendar as a lookup table, sized to the era"** to `SCOPE_MILESTONE_5.md`. A catalogue entry only — no code, and deliberately not a plan. It lands in milestone 5's catalogue rather than milestone 4's because milestone 4's file has become a task plan, and `sql/varka/AGENTS.md` sends work with no owner yet to the furthest-out scope document. By topic it belongs to the calendar family, and the item says so. **The idea.** ClickHouse's `DATE_LUT_SIZE` is `0x23AB1` — 146097, exactly one Gregorian era. It anchors that window at 1900 and falls back outside it, but 400 years is the calendar's *period*, so a table indexed by **day of era** needs no fallback for any `int32` date at all: every day reduces into it, and the year is `400 * (era - bias) + table[dayOfEra]`. Varka's prefix already computes that index — `emitEra` is the first thing it emits — so the table would replace everything after it. ClickHouse stores 16 bytes per day (year, month, day of month, day of week, days in month), so **one lookup yields every field**. That is the problem task 32 solves with a shared prefix, solved with memory instead, and it is the version of the idea worth measuring rather than the year-only one. **Already measured** (apache#85, `VarkaVectorApiProbeBenchmark`): with the column in a `MemorySegment` the way a real kernel has it, `year(d) = 1998` counted, an era-indexed year table runs at 2070.8 M rows/s against the arithmetic's 1329.3 — **1.6x** — at 571 KB for the table and about 10 KB touched by a seven-year query. ### Why are the changes needed? A 1.6x on a calendar node is worth writing down, and an idea with a measurement attached is the form this project's registers are supposed to take. Without an entry it becomes folklore from a conversation. The item is written so it can be **dropped as easily as built**: - It lists the five things that have to be measured before it is a task, led by the one comparison that actually matters and has *not* been run — a single gather yielding four fields against task 32's shared prefix at 797.7 M rows/s. Measuring only the year measures the weaker half of the idea. - It states the case against itself: the arithmetic is branch-free, needs no memory, and keeps getting cheaper — task 48 took four ops off the year tail, and apache#83 takes eighteen off `add_months` — while a table's cost is fixed and paid in cache. - It names the real cost if it proceeds: the emitter would gain a class of operand it has no notion of today, a constant table, and `GROUP_BUDGET` has no weight for a gather. It is deliberately **absent from section 5's ordering table**, with a sentence saying why: gated on that measurement, and not part of this milestone's decimal-and-aggregation spine. If the four-field number comes back the way the single-field one did, it becomes its own task and is scheduled then; if it does not, the entry is the record of why the idea was dropped, which is worth as much. ### Does this PR introduce _any_ user-facing change? No. One scope document; no code. ### How was this patch tested? Nothing to run. The numbers it cites are apache#85's committed benchmark rows rather than restated from a conversation, and the claim about what the gather can reach is the corrected one from that PR — an on-heap constant table indexed by off-heap data, not a gather from off-heap memory, which remains blocked. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixed link for PySpark API in Python programming guide to the latest version.