Skip to content

[SPARK-19140][SS]Allow update mode for non-aggregation streaming queries - #16520

Closed
zsxwing wants to merge 4 commits into
apache:masterfrom
zsxwing:update-without-agg
Closed

[SPARK-19140][SS]Allow update mode for non-aggregation streaming queries#16520
zsxwing wants to merge 4 commits into
apache:masterfrom
zsxwing:update-without-agg

Conversation

@zsxwing

@zsxwingzsxwing commented Jan 9, 2017

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR allow update mode for non-aggregation streaming queries. It will be same as the append mode if a query has no aggregations.

How was this patch tested?

Jenkins

@SparkQA

SparkQA commented Jan 10, 2017

Copy link
Copy Markdown

Test build #71096 has finished for PR 16520 at commit 7d62de4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@brkyvz

brkyvz commented Jan 10, 2017

Copy link
Copy Markdown
Contributor

LGTM! ultra nit: it will be equivalent to Append mode for some reason sounds better to me when reading it than it will be same as the Append mode.

brkyvz
brkyvz reviewed Jan 10, 2017
View reviewed changes
Comment threadsql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamWriter.scala
* - `complete`: all the rows in the streaming DataFrame/Dataset will be written to the sink
* every time these is some updates
*
* - `update`: only the rows that were updated in the streaming DataFrame/Dataset will

@brkyvzbrkyvzJan 10, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also please update pyspark docs?

@brkyvz

brkyvz commented Jan 10, 2017

Copy link
Copy Markdown
Contributor

Also could you please update pyspark docs?

@SparkQA

SparkQA commented Jan 10, 2017

Copy link
Copy Markdown

Test build #71157 has finished for PR 16520 at commit 9f2d877.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

brkyvz
brkyvz reviewed Jan 10, 2017
View reviewed changes
Comment threadpython/pyspark/sql/streaming.py Outdated
every time these is some updates
* `update`:only the rows that were updated in the streaming DataFrame/Dataset will be
written to the sink every time there are some updates. If the query doesn't contain
aggregations, it will be equivalent to the `append` mode.

@brkyvzbrkyvzJan 10, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: please remove the before append.

brkyvz
brkyvz reviewed Jan 10, 2017
View reviewed changes
Comment threadpython/pyspark/sql/streaming.py Outdated
every time these is some updates
* `update`:only the rows that were updated in the streaming DataFrame/Dataset will be
written to the sink every time there are some updates. If the query doesn't contain
aggregations, it will be equivalent to the `append` mode.

@brkyvzbrkyvzJan 10, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

brkyvz
brkyvz reviewed Jan 10, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/java/org/apache/spark/sql/streaming/OutputMode.java Outdated
* OutputMode in which only the rows that were updated in the streaming DataFrame/Dataset will
* be written to the sink every time there are some updates.
* be written to the sink every time there are some updates. If the query doesn't contain
* aggregations, it will be equivalent to the `Append` mode.

@brkyvzbrkyvzJan 10, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

brkyvz
brkyvz reviewed Jan 10, 2017
View reviewed changes
Comment threadsql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/streaming/InternalOutputModes.scala Outdated
* written to the sink every time these is some updates. This output mode can only be used in
* queries that contain aggregations.
* written to the sink every time these is some updates. If the query doesn't contain
* aggregations, it will be equivalent to the `Append` mode.

@brkyvzbrkyvzJan 10, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@brkyvz

brkyvz commented Jan 10, 2017

Copy link
Copy Markdown
Contributor

Left a nit (occurring multiple times), otherwise LGTM!

@brkyvz

brkyvz commented Jan 10, 2017

Copy link
Copy Markdown
Contributor

thanks LGTM!

@SparkQA

SparkQA commented Jan 10, 2017

Copy link
Copy Markdown

Test build #71159 has finished for PR 16520 at commit e4f2403.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Jan 11, 2017

Copy link
Copy Markdown

Test build #71168 has finished for PR 16520 at commit 889315d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@zsxwing

zsxwing commented Jan 11, 2017

Copy link
Copy Markdown
MemberAuthor

Thanks. Merging to master and 2.1.

asfgit pushed a commit that referenced this pull request Jan 11, 2017
@zsxwing
[SPARK-19140][SS] Allow update mode for non-aggregation streaming que…
230607d
…ries
## What changes were proposed in this pull request?
This PR allow update mode for non-aggregation streaming queries. It will be same as the append mode if a query has no aggregations.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixiong@databricks.com>
Closes#16520 from zsxwing/update-without-agg.
(cherry picked from commit bc6c56e)
Signed-off-by: Shixiong Zhu <shixiong@databricks.com>
@zsxwing
zsxwing deleted the update-without-agg branch January 11, 2017 06:12
uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
@zsxwing@uzadude
[SPARK-19140][SS] Allow update mode for non-aggregation streaming que…
b8ae6a2
…ries
## What changes were proposed in this pull request?
This PR allow update mode for non-aggregation streaming queries. It will be same as the append mode if a query has no aggregations.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixiong@databricks.com>
Closesapache#16520 from zsxwing/update-without-agg.
cmonkey pushed a commit to cmonkey/spark that referenced this pull request Feb 15, 2017
@zsxwing@cmonkey
[SPARK-19140][SS] Allow update mode for non-aggregation streaming que…
51a88f0
…ries
## What changes were proposed in this pull request?
This PR allow update mode for non-aggregation streaming queries. It will be same as the append mode if a query has no aggregations.
## How was this patch tested?
Jenkins
Author: Shixiong Zhu <shixiong@databricks.com>
Closesapache#16520 from zsxwing/update-without-agg.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@brkyvzbrkyvzbrkyvz left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zsxwing@SparkQA@brkyvz