Skip to content

[SPARK-21590][SS]Window start time should support negative values - #18903

Closed
KevinZwx wants to merge 4 commits into
apache:masterfrom
KevinZwx:dev
Closed

[SPARK-21590][SS]Window start time should support negative values#18903
KevinZwx wants to merge 4 commits into
apache:masterfrom
KevinZwx:dev

Conversation

@KevinZwx

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Remove the non-negative checks of window start time to make window support negative start time, and add a check to guarantee the absolute value of start time is less than slide duration.

How was this patch tested?

New unit tests.

@KevinZwx

KevinZwx commented Aug 11, 2017

Copy link
Copy Markdown
ContributorAuthor

@zsxwing@brkyvz Can you review this patch?

@brkyvz

Copy link
Copy Markdown
Contributor

ok to test

}
}

test("SPARK-21590: Start time works with negative values and return microseconds") {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you also add a DataFrame test for this with a negative value? then I'll feel a lot more comfortable that we don't have to have the start offset as a positive number in the window calculation.
Look for DataFrameTimeWindowingSuite.scala

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah thanks, I added some tests in DataFrameTimeWindowingSuite

@brkyvz

Copy link
Copy Markdown
Contributor

test this please

@KevinZwx

Copy link
Copy Markdown
ContributorAuthor

test this please

@SparkQA

Copy link
Copy Markdown

Test build #4154 has finished for PR 18903 at commit 07e98e7.

  • This patch fails PySpark unit tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA

Copy link
Copy Markdown

Test build #4175 has finished for PR 18903 at commit 07e98e7.

  • This patch fails PySpark unit tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA

Copy link
Copy Markdown

Test build #4178 has finished for PR 18903 at commit 07e98e7.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@kiszk

Copy link
Copy Markdown
Member

retest this please

@HyukjinKwon

Copy link
Copy Markdown
Member

ok to test

@SparkQA

Copy link
Copy Markdown

Test build #93048 has finished for PR 18903 at commit 07e98e7.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen

Copy link
Copy Markdown
Member

Merged to master

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@KevinZwx@brkyvz@SparkQA@kiszk@HyukjinKwon@srowen