[SPARK-19050][SS][Tests]Fix EventTimeWatermarkSuite 'delay in months and years handled correctly'

[zsxwing]

What changes were proposed in this pull request?

monthsSinceEpoch in this test is like math.floor(num), so monthDiff has two possible values.

How was this patch tested?

Jenkins.

[zsxwing]

cc @gatorsmile@tdas

[zsxwing]

I will merge this PR once this test passes since the master is broken now.

[zsxwing]

Merging to master and 2.1

[SparkQA]

Test build #70777 has finished for PR 16449 at commit c456c6e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

It sounds like our watermarkTime delay calculation causes this issue. Below are two typical cases:

Case 1: when setting the watermark delay to 1 month interval:

 .withWatermark("eventTime", "1 months")

the watermarket time is Thu Dec 01 20:05:34 PST 2016 and the current time is Sun Jan 01 20:05:34 PST 2017

Case 2: when setting the watermark delay to 1 month interval:

 .withWatermark("eventTime", "29 months")

the watermarket time is Thu Jul 17 21:10:50 PDT 2014 and the current time is Sun Jan 01 20:10:50 PST 2017

It sounds like it is caused by our intentional over-estimation (that is, by using 31 days per month)?

[zsxwing]

the watermarket time is Thu Jul 17 21:10:50 PDT 2014 and the current time is Sun Jan 01 20:10:50 PST 2017
It sounds like it is caused by our intentional over-estimation (that is, by using 31 days per month)?

@gatorsmile this is an expected behavior. It's intentional and it's correct as per the comment in Dataset.withWatermark,

the actual watermark used is only guaranteed to be at least delayThreshold behind the actual event time. In some cases we may still process records that arrive more than delayThreshold late.

If the user wants to filter data accurately, they need to use filter explicitly.

That's why I changed the test rather than the watermark calculation.

[gatorsmile]

Yeah, agree. This is a bug in the test cases. Thanks!