[SPARK-11198][STREAMING][KINESIS] Support de-aggregation of records during recovery by brkyvz · Pull Request #9403 · apache/spark

brkyvz · 2015-11-02T01:21:22Z

While the KCL handles de-aggregation during the regular operation, during recovery we use the lower level api, and therefore need to de-aggregate the records.

@tdas Testing is an issue, we need protobuf magic to do the aggregated records. Maybe we could depend on KPL for tests?

SparkQA · 2015-11-02T02:01:08Z

Test build #44784 has finished for PR 9403 at commit de3e91b.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):\n * case class RepartitionByExpression(\n

brkyvz · 2015-11-02T18:31:15Z

Tested that this patch successfully de-aggregates in recovery as well.

SparkQA · 2015-11-03T18:36:49Z

Test build #44941 has finished for PR 9403 at commit 6b0b290.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-11-03T21:08:18Z

Test build #44944 has finished for PR 9403 at commit 023055c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-11-03T22:28:59Z

Test build #44956 has finished for PR 9403 at commit 5f7a763.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-11-04T02:23:07Z

Test build #44975 has finished for PR 9403 at commit 7c1edd7.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-11-04T06:59:07Z

test this again.

tdas · 2015-11-04T07:48:57Z

Isn it easier to have a parameter in the constructor? Less code while subclassing class WithAggregationKinesisStreamSuite extends KinesisStreamTests(aggregateTestData = true)

That can also work :)

tdas · 2015-11-04T08:27:29Z

overall the code looks fine, but Kinesis tests are not passing

SparkQA · 2015-11-05T00:16:26Z

Test build #45049 has finished for PR 9403 at commit 83a0f13.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-11-06T00:19:55Z

Test build #45149 has finished for PR 9403 at commit 8bcd6c3.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-11-06T01:20:05Z

test this please

zsxwing · 2015-11-06T01:22:36Z

The failure may be because protobuf 2.6.1 was not enabled. BTW, if 2.6.1 is enabled, shouldn't protobuf 2.6.1 break other stuff in Spark and make the end-to-end test fail?

brkyvz · 2015-11-06T01:26:44Z

@zsxwing probably not. Locally tests pass. They fail on jenkins for some reason. I've enabled a lot of the logging to look deeper into it.

SparkQA · 2015-11-06T03:35:41Z

Test build #45181 has finished for PR 9403 at commit 4acf47e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-11-06T20:33:58Z

Test build #45249 has started for PR 9403 at commit 66fade4.

brkyvz · 2015-11-06T21:12:04Z

test this please

SparkQA · 2015-11-06T23:20:25Z

Test build #45255 has finished for PR 9403 at commit 66fade4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-11-09T07:54:04Z

Test build #45343 has finished for PR 9403 at commit 2ef1717.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-11-09T16:45:03Z

Huh. Didn't change much but the tests passed this time. I wonder if it was a Java 7 vs. 8 mismatch... Just to be sure, will re-run tests

brkyvz · 2015-11-09T16:45:08Z

test this please

SparkQA · 2015-11-09T18:43:03Z

Test build #45377 has finished for PR 9403 at commit 2ef1717.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-11-09T18:46:23Z

Aww man. The Kinesis tests passed, hive tests failed. Re-running

brkyvz · 2015-11-09T18:46:27Z

test this please

SparkQA · 2015-11-09T21:11:21Z

Test build #45391 has finished for PR 9403 at commit 2ef1717.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

brkyvz · 2015-11-09T21:50:03Z

test this please

brkyvz · 2015-11-09T23:25:42Z

test this please

SparkQA · 2015-11-10T00:10:44Z

Test build #45422 has finished for PR 9403 at commit 932485a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tdas · 2015-11-10T01:15:48Z

LGTM. Merging this to master and 1.6. Thanks @brkyvz

…uring recovery While the KCL handles de-aggregation during the regular operation, during recovery we use the lower level api, and therefore need to de-aggregate the records. tdas Testing is an issue, we need protobuf magic to do the aggregated records. Maybe we could depend on KPL for tests? Author: Burak Yavuz <brkyvz@gmail.com> Closes#9403 from brkyvz/kinesis-deaggregation. (cherry picked from commit 26062d2) Signed-off-by: Tathagata Das <tathagata.das1565@gmail.com>

SparkQA · 2015-11-10T01:48:39Z

Test build #45440 has finished for PR 9403 at commit 932485a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

lordnynex · 2015-11-17T23:59:42Z

Despite being closed, I'm confused about why this causes hive unit tests to fail. Is this unrelated?Also, the publisher in the unit test looks like it's publishing an intentionally non-aggregated stream. Is that expected? Am I reading this incorrectly?

brkyvz · 2015-11-18T00:49:46Z

Hi @lordnynex. Why did you think this causes Hive tests to fail? That was most probably a flaky test.
Regarding publishing a non-aggregated stream, yes that is correct. Not everyone uses KPL, therefore we need to make sure we support both non-aggregated and aggregated streams :)

zsxwing · 2015-11-30T21:47:40Z

KinesisStreamTests in test.py is broken because of this PR. See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46896/testReport/(root)/KinesisStreamTests/test_kinesis_stream/

Is the new dependency the failure cause?

The PR builds for this PR actually didn't report the Python failure because of #9669.

KinesisStreamTests in test.py is broken because of #9403. See https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46896/testReport/(root)/KinesisStreamTests/test_kinesis_stream/ Because Streaming Python didn’t work when merging #9403, the PR build didn’t report the Python test failure actually. This PR just disabled the test to unblock #10039 Author: Shixiong Zhu <shixiong@databricks.com> Closes#10047 from zsxwing/disable-python-kinesis-test.

Support de-aggregation during recovery
de3e91b

Added unit tests
6b0b290

brkyvz added 3 commits November 3, 2015 11:04

moved KinesisTestUtils
fed0785

really moved
023055c

remove custom endpoint
5f7a763

turn off metrics
7c1edd7

tdas reviewed Nov 4, 2015
View reviewed changes

tested locally
83a0f13

add logging for test
8bcd6c3

changed debug's to infos
4acf47e

reverting change just to make sure
66fade4

try new fix
2ef1717

test change
932485a

asfgit closed this in 26062d2Nov 10, 2015

zsxwing mentioned this pull request Nov 30, 2015
[SPARK-12058][Hotfix]Disable KinesisStreamTests #10047
Closed

brkyvz deleted the kinesis-deaggregation branch February 3, 2019 20:57

Uh oh!

Conversation

brkyvz commented Nov 2, 2015

Uh oh!

SparkQA commented Nov 2, 2015

Uh oh!

brkyvz commented Nov 2, 2015

Uh oh!

SparkQA commented Nov 3, 2015

Uh oh!

SparkQA commented Nov 3, 2015

Uh oh!

SparkQA commented Nov 3, 2015

Uh oh!

SparkQA commented Nov 4, 2015

Uh oh!

tdas commented Nov 4, 2015

Uh oh!

tdasNov 4, 2015

Choose a reason for hiding this comment

Uh oh!

brkyvzNov 4, 2015

Choose a reason for hiding this comment

Uh oh!

tdas commented Nov 4, 2015

Uh oh!

SparkQA commented Nov 5, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

brkyvz commented Nov 6, 2015

Uh oh!

zsxwing commented Nov 6, 2015

Uh oh!

brkyvz commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

brkyvz commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 6, 2015

Uh oh!

SparkQA commented Nov 9, 2015

Uh oh!

brkyvz commented Nov 9, 2015

Uh oh!

brkyvz commented Nov 9, 2015

Uh oh!

SparkQA commented Nov 9, 2015

Uh oh!

brkyvz commented Nov 9, 2015

Uh oh!

brkyvz commented Nov 9, 2015

Uh oh!

SparkQA commented Nov 9, 2015

Uh oh!

brkyvz commented Nov 9, 2015

Uh oh!

brkyvz commented Nov 9, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

tdas commented Nov 10, 2015

Uh oh!

SparkQA commented Nov 10, 2015

Uh oh!

lordnynex commented Nov 17, 2015

Uh oh!

brkyvz commented Nov 18, 2015

Uh oh!

zsxwing commented Nov 30, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development