Skip to content

[BEAM-8740] Remove unused dependency from Spark runner - #10564

Merged
lukecwik merged 2 commits into
apache:masterfrom
iemejia:spark-clean-dep
Jan 13, 2020
Merged

[BEAM-8740] Remove unused dependency from Spark runner#10564
lukecwik merged 2 commits into
apache:masterfrom
iemejia:spark-clean-dep

Conversation

@iemejia

Copy link
Copy Markdown
Member

@iemejia

Copy link
Copy Markdown
MemberAuthor

Run Spark ValidatesRunner

@iemejia

Copy link
Copy Markdown
MemberAuthor

Run Spark StructuredStreaming ValidatesRunner

ValidatesRunner.class,
UsesStatefulParDo.class,
UsesTimersInParDo.class,
UsesTestStream.class,

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to add this one because it was not correctly categorized and VR were breaking

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lukecwiklukecwikJan 13, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't the VR tests exclude UsesTestStreamWithOutputTimestamp?

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's the line exactly below that one. Notice that Spark runner does not implement TestStream so we probably need to make the test belong to both categories.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mind to maybe lgtm / merge this one @lukecwik. Seems most VR are broken because of it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see, I was under the impression that we would place things in the most appropriate category but now thinking about it makes sense to place it in this category as well.

@lukecwik
lukecwik merged commit 04498d5 into apache:masterJan 13, 2020
@iemejia
iemejia deleted the spark-clean-dep branch January 24, 2020 16:08
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@iemejia@lukecwik