Skip to content

[BEAM-2852] Add support for Kafka as source/sink on Nexmark - #5019

Merged
echauchot merged 3 commits into
apache:masterfrom
aromanenko-dev:BEAM-2852-nexmark-kafka-source-sink
Apr 10, 2018
Merged

[BEAM-2852] Add support for Kafka as source/sink on Nexmark#5019
echauchot merged 3 commits into
apache:masterfrom
aromanenko-dev:BEAM-2852-nexmark-kafka-source-sink

Conversation

@aromanenko-dev

@aromanenko-devaromanenko-dev commented Apr 4, 2018

Copy link
Copy Markdown
Contributor

Allows to use Kafka as as source/sink for Nexmark benchmark.
Based on original implementation of #3937 by @vectorijk

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue.
  • Write a pull request description that is detailed enough to understand:
    • What the pull request does
    • Why it does it
    • How it does it
    • Why this approach
  • Each commit in the pull request should have a meaningful subject line and body.
  • Run mvn clean verify to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

@aromanenko-dev

aromanenko-dev commented Apr 4, 2018

Copy link
Copy Markdown
ContributorAuthor

R: @echauchot

@echauchot
echauchot self-requested a review April 4, 2018 13:59
@echauchot

echauchot commented Apr 4, 2018

Copy link
Copy Markdown
Contributor

Thanks @aromanenko-dev !

@echauchot

echauchot commented Apr 4, 2018

Copy link
Copy Markdown
Contributor

Thanks @aromanenko-dev !
Please reword commit to include ticket number and meaningfull title, beware with authorship

echauchot
echauchot requested changes Apr 4, 2018
View reviewed changes

@echauchotechauchot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!
some small changes + one new small feature + include my PR

Comment threadsdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
/**
* Send {@code events} to Kafka.
*/
private void sinkEventsToKafka(PCollection<Event> events) {

@echauchotechauchotApr 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please wire this method up and add a COMBINED mode similar to what is done in Pub/Sub?

@echauchotechauchotApr 5, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMHO I think we should refactor the whole COMBINED mode:

  • See NexmarkLauncher#createSource: it does a switch on the source type to configure sink.
  • it sends synthetic events to sink when in COMBINED mode but NexmarkUtils#COMBINED states that combine modes is for "Both publish and consume, but as separate jobs".
    Once refactored to something more coherent, implement it for kafka.

@echauchotechauchotApr 5, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is important in the connection with MOM:

  • have the ability to keep a track of the generated events that lead to a benchmark result
  • be able to read events from a topic
  • write benchmark results to topic

Comment threadsdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java Outdated
* Send {@code events} to Kafka.
*/
private void sinkEventsToKafka(PCollection<Event> events) {
PTransform<PCollection<byte[]>, PDone> io = KafkaIO.<Long, byte[]>write()

@echauchotechauchotApr 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a comment that I had on the previous PR. IMHO it is very wired code: it explicitely uses PTransform in place of Write transform and also it specifies a key and the associated coder whereas there is no key in the input PCollection. To be quicker I submited a PR to your repo so that you could include the fix in that PR branch. See aromanenko-dev#2

Comment threadsdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java Outdated
throw new RuntimeException("Missing --bootstrapServers");
}

KafkaIO.Read<Long, byte[]> io = KafkaIO.<Long, byte[]>read()

@echauchotechauchotApr 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rename io to read

Comment threadsdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java Outdated
private PCollection<Event> sourceEventsFromKafka(Pipeline p) {
NexmarkUtils.console("Reading events from Kafka Topic %s", options.getKafkaSourceTopic());

if (Strings.isNullOrEmpty(options.getBootstrapServers())) {

@echauchotechauchotApr 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use checkArgument

Comment threadsdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java Outdated
checkArgument(!Strings.isNullOrEmpty(options.getBootstrapServers()),
"Missing --bootstrapServers");

PTransform<PCollection<String>, PDone> io = KafkaIO.<Long, String>write()

@echauchotechauchotApr 4, 2018

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same comment as above See my PR #2 on your repo

@aromanenko-dev
aromanenko-devforce-pushed the BEAM-2852-nexmark-kafka-source-sink branch 2 times, most recently from 2e744e5 to 8f7d724CompareApril 5, 2018 09:59
@echauchot

echauchot commented Apr 10, 2018

Copy link
Copy Markdown
Contributor

@aromanenko-dev please update gradle build to reflect maven one, squash this commit with code style one and run "Run Java PreCommit" to launch gradle build. LGTM at green lights of gradle build

@aromanenko-dev
aromanenko-devforce-pushed the BEAM-2852-nexmark-kafka-source-sink branch from ca3436c to 6df0addCompareApril 10, 2018 13:27
@aromanenko-dev

aromanenko-dev commented Apr 10, 2018

Copy link
Copy Markdown
ContributorAuthor

Run Java PreCommit

@echauchot
echauchot merged commit 506fb32 into apache:masterApr 10, 2018
@echauchot

echauchot commented Apr 10, 2018•
edited
Loading

Copy link
Copy Markdown
Contributor

Merging this PR, we will takle COMBINED mode (refactoring this mode in pub/sub and apply it to kafka) in another PR. I opened a ticket for that: https://issues.apache.org/jira/browse/BEAM-4048

@aromanenko-dev
aromanenko-dev deleted the BEAM-2852-nexmark-kafka-source-sink branch April 10, 2018 14:27
@vectorijk

vectorijk commented Apr 23, 2018

Copy link
Copy Markdown
Contributor

Thanks! @aromanenko-dev

@aromanenko-dev

aromanenko-dev commented Apr 24, 2018

Copy link
Copy Markdown
ContributorAuthor

@vectorijk Thank you for your initial work!

@scwhittlescwhittle mentioned this pull request Dec 3, 2020
Merged
4 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@echauchotechauchotechauchot approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@aromanenko-dev@echauchot@vectorijk