Skip to content

[BEAM-52] Kafka IO - #142

Closed
dhalperi wants to merge 2 commits into
apache:masterfrom
dhalperi:kafka-io
Closed

[BEAM-52] Kafka IO#142
dhalperi wants to merge 2 commits into
apache:masterfrom
dhalperi:kafka-io

Conversation

@dhalperi

@dhalperidhalperi commented Apr 7, 2016•
edited
Loading

Copy link
Copy Markdown
Contributor

Be sure to do all of the following to help us incorporate your contribution
quickly and easily:

  • Make sure the PR title is formatted like:
    [BEAM-<Jira issue #>] Description of pull request
  • Make sure tests pass via mvn clean verify. (Even better, enable
    Travis-CI on your fork and ensure the whole test matrix passes).
  • Replace "<Jira issue #>" in the title with the actual Jira issue
    number, if there is one.
  • If this contribution is large, please file an Apache
    Individual Contributor License Agreement.

TODO: @rangadi@dhalperi

R: @dpmills@dhalperi

@dhalperi

dhalperi commented Apr 7, 2016

Copy link
Copy Markdown
ContributorAuthor

Forward-porting GoogleCloudPlatform/DataflowJavaSDK#121. Note that the original author is a Googler (hence covered by CLA) and is still attributed to the actual work.

For more information, see the considerable review in the original PR.

I fully squashed the original PR and fixed some licenses. More work still in progress.

@dhalperidhalperi mentioned this pull request Apr 7, 2016
Closed
3 tasks
jbonofre
jbonofre reviewed Apr 7, 2016
View reviewed changes
Comment threadcontrib/examples/kafka/pom.xml Outdated

<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-contrib-kafka-examples</artifactId>
<name>Google Cloud Dataflow Kafka Examples</name>

@jbonofrejbonofreApr 7, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can change to use org.apache.beam Maven coordonates ?

@jbonofrejbonofreApr 7, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, aligned with the other IOs and current discussion, we can move this in the sdks/java/io module later.

@rangadirangadiApr 8, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

@jbonofre

jbonofre commented Apr 7, 2016

Copy link
Copy Markdown
Member

Just couple of small comments (about Maven coordonates and module). Great work anyway !

tgroh pushed a commit to tgroh/beam that referenced this pull request Apr 7, 2016
@davorbonaci
Merge pull requestapache#142from davorbonaci/backport-20
6f2fc72
Backport apache#20
@dhalperi

dhalperi commented Apr 8, 2016

Copy link
Copy Markdown
ContributorAuthor

Thanks @jbonofre ! We'll take these comments into account as we do the cleanups in the TODO list in the PR description.

@rangadi

rangadi commented Apr 12, 2016

Copy link
Copy Markdown
Contributor

Looking in to Travis failure.

rangadi
rangadi reviewed Apr 21, 2016
View reviewed changes
Comment thread.travis.yml Outdated
script:
- travis_retry mvn -B $MAVEN_OVERRIDE install -U
- travis_retry travis/test_wordcount.sh
- (cd contrib/kafka && mvn -B $MAVEN_OVERRIDE install -U)

@rangadirangadiApr 21, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to test this in travis?

@rangadi

rangadi commented Apr 21, 2016

Copy link
Copy Markdown
Contributor

@dhalperi , I made the package renaming changes this morning, only to realize you have already done that..

@rangadi

rangadi commented Apr 21, 2016

Copy link
Copy Markdown
Contributor

Thanks Dan! Glad to see KafkaIO is moved out of contrib.

Raghu Angadiand others added 2 commits April 21, 2016 16:13
Raghu Angadi@dhalperi
KafkaIO: unbounded source for reading from Apache Kafka
f5c809d
@dhalperi
Kafka: various fixes
1d20663
*) package renaming
*) move it out of contrib to I/O module
*) get it running as part of the normal travis build,
and remove specialization
*) remove examples, to be added in a separate PR.
Either there should be an addition to an existing example to
handle "multiple" streaming sources, or we should have this as an
integration test. No need, however, for an example just to show
how to use an I/O in a pipeline -- that's what Javadoc is for.
@dhalperi

dhalperi commented Apr 21, 2016

Copy link
Copy Markdown
ContributorAuthor

@rangadi can you give a look and an LGTM? Then I will merge.

Later, we should add an integration test and figure out what to do with the examples. (See comment on 1d20663)

@rangadi

rangadi commented Apr 21, 2016

Copy link
Copy Markdown
Contributor

LGTM @dhalperi. Thanks for making the changes.

I agree with your comment on reevaluating if and where example app fits in.

@dhalperi
dhalperi deleted the kafka-io branch April 22, 2016 00:38
iemejia referenced this pull request in iemejia/beam Jan 12, 2018
@davorbonaci
Thiscloses#142
a53de48
mareksimunek pushed a commit to mareksimunek/beam that referenced this pull request May 9, 2018
@xitep
Merge pull requestapache#142from seznam/pete/spark-task-context
76521b8
#! [euphoria-spark] Correctly transfer global state to executed UDFs
FuRyanf pushed a commit to FuRyanf/beam that referenced this pull request Feb 18, 2025
@yananhao12
Merge pull requestapache#142from yananhao12/remove_flink_options_co…
a6917b0
…nfigmap
remove flinkConfMap from FlinkPipelineOptions
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dhalperi@jbonofre@rangadi