Skip to content

[BEAM-10138] Python wrapper for Cross-language Java's KinesisIO - #12235

Closed
pjotrekk wants to merge 1 commit into
apache:masterfrom
pjotrekk:BEAM-10138-write-python-wrapper
Closed

[BEAM-10138] Python wrapper for Cross-language Java's KinesisIO#12235
pjotrekk wants to merge 1 commit into
apache:masterfrom
pjotrekk:BEAM-10138-write-python-wrapper

Conversation

@pjotrekk

@pjotrekkpjotrekk commented Jul 13, 2020

Copy link
Copy Markdown

Add a python wrapper for KinesisIO.Write.

Chronology:

  1. java write
  2. java read
  3. python wrapper
  4. python integration tests

PR for java external transform for write: #12234
PR for java external transform for read: #12297
PR for python integration tests: #12298

This test cannot be run on Jenkins because Beam does not have an AWS account. It has been tested locally on DirectRunner, FlinkRunner and SparkRunner.

R: @TheNeuralBit The same as with Java KinesisIO.Write external registrar - if you wanted to do this review I'd appreciate it :)


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

LangSDKDataflowFlinkSamzaSparkTwister2
GoBuild Status---Build Status---Build Status---
JavaBuild StatusBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild Status
Build Status
Build Status
Build Status
PythonBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
---Build Status---
XLang------Build Status---Build Status---

Pre-Commit Tests Status (on master branch)

---JavaPythonGoWebsite
Non-portableBuild StatusBuild Status
Build Status
Build StatusBuild Status
Portable---Build Status------

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@TheNeuralBit

Copy link
Copy Markdown
Member

Sorry for the delay @piotr-szuberski! I'll try to take a look at these today and/or tomorrow.

One high-level question - do we have a jira about not running KinesisIO tests continuously? If not, please file one and make sure it mentions both direct Java tests as well as xlang tests.

@pjotrekk
pjotrekkforce-pushed the BEAM-10138-write-python-wrapper branch from c7fe25f to 9842ff1CompareJuly 23, 2020 08:13
@pjotrekk

pjotrekk commented Jul 23, 2020

Copy link
Copy Markdown
Author

Sorry for the delay @piotr-szuberski! I'll try to take a look at these today and/or tomorrow.

No problem at all!

One high-level question - do we have a jira about not running KinesisIO tests continuously? If not, please file one and make sure it mentions both direct Java tests as well as xlang tests.

Yes, there is BEAM-601 - I've added a comment there and I'll mention it in the integration tests comment

@pjotrekk

pjotrekk commented Jul 23, 2020

Copy link
Copy Markdown
Author

@TheNeuralBit I've just figured out that I could use kinesis SDK which is already included in AWS_REQUIREMENTS (boto3) - It will probably make it possible to separate the read and write test.
EDIT: I followed the suggestion from Jira to use localstack with testcontainers.

@pjotrekk
pjotrekkforce-pushed the BEAM-10138-write-python-wrapper branch from af235cb to 4afc7dfCompareJuly 24, 2020 12:55
@pjotrekkpjotrekk changed the title [BEAM-10138] Python wrapper for Java's KinesisIO.Write with integration test[BEAM-10138] Python wrapper for Cross-language Java's KinesisIOJul 24, 2020
@pjotrekk
pjotrekkforce-pushed the BEAM-10138-write-python-wrapper branch from 4afc7df to 66fb7ceCompareJuly 24, 2020 13:18
@TheNeuralBit

Copy link
Copy Markdown
Member

Closing in favor of #12297

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@pjotrekk@TheNeuralBit