Skip to content

[BEAM-10137][BEAM-10138] Python wrapper KinesisIO integration tests - #12298

Closed
pjotrekk wants to merge 2 commits into
apache:masterfrom
pjotrekk:BEAM-10137-python-wrapper-kinesis-read
Closed

[BEAM-10137][BEAM-10138] Python wrapper KinesisIO integration tests#12298
pjotrekk wants to merge 2 commits into
apache:masterfrom
pjotrekk:BEAM-10137-python-wrapper-kinesis-read

Conversation

@pjotrekk

@pjotrekkpjotrekk commented Jul 17, 2020

Copy link
Copy Markdown

Add python wrapper for KinesisIO.Read.

Chronology:

  1. java write
  2. java read
  3. python wrapper
  4. python integration tests

PR for java external transform for write: #12234
PR for python wrapper: #12235
PR for java external transform for read: #12297

I haven't managed to run this test yet. In the previous version (1st commit) it succeeded on my AWS account. But when I try to run the same pipelines on localstack I get com.amazonaws.services.kinesis.producer.UserRecordFailedException.

Strange thing is that with below code I can use the container, so it works.

 client.put_record(
StreamName='beam',
Data=b'bytes',
PartitionKey='a',
)
shard_iterator = client.get_shard_iterator(
StreamName=self.stream_name,
ShardId='shardId-000000000000',
ShardIteratorType='TRIM_HORIZON'
)
response = client.get_records(
ShardIterator=shard_iterator['ShardIterator'],
Limit=2
)

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

LangSDKDataflowFlinkSamzaSparkTwister2
GoBuild Status---Build Status---Build Status---
JavaBuild StatusBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build StatusBuild Status
Build Status
Build Status
Build Status
PythonBuild Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
---Build Status---
XLangBuild Status---Build Status---Build Status---

Pre-Commit Tests Status (on master branch)

---JavaPythonGoWebsite
Non-portableBuild StatusBuild Status
Build Status
Build StatusBuild Status
Portable---Build Status------

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

@pjotrekk
pjotrekkforce-pushed the BEAM-10137-python-wrapper-kinesis-read branch 9 times, most recently from 9804607 to 0f73856CompareJuly 22, 2020 12:29
@pjotrekk

Copy link
Copy Markdown
Author

R: @TheNeuralBit Could I ask you for review?

@pjotrekk
pjotrekkforce-pushed the BEAM-10137-python-wrapper-kinesis-read branch from 65b7563 to e4bfe7cCompareJuly 24, 2020 13:17
@pjotrekkpjotrekk changed the title [BEAM-10137] Python wrapper for KinesisIO.Read with integration test[BEAM-10137] Python wrapper KinesisIO integration testsJul 24, 2020
@pjotrekkpjotrekk changed the title [BEAM-10137] Python wrapper KinesisIO integration tests[BEAM-10137][BEAM-10138] Python wrapper KinesisIO integration testsJul 24, 2020
@TheNeuralBit
TheNeuralBit self-requested a review July 25, 2020 00:20
@TheNeuralBit

Copy link
Copy Markdown
Member

Closing in favor of #12297

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@pjotrekk@TheNeuralBit