[BEAM-11254] New documentation on multi-language pipelines by davidwrede · Pull Request #13317 · apache/beam

davidwrede · 2020-11-12T13:35:36Z

Documentation on how to create and consume cross-language transforms in a multi-language pipeline.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang	---	---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

chamikaramj

Thanks Dave.

I think this is in a great state. My comments are mostly nits.

Also seems like there are some test failures that have to be addressed.

robertwb

Thanks, it'll be great to have this documented!

robertwb · 2020-11-13T23:33:43Z

Is the double cast needed? Perhaps it would be simpler to add the type parameters to ImmutableMap.of, i.e. ImmutableMap.of<String, Class<? extends ExternalTransformBuilder<?, ?, ?>>>(...).

Hmm, I tried the suggestion and it doesn't seem to be working. Note that this snippet is from actual class: https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L632

OK, it's a bit ugly we can leave it as is for now.

robertwb · 2020-11-13T23:42:09Z

Why not setTopics(config.topics) or, if a copy is needed, setTopics(ImmutableList.copyOf(config.topics))
(FWIW, the motivation to have a separate Builder and Configuration object that seem to largely play the same role is a bit unclear here--it'd be good to clarify that.)

robertwb · 2020-11-13T23:59:55Z

Note that this will only serve those transforms living in modules that have been imported...

Added a point to import modules.

robertwb · 2020-11-14T00:02:16Z

How do you specify the custom container?

I don't think we have fully implemented dependencies support for Python external transforms yet so we are limited to dependencies available with core Beam SDK Harness. Updated text to reflect this.

robertwb · 2020-11-14T00:04:49Z

self.expansion_service?

davidwrede · 2020-11-14T00:48:47Z

@robertwb I implemented some of your changes, but unfortunately, someone else will have to address the rest.

I'm happy to help merge in the PR as is, or hand it off to someone else.

robertwb · 2020-11-17T20:52:35Z

Thanks, @davidwrede! We'll take it from here.

chamikaramj

Thanks Robert. PTAL.

chamikaramj · 2020-11-17T22:07:56Z

Hmm, I tried the suggestion and it doesn't seem to be working. Note that this snippet is from actual class: https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L632

chamikaramj · 2020-11-17T22:19:45Z

chamikaramj · 2020-11-17T22:23:43Z

Added a point to import modules.

chamikaramj · 2020-11-17T22:28:19Z

I don't think we have fully implemented dependencies support for Python external transforms yet so we are limited to dependencies available with core Beam SDK Harness. Updated text to reflect this.

chamikaramj · 2020-11-17T22:40:15Z

chamikaramj · 2020-11-17T22:44:12Z

We have it there as well. See "Using an SDK wrapper". Primary audience of this sub-section are transforms authors. So information here is more on writing wrappers. There's some overlap but I think this is fine.

chamikaramj · 2020-11-17T22:58:12Z

chamikaramj · 2020-11-18T05:09:01Z

Updated to include information about both JavaJarExpansionService and BeamJarExpansionService.

chamikaramj · 2020-11-19T22:41:38Z

Also cc: @rosetn in case docs team have more comments here.

robertwb

Thanks, this looks good to me.

robertwb · 2020-11-20T20:27:06Z

OK, it's a bit ugly we can leave it as is for now.

chamikaramj · 2020-11-20T22:52:23Z

Thanks!

chamikaramj changed the title ~~New documentation on multi-language pipelines~~[BEAM-11254] New documentation on multi-language pipelinesNov 12, 2020

TheNeuralBit reviewed Nov 12, 2020
View reviewed changes

Comment threadwebsite/www/site/content/en/documentation/programming-guide.md Outdated

TheNeuralBit mentioned this pull request Nov 12, 2020
[BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper #12611
Merged
4 tasks

chamikaramj reviewed Nov 13, 2020
View reviewed changes

robertwb reviewed Nov 14, 2020
View reviewed changes

chamikaramj reviewed Nov 18, 2020
View reviewed changes

davidwredeand others added 6 commits November 19, 2020 16:00

New documentation on multi-language pipelines
229ccbb

Incorporated TR feedback
6c6c161

Fixed whitespace lint test failure
dafa9e6

Add license info to SVG
761de83

Addressed some of the new feedback
8c926fa

Addresses reviewer comments
a208491

chamikaramj force-pushed the multi-language-pipelines branch from f150e63 to a208491Compare November 20, 2020 00:01

robertwb approved these changes Nov 20, 2020
View reviewed changes

chamikaramj merged commit 626551e into apache:masterNov 20, 2020

davidwrede deleted the multi-language-pipelines branch December 4, 2020 18:57

damccorm mentioned this pull request Jun 4, 2022
Remove duplicate information in Python xlang module docstrings #20592
Open

Uh oh!

Conversation

davidwrede commented Nov 12, 2020• edited by chamikaramj Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

robertwb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidwrede commented Nov 14, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertwb commented Nov 17, 2020

Uh oh!

chamikaramj left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chamikaramj commented Nov 19, 2020

Uh oh!

robertwb left a comment

Choose a reason for hiding this comment

davidwrede commented Nov 12, 2020•
edited by chamikaramj
Loading

davidwrede commented Nov 14, 2020•
edited
Loading