[BEAM-13972] Add RunInference interface by yeandy · Pull Request #16917 · apache/beam

yeandy · 2022-02-22T14:15:26Z

Adding common interface to the RunInference transform. Arguments will be wrapper classes around PyTorch and Scikit-learn models, along with other common parameters.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

codecov · 2022-02-22T14:37:50Z

Codecov Report

Merging #16917 (71e2f8c) into master (b2f2128) will increase coverage by 36.74%.
The diff coverage is n/a.

@@ Coverage Diff @@## master #16917 +/- ##
===========================================
+ Coverage 46.89% 83.63% +36.74% 
===========================================
Files 204 453 +249 Lines 20122 62528 +42406 ===========================================
+ Hits 9436 52295 +42859 - Misses 9686 10233 +547 + Partials 1000 0 -1000

Flag	Coverage Δ
python	`83.63% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
sdks/go/pkg/beam/runners/direct/impulse.go
sdks/go/pkg/beam/io/databaseio/writer.go
...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go
sdks/go/pkg/beam/core/runtime/exec/fn_arity.go
sdks/go/pkg/beam/beam.shims.go
sdks/go/pkg/beam/core/runtime/symbols.go
sdks/go/pkg/beam/util/shimx/generate.go
sdks/go/pkg/beam/runners/direct/buffer.go
sdks/go/pkg/beam/core/graph/coder/registry.go
sdks/go/pkg/beam/core/runtime/exec/cogbk.go
... and 647 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b2f2128...71e2f8c. Read the comment docs.

robertwb · 2022-02-23T19:16:44Z

+ raise NotImplementedError("Please implement _validate_model")
+
+
+class PytorchModel(RunInferenceModel):


It feels like PytorchModel and SklearnModel should be in their own file, only imported if one or the other is needed, rather than having a single module that holds all model types.

Agreed. I'll refactor.

Let's keep comments like this open until they're done for easier tracking.

These implementations will be done in separate PRs (pytorch, sklearn) after this PR, as well as the PR for the base class #16970 are merged.

yeandy · 2022-03-01T22:39:40Z

+_K = TypeVar('_K')
+_INPUT_TYPE = TypeVar('_INPUT_TYPE')
+_OUTPUT_TYPE = TypeVar('_OUTPUT_TYPE')
+
+
+@dataclass
+class PredictionResult:
+ key: _K
+ example: _INPUT_TYPE
+ inference: _OUTPUT_TYPE
+
+
+@beam.ptransform_fn
+@beam.typehints.with_input_types(Union[_INPUT_TYPE, Tuple[_K, _INPUT_TYPE]])
+@beam.typehints.with_output_types(PredictionResult)


How should we deal with typing? TFX explicitly defines output as Union[_OUTPUT_TYPE, Tuple[_K, _OUTPUT_TYPE]]

yeah, TFX is doing it right I think. They either have output type or a key with output type.
I'm not the typing expert, IMO its ok to have a typing specific PR that just concentrates on that too, if this gets messy.
We need typing to support TFX's proto use case as well as ours new frameworks use case.

I've modified the output type and pushed the change. I feel like we should try to get typing as accurate as possible early on, especially if we want to ensure that we can accommodate TFX's implementation.
Could we have, instead of a bare PredictionResult type, a parent class from which PredictionResult and prediction_log_pb2.PredictionLog inherit? Or use a Union? (but this may get messy).
This definitely requires input from someone with more experience with typing.

ryanthompson591

Looks good. Add valentyn next as reviewer IMO.

ryanthompson591 · 2022-03-02T17:14:20Z

+_K = TypeVar('_K')
+_INPUT_TYPE = TypeVar('_INPUT_TYPE')
+_OUTPUT_TYPE = TypeVar('_OUTPUT_TYPE')
+
+
+@dataclass
+class PredictionResult:
+ key: _K
+ example: _INPUT_TYPE
+ inference: _OUTPUT_TYPE
+
+
+@beam.ptransform_fn
+@beam.typehints.with_input_types(Union[_INPUT_TYPE, Tuple[_K, _INPUT_TYPE]])
+@beam.typehints.with_output_types(PredictionResult)


yeah, TFX is doing it right I think. They either have output type or a key with output type.
I'm not the typing expert, IMO its ok to have a typing specific PR that just concentrates on that too, if this gets messy.
We need typing to support TFX's proto use case as well as ours new frameworks use case.

yeandy · 2022-03-02T17:51:35Z

CC: @TheNeuralBit and @tvalentyn

yeandy · 2022-03-02T19:20:59Z

CC: @kevingg

yeandy · 2022-03-03T22:28:05Z

R: @robertwb @tvalentyn
CC: @kerrydc

ryanthompson591

Looks great!

ryanthompson591 · 2022-03-09T16:51:02Z

+ """
+ @abc.abstractmethod
+ def get_model_loader(self):
+ "Returns ModelLoader object"


I think this is good for now.
Just a note, but I don't think you need to do anything: Going forward it seems to me that there will be an implementation that uses a service (instead of loading model) a that maybe returns none for get_model_loader. I haven't decided if this should return None by default or if a NotImplementedError is fine.

yeandy · 2022-03-09T17:17:08Z

Updated the interface per our discussion, PTAL. @robertwb

aaltay · 2022-03-18T00:54:28Z

@robertwb - could you please review this PR?

Other folks on the PR, feel free to ping reviewers here or in other places if a review is open for more than a few business days. And where it makes sense you could also ask other committers to review PRs and spread the review load across committers.

yeandy · 2022-03-18T15:21:56Z

@tvalentyn could you please review/merge?

robertwb · 2022-03-18T21:21:08Z

I'm actively involved in the review within the TFX codebase. I'm assuming, however, that the two will be reconciled. (And, generally, +1 to @aaltay 's advice about pining reviews that are lagging.)

yeandy · 2022-03-18T21:28:18Z

Thanks. They will be reconciled once the change to TFX is finished.

tvalentyn · 2022-03-18T21:43:33Z

Sorry, I misunderstood @yeandy in offline conversation, I thought you already reviewed this change, @robertwb .

robertwb · 2022-03-18T22:37:25Z

No problem. This is a rapidly developing space.

…

On Fri, Mar 18, 2022 at 2:43 PM tvalentyn ***@***.***> wrote: Sorry, I misunderstood @yeandy <https://github.com/yeandy> in offline conversation, I thought you already reviewed this change, @robertwb <https://github.com/robertwb> . — Reply to this email directly, view it on GitHub <#16917 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADWVAP2U5IJHBLPBSIDTBLVAT2JBANCNFSM5PBQUWJQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

Add initial runinference interface template
f73b890

github-actionsBot added the python label Feb 22, 2022

AnandInguva reviewed Feb 22, 2022
View reviewed changes

Comment threadsdks/python/apache_beam/ml/inference/model.py Outdated
Comment threadsdks/python/apache_beam/ml/inference/run_inference.py Outdated

robertwb requested changes Feb 23, 2022
View reviewed changes

yeandy added 2 commits February 28, 2022 15:59

Refactor ModelSpec into separate files; Clean up interface
d75631a

Clean up api; Add model tests; Add typing
e802bfe

yeandy commented Mar 1, 2022
View reviewed changes

Comment threadsdks/python/apache_beam/ml/inference/api.py Outdated

Rename model spec
18a3db5

yeandy commented Mar 1, 2022
View reviewed changes

Comment threadsdks/python/apache_beam/ml/inference/api.py Outdated

ryanthompson591 reviewed Mar 2, 2022
View reviewed changes

Address PR comments
cf31b18

AnandInguva reviewed Mar 2, 2022
View reviewed changes

Comment threadsdks/python/apache_beam/ml/inference/api_test.py Outdated

Fix assert equals
4280dfc

ryanthompson591 reviewed Mar 3, 2022
View reviewed changes

Comment threadsdks/python/apache_beam/ml/inference/api.py Outdated
Comment threadsdks/python/apache_beam/ml/inference/api.py Outdated
Comment threadsdks/python/apache_beam/ml/inference/api.py Outdated

Redo API doc; Rename model_url
b0cea39

yeandy requested a review from robertwb March 3, 2022 22:23

ryanthompson591 reviewed Mar 4, 2022
View reviewed changes

Comment threadsdks/python/apache_beam/ml/inference/api.py Outdated

yeandy added 2 commits March 4, 2022 11:06

Fix linting errors
e5c18d9

Remove framework implementations; Make ModelSpec a factory
71e2f8c

ryanthompson591 approved these changes Mar 9, 2022
View reviewed changes

tvalentyn approved these changes Mar 18, 2022
View reviewed changes

tvalentyn merged commit 9a548b2 into apache:masterMar 18, 2022

		raise NotImplementedError("Please implement _validate_model")


		class PytorchModel(RunInferenceModel):

Uh oh!

Conversation

yeandy commented Feb 22, 2022

GitHub Actions Tests Status (on master branch)

Uh oh!

codecovBot commented Feb 22, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ryanthompson591 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yeandy commented Mar 2, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yeandy commented Mar 2, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yeandy commented Mar 3, 2022

Uh oh!

Uh oh!

ryanthompson591 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yeandy commented Mar 9, 2022

Uh oh!

aaltay commented Mar 18, 2022

Uh oh!

yeandy commented Mar 18, 2022

Uh oh!

robertwb commented Mar 18, 2022

Uh oh!

yeandy commented Mar 18, 2022

Uh oh!

tvalentyn commented Mar 18, 2022

Uh oh!

robertwb commented Mar 18, 2022 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecovBot commented Feb 22, 2022•
edited
Loading

yeandy commented Mar 2, 2022•
edited
Loading