The Problem
Currently it is impossible make prediction from images if they do not come from file. PredictionEngine<IT,OT> and IEnumerable to IDV is impossible if T contains images/
The Root Cause
The fundamental issue is that in several places in our system, our types are hardcoded and limited to a small set. Actual IDataView type system IS open, so it should be possible to support any type. However, some of our machinery is not so extensible. For example, if we need to add full support of images, we would need to update:
- converters between IEnumerable <-> IDataView,
- PredictionEngine<>
- API Utils,
- Schema utils
- etc.
What we need is a dependency-injection mechanism to allow component developers to inject definitions of types supported by our IDV machinery. Such mechanism should be flexible and extensible. For an example of hard-coded implementation, check #3263. It is illustrative but incorrect as it is not flexible. Note @TomFinley's comment on different options on how to do this properly. We need to define a mechanism that is extensible, so that a dev can add new transforms to work on new data types, such as sound and enjoy all benefits of ML.NET
Definition of Done:
- Mechanism to register additional IDV types so that they are supported by PredictionEngine and IEnumerable converters.
- Add such registration for images and date types
- Ensure that the registration (at least for images) occurs automatically whenever appropriate component is used (for example, if image resizing transform is used, the image type is registered automatically )
- Ensure that prediction engine, conversion to and from IEnumerable work correctly with tests.
The following issues will also be solved by this issue:
#3369, #3460, #3582, #2121, #2495, #3582, #3274
In addition, this work is needed to properly support ONNX and TF scenarios for structured data such as image, speech, video, or Audio.
The Problem
Currently it is impossible make prediction from images if they do not come from file. PredictionEngine<IT,OT> and IEnumerable to IDV is impossible if T contains images/
The Root Cause
The fundamental issue is that in several places in our system, our types are hardcoded and limited to a small set. Actual IDataView type system IS open, so it should be possible to support any type. However, some of our machinery is not so extensible. For example, if we need to add full support of images, we would need to update:
What we need is a dependency-injection mechanism to allow component developers to inject definitions of types supported by our IDV machinery. Such mechanism should be flexible and extensible. For an example of hard-coded implementation, check #3263. It is illustrative but incorrect as it is not flexible. Note @TomFinley's comment on different options on how to do this properly. We need to define a mechanism that is extensible, so that a dev can add new transforms to work on new data types, such as sound and enjoy all benefits of ML.NET
Definition of Done:
The following issues will also be solved by this issue:
#3369, #3460, #3582, #2121, #2495, #3582, #3274
In addition, this work is needed to properly support ONNX and TF scenarios for structured data such as image, speech, video, or Audio.