[SPARK-16431] [ML] Add a unified method that accepts single instances to feature transformers and predictors

[hazimehh]

Summary

Add a new transformation method transformInstance that operates on single instances. This method can reduce the latency of predictions by 200x for typical ML tasks which facilitates serving models in production. See the JIRA ticket for details.

What changes were proposed in this pull request?

Current feature transformers in spark.ml can only operate on DataFrames and don't have a method that accepts single instances. A typical transformer has a User-Defined Function (udf) in its transform method which includes a set of operations on the features of a single instance:

val column_operation = udf {operations on single instance}

Adding a new method called transformInstance that operates directly on single instances and using it in the udf instead can be useful:

def transformInstance(features: featuresType): OutputType = {operations on single instance}
val column_operation = udf {transformInstance}

Predictors also don't have a public method that does predictions on single instances. transformInstance can be easily added to predictors by acting as a wrapper for the internal method predict (which takes features as input).

The proposed method in this change is added to all predictors and feature transformers except OnehotEncoder, VectorSlicer, and Word2Vec, which might require bigger changes due to dependencies on the dataset's schema (they can be fixed using simple hacks but this needs to be discussed)

How was this patch tested?

The current tests for transformers and predictors, which invoke transformInstance internally, passed.

[AmplabJenkins]

Can one of the admins verify this patch?

[hazimehh]

@rxin@jkbradley@mengxr can you review this?

[rxin]

I don't know ML that well.

cc @jkbradley@thunterdb@dbtsai@yanboliang

[jkbradley]

I just responded on the main JIRA. Can you please check that out and close this issue for now? Thanks!