ZEPPELIN-289: User can now enter custom expressions in notebooks' input fields

[rolmovel]

Actually, with Zeppelin we can use Spark SQL UDFs perfectly fine.

We developed a custom UDF library that parses absolute and relative dates. Feeding this library into Spark SQL using the standard UDF mechanism is suboptimal, since each UDF call is repeated for each row of the queried table.

Example:

select * from my_table where agg_date >= parseDate(“-5d”)

This repeats the call to parseDate(...) for every single row of 'my_table'.

Even worse, if we filter for a date range like in:

select * from my_table where agg_date >= parseDate(“-5d”) and agg_date <= parseDate(“now”)

the call to parseDate(...) is performed twice for each row in the table.

Since Spark's UDFs do not have a concept of 'execution context' we were not able to overcome the problem.

We implemented a mechanism of UDF evaluation in Zeppelin, before the query parameters are sent to the interpreter. Parametrizing queries as usual in Zeppelin, in Zeppelin's input forms you can now enter expressions like:

eval:parseDate("-5d")

or:

eval:com.company.custom.udf.UDFUtility.parseDate("-5d")

this is similar to how standard SQL works, where parameters are evaluated before being sent to the execution engine.

You can find more info in the org.apache.zeppelin.display.Evaluator javadoc.

The above mentioned query over a table of 1 million records lasts about 1 minute. Applying this PR the execution time is reduced to 15 seconds.

[bzz]

Looks interesting, thank you for contributing!

Please help me to understand, am I right that these changes potentially affect all interpreter's syntax and code-wise are not localised to your particular use-case with spark sql?

[lucarosellini]

Hi @bzz,
this feature is unaware of the underlying interpreter the code is being sent to, no interpreter specific code has been changed.
We've tested it successfully with spark sql, hive and markdown interpreters, it should work with any other interpreter as well.

[bzz]

@lucarosellini thanks for the explanation!

@rolmovel Could you merge latest master in to resolve conflicts as well as update zeppelin-distribution/src/bin_license/LICENSE with new dependencies added?

[corneadoug]

@rolmovel If this PRs is still needed, can we try to rebase it?

[felixcheung]

This looks to be an unique and important feature to have, will be great to have this in Zeppelin