[SPARK-24695] [SQL]: To add support to return Calendar interval from udf.

[priyankagargnitk]

What changes were proposed in this pull request?

This change adds capability to return Calender interval from udf.

Earlier, the udf of Type (String => CalendarInterval) was throwing Exception stating:
Schema for type org.apache.spark.unsafe.types.CalendarInterval is not supported
java.lang.UnsupportedOperationException: Schema for type org.apache.spark.unsafe.types.CalendarInterval is not supported
at org.apache.spark.sql.catalyst.ScalaReflection391anonfun.apply(ScalaReflection.scala:781)

How was this patch tested?

Added test case in ScalaReflectionSuite.scala and ExpressionEncoderSuite.scala
Also, tested by creating an udf that returns Calendar interval.

jira entry for detail: https://issues.apache.org/jira/browse/SPARK-24695

[maropu]

Since CalendarInterval is an internal class, I think users are not intended to use the class directly...

[maropu]

btw, can you update the title like [SPARK-24695][SQL]...

[gatorsmile]

In the short term, we do not plan to make CalendarInterval public.

[priyankagargnitk]

org.apache.spark.unsafe.types.CalenderInterval is already public, am i missing something.
Also, what if I want to do some computation on any data type and return Calender Interval... How should I solve this problem in the current scenario?

[maropu]

Actually, the unsafe package does not include user-facing classes.

[HyukjinKwon]

Yea, so it's kind of half public now but not completely exposed. It should be exposed before we extend its support.

[priyankagargnitk]

What if i make changes to expose it?

[HyukjinKwon]

I would better open a discussion thread in the mailing list before making a PR to expose it. Shell we leave this close this meanwhile? I think it's better to leave active and actionable PRs only.

[HyukjinKwon]

I think we should close this for now then.

[gatorsmile]

For your information, @cloud-fan and I are discussing how we expose calendar interval data types. Will post the proposal later.

[hvanhovell]

I think this one is reasonable. We already exposed it as an argument to a UDF, so it is pretty poor UX to not allow a return type. As for making CalendarInterval public, what is there to do besides moving it?

Reopening this one.

[hvanhovell]

@priyankagargnitk can you update this one.

[AmplabJenkins]

Can one of the admins verify this patch?

[maropu]

@priyankagargnitk ping, are you still there?

[priyankagargnitk]

I am working on this, I'll update the PR in a day or two.

[priyankagargnitk]

A new PR has been raised.

[priyankagargnitk]

#25022