[FEATURE REQUEST]: Update Apache.Arrow to 0.15.1

[imback82]

Spark .NET currently depends on Apache.Arrow 0.14.1. There is a newer version (0.15.1) of Apache.Arrow, which has new features such as date type support, etc. (see #359). We need to update it to the latest version, and also check if there are any breaking changes.

[eerhardt]

One change in 0.15.1 to be on the look out for is apache/arrow@4be02c7 and corresponding spec change - apache/arrow#4951.

You will probably need to set the new WriteLegacyIpcFormat = true option when writing arrow data until Apache Spark is updated with the new Arrow version.

[imback82]

Thanks @eerhardt for heads up. Spark 3.0 is using 0.15.1, so we will make sure to use the option when we support Spark 3.0. (#348).

[eerhardt]

No, you need to invert the thought. WriteLegacyIpcFormat = true will be needed for Spark < 3.0. Basically all older versions of Spark you will need to use the "Legacy IPC format".

[imback82]

Yes, that's what I was planning to do. I meant that I will use the option to support both Spark 2.x and 3.0.

[Niharikadutta]

I believe this issue can be closed now?