[Python] Refactor the Scalar classes

[asfimport]

The situation regarding scalars in Python is currently not optimal.

We have two different "types" of scalars:

• ArrayValue(Scalar) (and subclasses of that for all types): this is used when you access a single element of an array (eg arr[0])

• ScalarValue(Scalar) (and subclasses of that for some types): this is used when wrapping a C++ scalar into a python scalar, eg when you get back a scalar from a reduction like arr.sum().

  And while we have two versions of scalars, neither of them can actually easily be used as scalar as they both can't be constructed from a python scalar (there is no scalar(1) function to use when calling a kernel, for example).

  I think we should try to unify those scalar classes? (which probably means getting rid of the ArrayValue scalar)

  In addition, there is an issue of trying to re-use python scalar <-> arrow conversion code, as this is also logic for this in the python_to_arrow.cc code. But this is probably a bigger change. cc @kszucs

Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Krisztian Szucs / @kszucs

Related issues:

• [C++] Array::GetScalar not implemented for decimal type (supercedes)

PRs and other links:

• GitHub Pull Request #7519

_{Note: This issue was originally created as ARROW-9017. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
And comment from Ben: relevant recent addition:

 Result<std::shared_ptr<Scalar>> Array::GetScalar(int64_t i) const;

[asfimport]

Antoine Pitrou / @pitrou:
We should definitely unify those around a Cython wrapper to C++ arrow::Scalar, IMHO.

[asfimport]

Krisztian Szucs / @kszucs:
I started to factor out the elementwise conversion code required to convert single python objects to intermediate C representation. I hit a couple of roadblocks and there were also missing utilities like the GetScalar Ben has implemented recently.

We also have an outstanding issue with the auto chunking during conversion: in case of nested types a binary/string field gets chunked if the size limited is reached but the rest of the fields have a single chunk resulting a corrupted nested array.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 7519
#7519