[Python] Casting an empty dictionary array loses category information

[stinodego]

Describe the bug, including details regarding any error messages, version, and platform.

Casting a DictionaryArray to another dictionary type will preserve the original dictionary, unless the array is empty.

Minimal example

import pyarrow as pa

arr = pa.array(["foo", "bar"], pa.dictionary(pa.int32(), pa.string()))
arr_empty = arr[:0]
arr_empty_cast = arr_empty.cast(pa.dictionary(pa.int64(), pa.string()))
print(arr_empty_cast.dictionary) # []

Additional examples

import pyarrow as pa

arr = pa.array(["foo", "bar"], pa.dictionary(pa.int32(), pa.string()))
print(arr.dictionary) # ["foo", "bar"]

arr_short_cast = arr[:1].cast(pa.dictionary(pa.int64(), pa.string()))
print(arr_short_cast.dictionary) # ["foo", "bar"]

arr_empty = arr[:0]
print(arr_empty.dictionary) # ["foo", "bar"]

arr_empty_cast = arr_empty.cast(pa.dictionary(pa.int64(), pa.string()))
print(arr_empty_cast.dictionary) # []

Expected output

The original categories should be preserved when changing the index type, even if the array is empty.

Component(s)

Python

[kou]

Why do we need to do this? Could you explain your use case?

[kou]

@pitrou Do you have any opinion for this case?

[pitrou]

I agree that the original dictionary should probably be preserved for consistency, if it's not too costly to do so.
cc @felipecrv @jorisvandenbossche

[stinodego]

Why do we need to do this? Could you explain your use case?

For Polars, we use pyarrow for the converting our data to pandas. Our Enum data type is similar to a pyarrow Dictionary type with a set number of categories. This converts into a pandas categorical type which also has a set number of categories.

In this conversion, we have to cast the index type from UInt32 (our default) to Int64 since pandas does not support unsigned indices. If the data is empty, this cast now loses the category information, and data type after the conversion is wrong.

See the original issue opened in our repo: pola-rs/polars#14582

[pitrou]

@stinodego Do you think you would be able to work on a PR?

[stinodego]

I'm not familiar with the Arrow code base, but I suppose I could give it a shot.

EDIT: If the implementation here is in C++ then it's probably better if someone else picks this up - I have no experience programming in that language.

[jorisvandenbossche]

Agreed that we should be consistent for empty arrays.

Haven't tested this, but it might be caused by shortcutting calling the actual kernel for empty input here:

arrow/cpp/src/arrow/compute/exec.cc

Lines 781 to 795 in 4dc3d04

	class ScalarExecutor : public KernelExecutorImpl<ScalarKernel> {
	public:
	Status Execute(const ExecBatch& batch, ExecListener* listener) override {
	RETURN_NOT_OK(span_iterator_.Init(batch, exec_context()->exec_chunksize()));
	if (batch.length == 0) {
	// For zero-length batches, we do nothing except return a zero-length
	// array of the correct output type
	ARROW_ASSIGN_OR_RAISE(std::shared_ptr<Array> result,
	MakeArrayOfNull(output_type_.GetSharedPtr(), /*length=*/0,
	exec_context()->memory_pool()));
	return EmitResult(result->data(), listener);
	}
	// If the executor is configured to produce a single large Array output for

This creates a generic array of length 0 of the output type, but so in case of a dictionary output type, it will never have an actual dictionary attached to that return value.

We might want to generally skip that fast-path in case output_type_ is a dictionary type?

[pitrou]

We might want to generally skip that fast-path in case output_type_ is a dictionary type?

Ideally we would also do that for nested types that have a dictionary field somewhere.

But we can start with your idea and see what happens (not all scalar kernels might like zero-length inputs?).