GH-42018: [Python] Add NumPy StringDType to Arrow conversion

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018

[ngoldbaum]

Rationale for this change

NumPy 2.0 added StringDType, a variable-width UTF-8 string dtype. pa.array currently rejects it with Unsupported numpy type 2056. Now that pyarrow requires NumPy 2.0 the conversion can use the public NpyString_* C API.

This supersedes #50951. The first commit is @alippai's first commit from that PR, unchanged. His later commits added batching and new bulk builder APIs in Arrow C++; those can come later in their own PR with benchmarks, as discussed there. Nothing outside the NumPy conversion code changes here.

What changes are included in this PR?

• NumPyDtypeToArrow maps StringDType to string, so it is also the inferred type for pa.array, pa.infer_type, pa.from_numpy_dtype and lists of such arrays. large_string and string_view can be requested with type. Any other type raises a TypeError naming the requested type.
• The conversion holds the dtype's allocator lock for the whole array and never holds the GIL while taking it.
• A null entry in a StringDType array becomes an Arrow null unless the dtype's na_object is a str. NumPy reports such entries as that string from __getitem__, in ufuncs and comparisons, and in casts, and there is no way to tell them apart from a regular entry holding the same string, so the conversion writes the string too. NaN-like sentinels such as np.nan and pd.NA, None, and arbitrary objects all become arrow nulls.
• A short section in docs/source/python/numpy.rst describing the support and the semantics for converting missing data.

If left unfixed, #51156 also becomes possible to trigger from Python by creating a StringDType array with a very large entry.

Are these changes tested?

Yes. Tests cover the three target types, strings crossing NumPy's short, medium and long storage thresholds, non-ASCII input, strided input, mask, None, NaN and string sentinels, the rejected target type, and list inference. I also ran a threaded test that converts an array while other threads hit NumPy's null-comparison error path on the same array to check the lock ordering.

Are there any user-facing changes?

Yes. pa.array and the inference functions accept StringDType arrays.

AI disclosure: I used an AI model to work on the follow-up commits on top of @alippai's first commit and for code review.

• GitHub Issue: [Python] Conversion to/from numpy 2.0+ new StringDType #42018