Similar to ARROW-15478, but not for nested fields but just for casting a full Table (in theory that could be the same code, but currently the Table.cast logic is implemented in cython).
So currently when casting a Table to a new schema, the nullability of the fields in the schema is ignored (and as a result you get an "invalid" schema indicating a field is non-nullable that actually can have nulls):
>>> table = pa.table({'a': [None, 1]})
>>> tablepyarrow.Tablea: int64
----
a: [[null,1]]
>>> new_schema = pa.schema([pa.field("a", "int64", nullable=False)])
>>> table.cast(new_schema)
pyarrow.Tablea: int64notnull
----
a: [[null,1]]Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Kshiteej K / @kshitij12345
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-16651. Please see the migration documentation for further details.
Similar to ARROW-15478, but not for nested fields but just for casting a full Table (in theory that could be the same code, but currently the Table.cast logic is implemented in cython).
So currently when casting a Table to a new schema, the nullability of the fields in the schema is ignored (and as a result you get an "invalid" schema indicating a field is non-nullable that actually can have nulls):
Reporter: Joris Van den Bossche / @jorisvandenbossche
Assignee: Kshiteej K / @kshitij12345
Related issues:
PRs and other links:
Note: This issue was originally created as ARROW-16651. Please see the migration documentation for further details.