Describe the bug, including details regarding any error messages, version, and platform.
pyarrow has no issue casting timestamps like '2000-01-01 00:00:00' to date32, but for some reason it doesn't like doing it when reading a CSV. Generally, if types are safe to cast, I'd expect that read_csv can directly convert to the desired type.
fromioimportBytesIOimportpyarrowaspafrompyarrowimportcsvdata="\n".join(
[
"time,value",
"2000-01-01 00:00:00,-1",
"2000-01-02 00:00:00,-2",
"2000-01-03 00:00:00,-3",
]
)
schema=pa.schema(
{
"time": pa.date32(),
"value": pa.int8(),
}
)
withBytesIO(data.encode("utf8")) asfile:
table=csv.read_csv(file)
table=table.cast(schema) # ✔ casting OKwithBytesIO(data.encode("utf8")) asfile:
table=csv.read_csv( # ✘ ArrowInvalidfile,
convert_options=csv.ConvertOptions(column_types=schema),
)Component(s)
Python
Describe the bug, including details regarding any error messages, version, and platform.
pyarrowhas no issue casting timestamps like'2000-01-01 00:00:00'todate32, but for some reason it doesn't like doing it when reading a CSV. Generally, if types are safe to cast, I'd expect that read_csv can directly convert to the desired type.Component(s)
Python