convert time columns to dbtime by default in `to_dataframe`

[tswast]

Currently TIME columns are just exposed as string objects. This would be a better experience and align with better with the expectations for working with timeseries in pandas https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html

Presumably one could combine a date column with a time column to create a datetime by adding them.

[jimfulton]

Why timedelta and not time?

Do people use BQ TIME columns to represent time deltas?

If we're looking for a way to express timedeltas, I wonder if it might be better to leverage structs somehow.

[tswast]

Is there a time dtype? I didn't see it here: https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.html#basics-dtypes

timedelta seemed to make the most sense to me as date + timedelta = datetime

[tswast]

Note: this is related to the expected dtypes in the v3 branch:

python-bigquery/tests/unit/test_table_pandas.py

Lines 99 to 111 in eed311e

	# Check for expected dtypes.
	# Keep these in sync with tests/system/test_pandas.py
	assert df.dtypes["bignumeric_col"].name == "object"
	assert df.dtypes["bool_col"].name == "boolean"
	assert df.dtypes["bytes_col"].name == "object"
	assert df.dtypes["date_col"].name == "object"
	assert df.dtypes["datetime_col"].name == "datetime64[ns]"
	assert df.dtypes["float64_col"].name == "float64"
	assert df.dtypes["int64_col"].name == "Int64"
	assert df.dtypes["numeric_col"].name == "object"
	assert df.dtypes["string_col"].name == "object"
	assert df.dtypes["time_col"].name == "object"
	assert df.dtypes["timestamp_col"].name == "datetime64[ns, UTC]"

[tswast]

Looking at that, I think date_col should have a datetime64[ns] dtype too (at least when within allowed range in pandas)

https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#generating-ranges-of-timestamps

That's what #861 is about -- picking the right dtype for date columns by default.

[jimfulton]

Is there a time dtype? I didn't see it here: https://pandas.pydata.org/pandas-docs/stable/user_guide/basics.html#basics-dtypes

Well, I was thinking of datetime.time, which would have a dtype of object.

[jimfulton]

timedelta seemed to make the most sense to me as date + timedelta = datetime

Sure, but would BQ users expect TIME columns to be used that way? I think I'd be surprised.

>>> df = pandas.DataFrame(dict(t=[datetime.time(0, 0, 0), datetime.time(14, 40, 0)]))
>>> df
 t
0 00:00:00
1 14:40:00
>>> df = pandas.DataFrame(dict(t=[datetime.timedelta(hours=0, minutes=0, seconds=0), datetime.timedelta(hours=14, minutes=40, seconds=0)]))
>>> df
 t
0 0 days 00:00:00
1 0 days 14:40:00

🤷

In Python, you can't add datetime.datetime and datetime.time values.

[jimfulton]

Looking at that, I think date_col should have a datetime64[ns] dtype too (at least when within allowed range in pandas)

A datetime/timestamp represents an instantaneous point in time.

A date represents a time interval (typically from midnight to midnight). Mapping a date to a datetime is similar, if less egregious :), to mapping a month or year to a datetime.

https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#generating-ranges-of-timestamps

That's what #861 is about -- picking the right dtype for date columns by default.

Sure, but "right" is somewhat relative, and sometimes hard to pin down. Is rightness judged by accuracy or convenience?

[jimfulton]

Anyway, I'll implement whatever you wish. :)