[BigQuery] Can't insert correct TIMESTAMP.

[kamatama41]

In my environment, I can't insert correct data to the TIMESTAMP column.

• gcloud 0.8.0
• Python 2.7.10
• MacOS X 10.10.5

The following snippet is the smallest reproducible example.
Please let me know that it's a bug or my mistake.

import os
from datetime import datetime
from gcloud import bigquery

GCP_KEY_PATH = os.path.join(os.path.dirname(__file__), './gcpkey.json')
project_id = 'my_project'
dataset = 'tmp'
table_name = 'test'

bq = bigquery.Client.from_service_account_json(GCP_KEY_PATH, project=project_id)
table = bq.dataset(dataset).table(name=table_name)

if not table.exists():
 table.schema = [bigquery.SchemaField('time', 'TIMESTAMP')]
 table.create()

table.reload()
rows = [[datetime.now()]]
table.insert_data(rows)

[kamatama41]

I suspect that here is the cause.
BigQuery's TIMESTAMP requires the UNIX timestamp as seconds but the lines convert datetime to milliseconds.

ref: https://cloud.google.com/bigquery/preparing-data-for-bigquery?hl=en#datatypes

[dhermes]

Totally agree. @tseaver and I had a discussion about this before and we thought we had a found a decisive place indicating how the value should be sent. It seems we got it wrong.

[kamatama41]

Thank you for your comment!
Does it mean that this is a bug and you're going to fix it?
(Sorry for my poor English proficiency)

[dhermes]

I can't say 100% it's a bug, but it seems pretty likely. It means we'll dig into verifying and then fixing if we can find a verifiable bug. (Again seems very likely.)

[kamatama41]

OK, I got it. I hope it's fixed. 😄

[tseaver]

@kamatama41 The docs you link above state:

A positive or negative decimal number. A positive number specifies the number of seconds since the epoch (1970-01-01 00:00:00 UTC), and a negative number specifies the number of seconds before the epoch. Up to 6 decimal places (microsecond precision) are preserved.

So, rather than passing milliseconds, we should just pass seconds.

[dhermes]

FYI @tseaver I am fixing this right now

[dhermes]

I'm also finding lots of out-of-date docs in http://gcloud-python.readthedocs.org/en/latest/bigquery-usage.html