Support (non-legacy) SQL nested data types

[dhermes]

STRUCT<t> and ARRAY<t> seem to be the types.

See #2342. h/t to @c0b for filing.

---

From #2585: Sample app that reproduces this

Stacktrace:

Traceback (most recent call last):
 File "simple_app.py", line 57, in <module>
 query_shakespeare()
 File "simple_app.py", line 47, in query_shakespeare
 page_token=page_token)
 File "lib/python2.7/site-packages/google/cloud/bigquery/query.py", line 401, in fetch_data
 rows_data = _rows_from_json(response.get('rows', ()), self.schema)
 File "lib/python2.7/site-packages/google/cloud/bigquery/_helpers.py", line 98, in _rows_from_json
 for item in cell['v']])
 File "lib/python2.7/site-packages/google/cloud/bigquery/_helpers.py", line 61, in _record_from_json
 for subfield, cell in zip(field.fields, value['f']):
KeyError: 'f'

---

Stepping into the debugger:

ipdb> value
{u'v': {u'f': [{u'v': u'hamlet'}, {u'v': u'5318'}]}}
ipdb> field.__dict__
{'field_type': u'RECORD', 'description': None, 'name': u'title', 'fields': [<google.cloud.bigquery.schema.SchemaField object at 0x7f4ba9227550>, <google.cloud.bigquery.schema.SchemaField object at 0x7f4ba9227690>], 'mode': u'REPEATED'}

(Added by @dhermes)

[dhermes]

A simple statement works fine

>>> from google.cloud import bigquery
>>> client = bigquery.Client()
>>> query = client.run_sync_query('SELECT 1')
>>> query.use_legacy_sql = False
>>> query.run()
>>> query.rows
[(1,)]

Sending a query with nested types also works fine

In [1]: from google.cloud import bigquery

In [2]: client = bigquery.Client()

In [3]: sql_str = (
 ...: 'SELECT ARRAY['
 ...: 'STRUCT("zx83" AS id, 1 AS a, 2 AS b), '
 ...: '("f8f7", 4, 7)'
 ...: '] '
 ...: 'd')

In [4]: query = client.run_sync_query(sql_str)

In [5]: query.use_legacy_sql = False

In [6]: query.run()

In [7]: query.__dict__
Out[7]:
{'_client': <google.cloud.bigquery.client.Client at 0x7fb7cf39a290>,
 '_configuration': <google.cloud.bigquery.query._SyncQueryConfiguration at 0x7fb7cf3b0fd0>,
 '_job': None,
 '_properties': {u'cacheHit': False,
 u'jobComplete': True,
 u'jobReference': {u'jobId': u'job_ruoSHLwjeg_lSfYZq8dmrjQ0Eg4',
 u'projectId': u'precise-truck-742'},
 u'kind': u'bigquery#queryResponse',
 u'rows': [{u'f': [{u'v': [{u'v': {u'f': [{u'v': u'zx83'},
 {u'v': u'1'},
 {u'v': u'2'}]}},
 {u'v': {u'f': [{u'v': u'f8f7'}, {u'v': u'4'}, {u'v': u'7'}]}}]}]}],
 u'schema': {u'fields': [{u'fields': [{u'mode': u'NULLABLE',
 u'name': u'id',
 u'type': u'STRING'},
 {u'mode': u'NULLABLE', u'name': u'a', u'type': u'INTEGER'},
 {u'mode': u'NULLABLE', u'name': u'b', u'type': u'INTEGER'}],
 u'mode': u'REPEATED',
 u'name': u'd',
 u'type': u'RECORD'}]},
 u'totalBytesProcessed': u'0',
 u'totalRows': u'1'},
 '_udf_resources': (),
 'query': 'SELECT ARRAY[STRUCT("zx83" AS id, 1 AS a, 2 AS b), ("f8f7", 4, 7)] d'}

but the rows accessor fails

In [8]: query.rows
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
<ipython-input-8-27b591eefa89> in <module>()
----> 1 query.rows

.../google/cloud/bigquery/query.pyc in rows(self)
 223 :returns: fields describing the schema (None until set by the server).
 224 """
--> 225 return _rows_from_json(self._properties.get('rows', ()), self.schema)
 226
 227 @property

.../google/cloud/bigquery/_helpers.pyc in _rows_from_json(rows, schema)
 96 if field.mode == 'REPEATED':
 97 row_data.append([converter(item, field)
---> 98 for item in cell['v']])
 99 else:
 100 row_data.append(converter(cell['v'], field))

.../google/cloud/bigquery/_helpers.pyc in _record_from_json(value, field)
 59 if _not_null(value, field):
 60 record = {}
---> 61 for subfield, cell in zip(field.fields, value['f']):
 62 converter = _CELLDATA_FROM_JSON[subfield.field_type]
 63 if field.mode == 'REPEATED':

KeyError: 'f'

[dhermes]

Just for fun, here are those last properties as JSON:

{
 "cacheHit": false, 
 "jobComplete": true, 
 "jobReference": {
 "jobId": "job_ruoSHLwjeg_lSfYZq8dmrjQ0Eg4", 
 "projectId": "precise-truck-742"
 }, 
 "kind": "bigquery#queryResponse", 
 "rows": [
 {
 "f": [
 {
 "v": [
 {
 "v": {
 "f": [
 {
 "v": "zx83"
 }, 
 {
 "v": "1"
 }, 
 {
 "v": "2"
 }
 ]
 }
 }, 
 {
 "v": {
 "f": [
 {
 "v": "f8f7"
 }, 
 {
 "v": "4"
 }, 
 {
 "v": "7"
 }
 ]
 }
 }
 ]
 }
 ]
 }
 ], 
 "schema": {
 "fields": [
 {
 "fields": [
 {
 "mode": "NULLABLE", 
 "name": "id", 
 "type": "STRING"
 }, 
 {
 "mode": "NULLABLE", 
 "name": "a", 
 "type": "INTEGER"
 }, 
 {
 "mode": "NULLABLE", 
 "name": "b", 
 "type": "INTEGER"
 }
 ], 
 "mode": "REPEATED", 
 "name": "d", 
 "type": "RECORD"
 }
 ]
 }, 
 "totalBytesProcessed": "0", 
 "totalRows": "1"
}

[c0b]

and where can I get alias names? in a query like below I am expecting to access results in a dict way like result.a result.b or result["a"] result["b"] because in my actual app (in nodejs) I have more complicated query generated schema, there are more fields, to access all those fields in the tuple (1, 2) way is not convenient and error prone; please let me know should it be another issue by its own? or can be documentation request if you already have it (but I don't see an example in current doc)

https://googlecloudplatform.github.io/google-cloud-python/stable/bigquery-query.html

In [56]: query = client.run_sync_query('SELECT 1 AS a, 2 AS b'); query.use_legacy_sql = False; query.run()

In [57]: query.__dict__
Out[57]: 
{'_client': <gcloud.bigquery.client.Client at 0x7f89a047ad90>,
 '_configuration': <gcloud.bigquery.query._SyncQueryConfiguration at 0x7f89a0404710>,
 '_job': None,
 '_properties': {u'cacheHit': False,
 u'jobComplete': True,
 u'jobReference': {u'jobId': u'job_POdEXqw6MgHtL-WDZX21BA7_fII',
 u'projectId': u'hb-inspector'},
 u'kind': u'bigquery#queryResponse',
 u'rows': [{u'f': [{u'v': u'1'}, {u'v': u'2'}]}],
 u'schema': {u'fields': [{u'mode': u'NULLABLE',
 u'name': u'a',
 u'type': u'INTEGER'},
 {u'mode': u'NULLABLE', u'name': u'b', u'type': u'INTEGER'}]},
 u'totalBytesProcessed': u'0',
 u'totalRows': u'1'},
 '_udf_resources': (),
 'query': 'SELECT 1 AS a, 2 AS b'}

In [58]: query.rows
Out[58]: [(1, 2)]

[c0b]

I just found bq tool has a command switch --apilog can save the detail log, exposed the library used by bq tool's python code is google-api-python-client; that one is able to handle nested google-api-python-client; at least not crashing, but the data types handling are still not there googleapis/google-api-python-client#281

INFO:root:user-agent: google-api-python-client/1.3.1 (gzip)

[theacodes]

Bump - this is blocking a critical sample.

[dhermes]

@tseaver You still planning to handle?

[theacodes]

Another bump here.