feat: add json_arrow_type parameter to allow overriding the JSON data type in to_arrow and to_arrow_iterable

[tswast]

Follow-up to #1876 (comment)

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #<issue_number_goes_here> 🦕

[tswast]

Currently fails with

 rows = google.cloud.bigquery.table.RowIterator(
client=None,
api_request=None,
path=None,
schema=[
bigquery.SchemaField.from_api_repr(field)
for field in resource["schema"]["fields"]
],
first_page_response=resource,
)
> records = rows.to_arrow(json_arrow_type=db_dtypes.JSONArrowType())
tests/unit/test_table_arrow.py:275: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ google/cloud/bigquery/table.py:2227: in to_arrow
return pyarrow.Table.from_batches(record_batches, schema=arrow_schema)
pyarrow/table.pxi:4994: in pyarrow.lib.Table.from_batches
???
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > ???
E pyarrow.lib.ArrowInvalid: Schema at index 0 was different: E json: extension<dbjson<JSONArrowType>>
E reference: struct<uri: string, authorizer: string, version: string, details: extension<dbjson<JSONArrowType>>>
E repeated_json: list<item: extension<dbjson<JSONArrowType>>> not null
E vs
E json: string
E reference: struct<uri: string, authorizer: string, version: string, details: string>
E repeated_json: list<item: string> not null
pyarrow/error.pxi:92: ArrowInvalid

I think we need to call https://arrow.apache.org/docs/python/generated/pyarrow.compute.cast.html instead of relying on the Table.from_batches constructor.

[tswast]

I think we'll need some kind of two-phase thing where we take the schema as we would get from the BQ Storage Read API and manually cast each field/subfield into the desired type to make it work.

I don't think the approach I'm using in this PR as of now will work. It'll have to do some post-processing on the arrow Table we'd normally get.

[tswast]

Closing as obsolete. Ideally we stay consistent with the BQ Storage API. We should revisit this if the BQ Storage API adds support for the JSON extension type.