Resource objects such as *JobConfig do not error or warn when setting unknown properties

[Nakulbajaj101]

Thanks for stopping by to let us know something could be better!

PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.

Please run down the following list and make sure you've tried the usual "quick fixes":

• Search the issues already opened: https://github.com/googleapis/python-bigquery/issues
• Search StackOverflow: https://stackoverflow.com/questions/tagged/google-cloud-platform+python

If you are still having issues, please be sure to include as much information as possible:

Environment details

• OS type and version: MACOS Catalina 10.15.5
• Python version: 3.7.6
• pip version: 20.0.2
• google-cloud-bigquery version: 1.28.0

Steps to reproduce

Step 1. Extract BQ tabke data to bucket

Code example

# from google.cloud import bigquery
# client = bigquery.Client()
# bucket_name = 'my-bucket'

destination_uri = "gs://{}/{}".format(bucket_name, "shakespeare.json")
dataset_ref = bigquery.DatasetReference(project, dataset_id)
table_ref = dataset_ref.table("shakespeare")
job_config = bigquery.job.ExtractJobConfig()
job_config.destinationFormat = bigquery.DestinationFormat.NEWLINE_DELIMITED_JSON
job_config.compression = bigquery.Compression.GZIP

extract_job = client.extract_table(
 table_ref,
 destination_uri,
 # Location must match that of the source table.
 location="US",
 job_config=job_config,
) # API request
extract_job.result() # Waits for job to complete.

2. Fetch the table from bucket and load into bq

Code example

from google.cloud import bigquery

# Construct a BigQuery client object.
client = bigquery.Client()

# TODO(developer): Set table_id to the ID of the table to create.
# table_id = "your-project.your_dataset.your_table_name"

job_config = bigquery.LoadJobConfig(
 schema=[
 bigquery.SchemaField("name", "STRING"),
 bigquery.SchemaField("post_abbr", "STRING"),
 ], #This is not shakespeare schema, as just copied pasted the code provided on google docs
 source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON,
)
uri = "gs://{}/{}".format(bucket_name, "shakespeare.json")

load_job = client.load_table_from_uri(
 uri,
 table_id,
 location="US", # Must match the destination dataset location.
 job_config=job_config,
) # Make an API request.

load_job.result() # Waits for the job to complete.

destination_table = client.get_table(table_id)
print("Loaded {} rows.".format(destination_table.num_rows))

Stack trace

Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.

Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0

Error while reading data, error message: JSON parsing error in row starting at position 0: Parser terminated before end of string

However when I extract the table using bq command line and follow the example of:

bq extract \
--compression GZIP \
--destination_format NEWLINE_DELIMITED_JSON \
'mydataset.mytable' \
gs://example-bucket/myfile.json

And then run the second step, I don't get any error and my table is successfully imported.

Making sure to follow these steps will guarantee the quickest resolution possible.

Thanks!

[tswast]

I see a typo. There is no destinationFormat property on ExtractJobConfig

job_config.destinationFormat = bigquery.DestinationFormat.NEWLINE_DELIMITED_JSON

The correct name is destination_format

https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.ExtractJobConfig.html#google.cloud.bigquery.job.ExtractJobConfig.destination_format

Perhaps we need ways of warning about unknown properties?

[tswast]

On internal Q&A forum question 5801905264132096, this same issue was encountered, but with setting Table properties on a TableReference object.

[Nakulbajaj101]

Ah thanks, that might be it, a warning would be lovely