Skip to content

Loading AVRO files from local filesystem. - #3427

Merged
tseaver merged 4 commits into
googleapis:masterfrom
tswast:bq-3416
May 16, 2017
Merged

Loading AVRO files from local filesystem.#3427
tseaver merged 4 commits into
googleapis:masterfrom
tswast:bq-3416

Conversation

@tswast

@tswasttswast commented May 16, 2017

Copy link
Copy Markdown
Contributor

Adds an integration test for loading AVRO files.

Don't set the schema if none is provided (patch).

Closes#3416.

tswast added 3 commits May 16, 2017 13:01
@tswast
Add test avro data.
c27100c
Generated with `avro-tools`
$ avro-tools fromjson \
--schema-file tests/data/colors.avroschema \
tests/data/colors.json > tests/data/colors.avro
See:
http://www.michael-noll.com/blog/2013/03/17/reading-and-writing-avro-files-from-the-command-line/
@tswast
Add test to reproducegoogleapis#3416.
dcb05b3
@tswast
Don't set schema in load job if none is provided.
2b1f149
@tswast
tswast requested a review from tseaverMay 16, 2017 21:00
@googlebotgooglebot added the cla: yes This human has signed the Contributor License Agreement. label May 16, 2017
@tseaver

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

@tswast Can you push your branch to this repository, so that the system test actually gets run by the CI before we merge?

@dhermes

dhermes commented May 16, 2017

Copy link
Copy Markdown
Contributor

@tswast If you don't have push privileges one of us can do it

@tseaver

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

Also, @tweeter0830reported that the patch did not fix the issue with uploading AVRO data.

@tswast

tswast commented May 16, 2017

Copy link
Copy Markdown
ContributorAuthor

I saw that, but I was not able to reproduce the issue after the patch.

@tseavertseaver added type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. api: bigquery Issues related to the BigQuery API. labels May 16, 2017
@tswast

tswast commented May 16, 2017

Copy link
Copy Markdown
ContributorAuthor

Hmm, I see that this change made the Avro load test pass, but now the CSV load test fails.
https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/1920

@tseaver

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

@tswast That is weird, as the testcase clearly sets the schema on the table: it should be passing the schema along.

@tswast

tswast commented May 16, 2017

Copy link
Copy Markdown
ContributorAuthor

We missed the "fields" property of the schema object in the patch. Seems to be running fine now. https://circleci.com/gh/GoogleCloudPlatform/google-cloud-python/1922

@tseaver

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

I've pushed the branch here (as tswast-bq-3146) to get the system tests running under CI before merging.

@tseaver

tseaver commented May 16, 2017

Copy link
Copy Markdown
Contributor

BQ system tests are green on Circle CI.

@tseaver
tseaver merged commit 4877315 into googleapis:masterMay 16, 2017
@tswast
tswast deleted the bq-3416 branch May 16, 2017 23:14
@tseavertseaver mentioned this pull request Jun 26, 2017
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tseavertseaverAwaiting requested review from tseaver

Assignees

No one assigned

Labels

api: bigqueryIssues related to the BigQuery API.cla: yesThis human has signed the Contributor License Agreement.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tswast@tseaver@dhermes@googlebot