Skip to content

feat(pandas-gbq): add arrow decoder for read rows response - #17958

Merged
shuoweil merged 6 commits into
mainfrom
shuowei-gbq-pandas-gbq-arrow
Aug 3, 2026
Merged

feat(pandas-gbq): add arrow decoder for read rows response#17958
shuoweil merged 6 commits into
mainfrom
shuowei-gbq-pandas-gbq-arrow

Conversation

@shuoweil

@shuoweilshuoweil commented Jul 30, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Adds pandas_gbq.arrow.from_read_rows_response to decode BigQuery Storage Read API ReadRowsResponse protobuf messages into pyarrow.RecordBatch instances.

Changes

  • pandas_gbq/arrow.py: Implements from_read_rows_response(message, arrow_schema=None) to deserialize IPC stream and message buffers with empty-batch fallback.
  • pandas_gbq/__init__.py: Exposes arrow in package __all__.
  • tests/unit/test_arrow.py: Adds unit tests for valid batch parsing, empty message handling, and missing pyarrow error handling.

Fixes #<540939659> 🦕

@shuoweilshuoweil self-assigned this Jul 30, 2026
@shuoweil
shuoweil requested review from a team as code ownersJuly 30, 2026 21:28
gemini-code-assist[bot]
gemini-code-assistBot reviewed Jul 30, 2026
View reviewed changes

@gemini-code-assistgemini-code-assistBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new arrow integration submodule in pandas-gbq to decode ReadRowsResponse protobuf messages into pyarrow.RecordBatch objects, along with corresponding unit tests. The review feedback points out that pyarrow.Message does not have a schema attribute, which would cause decoding to fail or fall back incorrectly. Additionally, it is recommended to catch specific pyarrow.ArrowException errors instead of using broad except Exception blocks.

Comment threadpackages/pandas-gbq/pandas_gbq/arrow.py Outdated
shuoweiland others added 2 commits July 30, 2026 14:32
@shuoweil@gemini-code-assist
Update packages/pandas-gbq/pandas_gbq/arrow.py
622c007
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@shuoweil
fix(pandas-gbq): catch OSError in arrow IPC deserialization
108fca3
@parthea

parthea commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

@shuoweil, Please can you merge main into this branch to resolve the issue with the presubmit checks? all-tests and mypy-status. These checks were added in #17438

@partheaparthea mentioned this pull request Jul 31, 2026
Merged
@shuoweil

shuoweil commented Jul 31, 2026

Copy link
Copy Markdown
ContributorAuthor

@shuoweil, Please can you merge main into this branch to resolve the issue with the presubmit checks? all-tests and mypy-status. These checks were added in #17438

Thanks, Anthonios!

sycai
sycai reviewed Aug 3, 2026
View reviewed changes
Comment threadpackages/pandas-gbq/tests/unit/test_arrow.py Outdated
mock_message = mock.MagicMock()

with mock.patch.object(pandas_gbq.arrow, "pa", None):
import pytest

@sycaisycaiAug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: let's import pytest at the top of the file.

@shuoweilshuoweilAug 3, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, I move import pytest to the top-level imports.

Comment threadpackages/pandas-gbq/tests/unit/test_arrow.py
import pyarrow as pa

import pandas_gbq.arrow

@sycaisycaiAug 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a test case where the arrow schema is not provided?

@shuoweilshuoweilAug 3, 2026

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, test case is added.

@shuoweil
shuoweil requested a review from sycaiAugust 3, 2026 19:27
@shuoweil
shuoweil merged commit e1df28c into mainAug 3, 2026
42 of 43 checks passed
@shuoweil
shuoweil deleted the shuowei-gbq-pandas-gbq-arrow branch August 3, 2026 23:02
@release-pleaserelease-pleaseBot mentioned this pull request Aug 3, 2026
Merged
shuoweil pushed a commit that referenced this pull request Aug 6, 2026
@release-please
chore(main): release pandas-gbq 0.35.1 (#17496)
453a77d
🤖 I have created a release *beep* *boop*
---
##
[0.35.1](pandas-gbq-v0.35.0...pandas-gbq-v0.35.1)
(2026-08-06)
### Features
* **pandas-gbq:** add arrow decoder for read rows response
([#17958](#17958))
([e1df28c](e1df28c))
### Bug Fixes
* require Protobuf 6.33.5+
([#17743](#17743))
([d267342](d267342))
### Documentation
* add project ID to pandas-gbq run sample
([#17692](#17692))
([bd5d1a4](bd5d1a4))
---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sycaisycaisycai approved these changes
@tswasttswastAwaiting requested review from tswast
@GarrettWuGarrettWuAwaiting requested review from GarrettWu
@chalmerlowechalmerloweAwaiting requested review from chalmerlowe
+1 more reviewer
@gemini-code-assistgemini-code-assist[bot]gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@shuoweilshuoweil

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shuoweil@parthea@sycai