Uh oh!
There was an error while loading. Please reload this page.
This repository was archived by the owner on Apr 1, 2026. It is now read-only.
feat: optimize read_rows - #852
Merged
daniel-sanche merged 79 commits intoAug 17, 2023
Merged
Conversation
daniel-sancheforce-pushed
the
optimize_read_rows_2
branch
from
August 9, 2023 21:08
0145f1a to
62f1bf7Comparedaniel-sancheforce-pushed
the
optimize_read_rows_2
branch
from
August 9, 2023 23:51
59946f7 to
4cf4aabCompareThis reverts commit 2a82343.
daniel-sanche
marked this pull request as ready for review
August 14, 2023 19:34
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| if self._remaining_count is not None: | ||
| self._remaining_count -= 1 | ||
| if self._remaining_count < 0: | ||
| raise RuntimeError("emit count exceeds row limit") |
Contributor
There was a problem hiding this comment.
Should this be a more specific error? Maybe add a todo to improve it in the future
ContributorAuthor
There was a problem hiding this comment.
Sure, I opened an issue to track this here: #853
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
| new_gapic_stream.cancel() | ||
| raise exc | ||
| raise InvalidChunk("reset row with data") | ||
| continue |
Contributor
There was a problem hiding this comment.
I think you have to reset the shared family & qualifier variables
ContributorAuthor
There was a problem hiding this comment.
Those should be reset on each row iteration on L232 and L233
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
daniel-sanche added a commit
that referenced
this pull request
Feb 5, 2024
* feat: add new v3.0.0 API skeleton (#745) * feat: improve rows filters (#751) * feat: read rows query model class (#752) * feat: implement row and cell model classes (#753) * feat: add pooled grpc transport (#748) * feat: implement read_rows (#762) * feat: implement mutate rows (#769) * feat: literal value filter (#767) * feat: row_exists and read_row (#778) * feat: read_modify_write and check_and_mutate_row (#780) * feat: sharded read rows (#766) * feat: ping and warm with metadata (#810) * feat: mutate rows batching (#770) * chore: restructure module paths (#816) * feat: improve timeout structure (#819) * fix: api errors apply to all bulk mutations * chore: reduce public api surface (#820) * feat: improve error group tracebacks on < py11 (#825) * feat: optimize read_rows (#852) * chore: add user agent suffix (#842) * feat: optimize retries (#854) * feat: add test proxy (#836) * chore(tests): add conformance tests to CI for v3 (#870) * chore(tests): turn off fast fail for conformance tets (#882) * feat: add TABLE_DEFAULTS enum for table method arguments (#880) * fix: pass None for retry in gapic calls (#881) * feat: replace internal dictionaries with protos in gapic calls (#875) * chore: optimize gapic calls (#863) * feat: expose retryable error codes to users (#879) * chore: update api_core submodule (#897) * chore: merge main into experimental_v3 (#900) * chore: pin conformance tests to v0.0.2 (#903) * fix: bulk mutation eventual success (#909) --------- Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rewrite the row merger logic to improve throughput
This change brings us to ~75k RPS with retries, ~90 with retries disabled (retry logic will be optimized in a future PR)
Note: the idle timeout has been removed as a part of these changes. Other features are preserved
#857