Skip to content
This repository was archived by the owner on Apr 1, 2026. It is now read-only.

feat: mutate rows batching - #770

Merged
daniel-sanche merged 159 commits into
googleapis:v3from
daniel-sanche:mutate_rows_batch
Jun 26, 2023
Merged

feat: mutate rows batching#770
daniel-sanche merged 159 commits into
googleapis:v3from
daniel-sanche:mutate_rows_batch

Conversation

@daniel-sanche

@daniel-sanchedaniel-sanche commented May 5, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Blocked on #769. This PR builds off of changes from this branch

Implements MutationsBatcher to submit mutations in the background.

@daniel-sanche
added initial implementation of mutate_rows
1d02154
@daniel-sanche
implemented mutation models
ab63cba
@daniel-sanche
added retries to mutate_row
cf9daa5
@daniel-sanche
return exception group if possible
1247da4
@daniel-sanche
check for idempotence
3b3ed8c
@daniel-sanche
initial implementation for bulk_mutations
5d20037
@daniel-sanche
include successes in bulk mutation error message
3d322a1
@daniel-sanche
fixed style checks
a31232b
@daniel-sanche
added basic system tests
8da2d65
@daniel-sanche
added unit tests for mutate_row
2b89d9c
@daniel-sanche
ran blacken
47c5985
@daniel-sanche
improved exceptions
38fdcd7
@daniel-sanche
added bulk_mutate_rows unit tests
504d2d8
@daniel-sanche
ran blacken
b16067f
@daniel-sanche
support __new___ for exceptions for python3.11+
3ab1405
@daniel-sanche
added exception unit tests
0a6c0c6
@daniel-sanche
makde exceptions tuple
ec043cf
@daniel-sanche
got exceptions to print consistently across versions
518530e
@daniel-sanche
added test for 311 rich traceback
9624729
@daniel-sanche
moved retryable row mutations to new file
3087081
@daniel-sanche
use index map
9df588f
@daniel-sanche
added docstring
7ed8be3
@daniel-sanche
added predicate check to failed mutations
2536cc4
@daniel-sanche
added _mutate_rows tests
1f6875c
@daniel-sanche
improved client tests
1ea24e6
@daniel-sanche
refactored to loop by raising exception
25ca2d2
@daniel-sanche
refactored retry deadline logic into shared wrapper
c0787db
@daniel-sanche
ran black
3ed5c3d
@daniel-sanche
pulled in table default timeouts
a91fbcb
@daniel-sanche
added tests for shared deadline parsing function
df8a058
@daniel-sanche

daniel-sanche commented Jun 21, 2023

Copy link
Copy Markdown
ContributorAuthor

@igorbernstein2@mutianf

I added timeout configuration to the batcher init signature. Previously, it would always use the table defaults. This is part of the public API, so just wanted to make sure you saw the change in case you have thoughts

daniel-sanche
daniel-sanche commented Jun 23, 2023
View reviewed changes
Comment threadtests/unit/test_mutations_batcher.py
Comment threadgoogle/cloud/bigtable/mutations_batcher.py
Comment threadgoogle/cloud/bigtable/mutations_batcher.py Outdated
20 exceptions have been encountered since the last flush, only the
first and last 10 exceptions will be stored.
"""
return self._oldest_exceptions + self._newest_exceptions

@daniel-sanchedaniel-sancheJun 23, 2023•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to make the staged exception list public? Or maybe just make a has_errors function?

@igorbernstein2igorbernstein2Jun 23, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what you mean

igorbernstein2
igorbernstein2 reviewed Jun 23, 2023
View reviewed changes
Comment threadgoogle/cloud/bigtable/client.py Outdated
Comment threadgoogle/cloud/bigtable/mutations_batcher.py Outdated
Comment threadgoogle/cloud/bigtable/mutations_batcher.py
Comment threadgoogle/cloud/bigtable/mutations_batcher.py Outdated
20 exceptions have been encountered since the last flush, only the
first and last 10 exceptions will be stored.
"""
return self._oldest_exceptions + self._newest_exceptions

@igorbernstein2igorbernstein2Jun 23, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what you mean

igorbernstein2
igorbernstein2 reviewed Jun 23, 2023
View reviewed changes
Comment threadgoogle/cloud/bigtable/mutations_batcher.py
timer_task = asyncio.create_task(timer_routine(self, interval))
return timer_task

async def append(self, mutation_entry: RowMutationEntry):

@igorbernstein2igorbernstein2Jun 23, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a TODO to return a future with the result of this mutation

Comment threadgoogle/cloud/bigtable/mutations_batcher.py Outdated
in_process_requests.append(batch_task)
# wait for all inflight requests to complete
found_exceptions = await self._wait_for_batch_results(*in_process_requests)
# allow previous flush tasks to finalize before adding new exceptions to list

@igorbernstein2igorbernstein2Jun 23, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this comment stale?

Comment threadgoogle/cloud/bigtable/mutations_batcher.py Outdated
Comment threadgoogle/cloud/bigtable/mutations_batcher.py Outdated
Comment threadgoogle/cloud/bigtable/mutations_batcher.py Outdated
mutianf
mutianf reviewed Jun 26, 2023
View reviewed changes
Comment threadgoogle/cloud/bigtable/mutations_batcher.py
@daniel-sanche
daniel-sanche merged commit 1ecf65f into googleapis:v3Jun 26, 2023
daniel-sanche added a commit that referenced this pull request Feb 5, 2024
@daniel-sanche@gcf-owl-bot
chore: add experimental async data client (#920)
7088e39
* feat: add new v3.0.0 API skeleton (#745)
* feat: improve rows filters (#751)
* feat: read rows query model class (#752)
* feat: implement row and cell model classes (#753)
* feat: add pooled grpc transport (#748)
* feat: implement read_rows (#762)
* feat: implement mutate rows (#769)
* feat: literal value filter (#767)
* feat: row_exists and read_row (#778)
* feat: read_modify_write and check_and_mutate_row (#780)
* feat: sharded read rows (#766)
* feat: ping and warm with metadata (#810)
* feat: mutate rows batching (#770)
* chore: restructure module paths (#816)
* feat: improve timeout structure (#819)
* fix: api errors apply to all bulk mutations
* chore: reduce public api surface (#820)
* feat: improve error group tracebacks on < py11 (#825)
* feat: optimize read_rows (#852)
* chore: add user agent suffix (#842)
* feat: optimize retries (#854)
* feat: add test proxy (#836)
* chore(tests): add conformance tests to CI for v3 (#870)
* chore(tests): turn off fast fail for conformance tets (#882)
* feat: add TABLE_DEFAULTS enum for table method arguments (#880)
* fix: pass None for retry in gapic calls (#881)
* feat: replace internal dictionaries with protos in gapic calls (#875)
* chore: optimize gapic calls (#863)
* feat: expose retryable error codes to users (#879)
* chore: update api_core submodule (#897)
* chore: merge main into experimental_v3 (#900)
* chore: pin conformance tests to v0.0.2 (#903)
* fix: bulk mutation eventual success (#909)
---------
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

2 more reviewers
@mutianfmutianfmutianf left review comments
@igorbernstein2igorbernstein2igorbernstein2 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: bigtableIssues related to the googleapis/python-bigtable API.size: xlPull request size is extra large.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@daniel-sanche@mutianf@igorbernstein2