fix: add ability to use single-row transactions

[mackenziestarr]

Fixes#10018 🦕

Tested manually with the following script, no longer getting google.api_core.exceptions.FailedPrecondition: 400 Single-row transactions are not allowed by this app profile and confirmed data was written to my test table.

#!/usr/bin/env pythonfromgoogle.cloud.bigtable.clientimportClientfromgoogle.cloud.bigtable.rowimportConditionalRowfromgoogle.cloud.bigtable.row_filtersimportPassAllFilterPROJECT_ID="<PROJECT_ID>"INSTANCE_ID="<INSTANCE_ID>"TABLE_ID="<TABLE_ID>"APP_PROFILE_ID="<APP_PROFILE_ID>"client=Client(project=PROJECT_ID)
instance=client.instance(instance_id=INSTANCE_ID)
table=instance.table(table_id=TABLE_ID, app_profile_id=APP_PROFILE_ID)
row_cond=ConditionalRow(b'test_conditional', table, PassAllFilter(True))
row_cond.set_cell('test_column_family', 'test_column_qualifier', 'test_cell_value', timestamp=None, state=False)
row_cond.commit()
row=table.row("test_append", append=True)
row.append_cell_value(u'test_column_family', b'test_column_qualifier', b'test-value')
row.commit()

[igorbernstein2]

Thanks for catching this!

[kolea2]

Thanks for adding this! Would you be able to add a unit test for this as well?

[mackenziestarr]

hey @kolea2 see 20a4838, let me know if there is a better way to access the arguments passed to the mock

[crwilcox]

Thanks for this contribution. It seems 2 of the testcases are failing as well:

=================================== FAILURES ===================================
________________________ TestConditionalRow.test_commit ________________________
self = <tests.unit.test_row.TestConditionalRow testMethod=test_commit>
def test_commit(self):
from google.cloud.bigtable.row_filters import RowSampleFilter
from google.cloud.bigtable_v2.gapic import bigtable_client
project_id = "project-id"
row_key = b"row_key"
table_name = "projects/more-stuff"
column_family_id1 = u"column_family_id1"
column_family_id2 = u"column_family_id2"
column_family_id3 = u"column_family_id3"
column1 = b"column1"
column2 = b"column2"
api = bigtable_client.BigtableClient(mock.Mock())
credentials = _make_credentials()
client = self._make_client(
project=project_id, credentials=credentials, admin=True
)
table = _Table(table_name, client=client)
row_filter = RowSampleFilter(0.33)
row = self._make_one(row_key, table, filter_=row_filter)
# Create request_pb
value1 = b"bytes-value"
# Create response_pb
predicate_matched = True
response_pb = _CheckAndMutateRowResponsePB(predicate_matched=predicate_matched)
# Patch the stub used by the API method.
api.transport.check_and_mutate_row.side_effect = [response_pb]
client._table_data_client = api
# Create expected_result.
expected_result = predicate_matched
# Perform the method and check the result.
row.set_cell(column_family_id1, column1, value1, state=True)
row.delete(state=False)
row.delete_cell(column_family_id2, column2, state=True)
row.delete_cells(column_family_id3, row.ALL_COLUMNS, state=True)
> result = row.commit()
tests/unit/test_row.py:446:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <google.cloud.bigtable.row.ConditionalRow object at 0x7fef09396f60>
def commit(self):
"""Makes a ``CheckAndMutateRow`` API request.
If no mutations have been created in the row, no request is made.
The mutations will be applied conditionally, based on whether the
filter matches any cells in the :class:`ConditionalRow` or not. (Each
method which adds a mutation has a ``state`` parameter for this
purpose.)
Mutations are applied atomically and in order, meaning that earlier
mutations can be masked / negated by later ones. Cells already present
in the row are left unchanged unless explicitly changed by a mutation.
After committing the accumulated mutations, resets the local
mutations.
For example:
.. literalinclude:: snippets_table.py
:start-after: [START bigtable_row_commit]
:end-before: [END bigtable_row_commit]
:rtype: bool
:returns: Flag indicating if the filter was matched (which also
indicates which set of mutations were applied by the server).
:raises: :class:`ValueError <exceptions.ValueError>` if the number of
mutations exceeds the :data:`MAX_MUTATIONS`.
"""
true_mutations = self._get_mutations(state=True)
false_mutations = self._get_mutations(state=False)
num_true_mutations = len(true_mutations)
num_false_mutations = len(false_mutations)
if num_true_mutations == 0 and num_false_mutations == 0:
return
if num_true_mutations > MAX_MUTATIONS or num_false_mutations > MAX_MUTATIONS:
raise ValueError(
"Exceed the maximum allowable mutations (%d). Had %s true "
"mutations and %d false mutations."
% (MAX_MUTATIONS, num_true_mutations, num_false_mutations)
)
data_client = self._table._instance._client.table_data_client
resp = data_client.check_and_mutate_row(
table_name=self._table.name,
row_key=self._row_key,
predicate_filter=self._filter.to_pb(),
> app_profile_id=self._table._app_profile_id,
true_mutations=true_mutations,
false_mutations=false_mutations,
)
E AttributeError: '_Table' object has no attribute '_app_profile_id'
google/cloud/bigtable/row.py:585: AttributeError
__________________________ TestAppendRow.test_commit ___________________________
self = <tests.unit.test_row.TestAppendRow testMethod=test_commit>
def test_commit(self):
from google.cloud._testing import _Monkey
from google.cloud.bigtable import row as MUT
from google.cloud.bigtable_v2.gapic import bigtable_client
project_id = "project-id"
row_key = b"row_key"
table_name = "projects/more-stuff"
column_family_id = u"column_family_id"
column = b"column"
api = bigtable_client.BigtableClient(mock.Mock())
credentials = _make_credentials()
client = self._make_client(
project=project_id, credentials=credentials, admin=True
)
table = _Table(table_name, client=client)
row = self._make_one(row_key, table)
# Create request_pb
value = b"bytes-value"
# Create expected_result.
row_responses = []
expected_result = object()
# Patch API calls
client._table_data_client = api
def mock_parse_rmw_row_response(row_response):
row_responses.append(row_response)
return expected_result
# Perform the method and check the result.
with _Monkey(MUT, _parse_rmw_row_response=mock_parse_rmw_row_response):
row.append_cell_value(column_family_id, column, value)
> result = row.commit()
tests/unit/test_row.py:595:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <google.cloud.bigtable.row.AppendRow object at 0x7fef0938d278>
def commit(self):
"""Makes a ``ReadModifyWriteRow`` API request.
This commits modifications made by :meth:`append_cell_value` and
:meth:`increment_cell_value`. If no modifications were made, makes
no API request and just returns ``{}``.
Modifies a row atomically, reading the latest existing
timestamp / value from the specified columns and writing a new value by
appending / incrementing. The new cell created uses either the current
server time or the highest timestamp of a cell in that column (if it
exceeds the server time).
After committing the accumulated mutations, resets the local mutations.
For example:
.. literalinclude:: snippets_table.py
:start-after: [START bigtable_row_commit]
:end-before: [END bigtable_row_commit]
:rtype: dict
:returns: The new contents of all modified cells. Returned as a
dictionary of column families, each of which holds a
dictionary of columns. Each column contains a list of cells
modified. Each cell is represented with a two-tuple with the
value (in bytes) and the timestamp for the cell.
:raises: :class:`ValueError <exceptions.ValueError>` if the number of
mutations exceeds the :data:`MAX_MUTATIONS`.
"""
num_mutations = len(self._rule_pb_list)
if num_mutations == 0:
return {}
if num_mutations > MAX_MUTATIONS:
raise ValueError(
"%d total append mutations exceed the maximum "
"allowable %d." % (num_mutations, MAX_MUTATIONS)
)
data_client = self._table._instance._client.table_data_client
row_response = data_client.read_modify_write_row(
table_name=self._table.name,
row_key=self._row_key,
rules=self._rule_pb_list,
> app_profile_id=self._table._app_profile_id,
)
E AttributeError: '_Table' object has no attribute '_app_profile_id'
google/cloud/bigtable/row.py:915: AttributeError

Also, it isn't currently passing lint. if you run nox -s blacken it will autoformat.

[mackenziestarr]

thanks @crwilcox updating now

[mackenziestarr]

@crwilcox added unit tests for the changes in 20a4838. unit-2.7 and unit-3.6 pass locally for me (still waiting on Kokoro), assuming this get merged in shortly when can we expect it to make it into a release? I'm currently blocked on not being able to use single-row transactions.

[crwilcox]

Thanks @mackenziestarr for tidying up the tests and your contribution!

[kolea2]

Hey @mackenziestarr, we just completed a release of this: https://github.com/googleapis/google-cloud-python/releases/tag/bigtable-1.2.1

[mackenziestarr]

thanks @kolea2 and @crwilcox for the quick turnaround on the release!