Skip to content

feat(bigtable): make write batch size and concurrency configurable - #6847

Open
Reactor11 wants to merge 1 commit into
feast-dev:masterfrom
Reactor11:feat/bigtable-configurable-write-batching
Open

Reactor11 wants to merge 1 commit into
feast-dev:masterfrom
Reactor11:feat/bigtable-configurable-write-batching

Conversation

@Reactor11

Copy link
Copy Markdown

What this PR does / why we need it

The Bigtable online store hardcodes two values that govern its write path:

  • MUTATIONS_PER_OP = 50_000 — target mutations per MutateRows request
  • BIGTABLE_CLIENT_CONNECTION_POOL_SIZE = 10 — the ThreadPoolExecutor size used to parallelize writes in online_write_batch

On a shared Bigtable instance this is a problem: a large materialization fans its writes out across the thread pool with no way to tune the request size or concurrency, issuing an unthrottled write burst that can saturate the instance and inflate read-path tail latency for other workloads sharing it. Today the only way to soften that burst is to fork the online store.

This PR exposes both as optional BigtableOnlineStoreConfig fields:

Field Default Controls
mutations_per_write 50000 mutations per Bigtable write request
write_concurrency 10 concurrent write requests (thread-pool size) per worker

Both default to the existing module constants, so behavior is unchanged unless explicitly configured. Operators running against a shared instance can now lower either value to reduce the write load a materialization places on Bigtable, trading materialization speed for lower peak write pressure. Both are validated as PositiveInt, and online_write_batch now floors rows-per-request at 1 so a very wide feature view combined with a small mutations_per_write can't produce a zero-sized batch.

Example:

online_store:
  type: bigtable
  instance: my-instance
  mutations_per_write: 10000
  write_concurrency: 2

Which issue(s) this PR fixes

N/A — backward-compatible enhancement.

Misc

  • Adds sdk/python/tests/unit/infra/online_store/test_bigtable_online_store.py covering defaults matching the legacy constants, positive-int validation, batch chunking, the wide-feature-view floor, and the configurable thread-pool size.

The Bigtable online store hardcodes the mutations-per-write batch size
(MUTATIONS_PER_OP = 50_000) and the write thread-pool size
(BIGTABLE_CLIENT_CONNECTION_POOL_SIZE = 10). On a shared Bigtable
instance, a large materialization issues its writes as an unthrottled
burst that can saturate the instance and inflate read-path tail latency
for other workloads sharing it.

Expose both as optional BigtableOnlineStoreConfig fields,
mutations_per_write and write_concurrency, defaulting to the existing
constants so behavior is unchanged. Operators can lower either to reduce
the write load a materialization places on the instance, at the cost of
longer materialization time.

Add unit tests covering the defaults, positive-int validation, batch
chunking, the one-row-per-request floor for very wide feature views, and
the configurable thread-pool size.

Signed-off-by: Manas Bhardwaj <manas1109bhardwaj@gmail.com>
@Reactor11
Reactor11 requested a review from a team as a code owner September 18, 2026 08:40
@Reactor11

Copy link
Copy Markdown
Author

Hi @franciscojavierarceo please review this.

At our org, we want to control the number of writes and concurrency. As of now it is hard coded.

@shuchu shuchu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants