Skip to content
This repository was archived by the owner on Oct 18, 2023. It is now read-only.

Asynchronous snapshot upload - #574

Merged
Horusiath merged 8 commits into
libsql:mainfrom
Horusiath:bart/async-snapshot-upload
Sep 5, 2023
Merged

Asynchronous snapshot upload#574
Horusiath merged 8 commits into
libsql:mainfrom
Horusiath:bart/async-snapshot-upload

Conversation

@Horusiath

Copy link
Copy Markdown
Contributor

Build on top of #547

This PR introduces async DB file snapshotting. Basically when checkpoint happens, we trigger asynchronous DB file backup. If the next checkpoint happens, it first waits until previous snapshot upload completes before continuing.

Comment threadsqld/src/replication/primary/logger.rs
@psarna

Copy link
Copy Markdown
Contributor

I'm on vacation next two weeks, so just to keep priorities straight - either this PR, or #547, need to be applied, so that we're 100% sure we don't have autocheckpoint enabled on connections, and only control the checkpoints ourselves. More importantly, in the fiber that performs the checkpoint, we need to make sure that it's performed from a connection that has our custom WAL methods registered, so that we call the on_checkpoint behavior for bottomless, implemented here:

Comment threadbottomless/src/replicator.rs
@Horusiath
Horusiathforce-pushed the bart/async-snapshot-upload branch from eac46d2 to 495d77cCompareAugust 29, 2023 10:41
@Horusiath

Copy link
Copy Markdown
ContributorAuthor

@haaawk we still need unlock from @psarna to merge this.

@Horusiath
Horusiath enabled auto-merge August 29, 2023 11:10

@psarnapsarna left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like #547 is applied and turns off autocheckpointing where applicable

@Horusiath
Horusiathforce-pushed the bart/async-snapshot-upload branch from 495d77c to c9126e1CompareSeptember 4, 2023 12:38
@Horusiath
Horusiath added this pull request to the merge queueSep 4, 2023
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Sep 4, 2023
@Horusiath
Horusiath added this pull request to the merge queueSep 4, 2023
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Sep 4, 2023
@Horusiath
Horusiath added this pull request to the merge queueSep 5, 2023
Merged via the queue into libsql:main with commit 4b2e9f4Sep 5, 2023
@Horusiath
Horusiath deleted the bart/async-snapshot-upload branch September 5, 2023 06:54
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@Horusiath@psarna@haaawk