Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
G8XSU merged 2 commits into from
Apr 24, 2023
Merged

Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
79e408f
Add ListKeyVersions Api Implementation
G8XSU Apr 13, 2023
69fb119
Add ListKeyVersions Api AbstractKVStore Integration Tests
G8XSU Apr 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 51 additions & 2 deletions app/src/main/java/org/vss/impl/postgres/PostgresBackendImpl.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.inject.Singleton;
import org.jooq.DSLContext;
import org.jooq.Insert;
Expand All@@ -21,12 +22,12 @@
import org.vss.exception.ConflictException;
import org.vss.postgres.tables.records.VssDbRecord;

import static org.jooq.impl.DSL.val;
import static org.vss.postgres.tables.VssDb.VSS_DB;

@Singleton
public class PostgresBackendImpl implements KVStore {

private static final int LIST_KEY_VERSIONS_MAX_PAGE_SIZE = 100;
private final DSLContext context;

@Inject
Expand DownExpand Up@@ -127,6 +128,54 @@ private VssDbRecord buildVssRecord(String storeId, KeyValue kv) {

@Override
public ListKeyVersionsResponse listKeyVersions(ListKeyVersionsRequest request) {
throw new UnsupportedOperationException("Operation not implemented");
String storeId = request.getStoreId();
String keyPrefix = request.getKeyPrefix();
String pageToken = request.getPageToken();
int pageSize = request.hasPageSize() ? request.getPageSize() : Integer.MAX_VALUE;

// Only fetch global_version for first page.
// Fetch global_version before fetching any key_versions to ensure that,
// all current key_versions were stored at global_version or later.
Long globalVersion = null;
if (!request.hasPageToken()) {
GetObjectRequest getGlobalVersionRequest = GetObjectRequest.newBuilder()
.setStoreId(storeId)
.setKey(GLOBAL_VERSION_KEY)
.build();
globalVersion = get(getGlobalVersionRequest).getValue().getVersion();
}

List<VssDbRecord> vssDbRecords = context.select(VSS_DB.KEY, VSS_DB.VERSION).from(VSS_DB)
.where(VSS_DB.STORE_ID.eq(storeId)
.and(VSS_DB.KEY.startsWith(keyPrefix)))
.orderBy(VSS_DB.KEY)
.seek(pageToken)
.limit(Math.min(pageSize, LIST_KEY_VERSIONS_MAX_PAGE_SIZE))
.stream()
.map(record -> record.into(VssDbRecord.class))
.toList();

List<KeyValue> keyVersions = vssDbRecords.stream()
.filter(kv -> !GLOBAL_VERSION_KEY.equals(kv.getKey()))

@jkczyzjkczyzApr 20, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean the number of entries may be one less than pageSize even though there are pageSize matches when the key prefix is empty? Should we filter at the SQL level instead?

@G8XSUG8XSUApr 21, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can mean number of entries in response can be one less than pageSize.
But that shouldn't be a concern and client should not assume response to contain specific number of entries.
For e.g. max number of results in paginated response can change at anytime with no notice to client.

We already caution against this in api doc in proto:
"Caution: Clients must not assume a specific number of key_versions to be present in a page for paginated response."
Only way to know whether nextPage exists or not is by presence of nextPageToken.

@jkczyzjkczyzApr 21, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't answer my second question. :)

Should we filter at the SQL level instead?

Is there a reason not to?

@G8XSUG8XSUApr 21, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no big reason in case of SQL, i can do that. (apart from client/api-expectation and precedent)
It is just something to keep in mind that this operation might not be supported by all KV-database i.e. (list along with key not equals).

@jkczyzjkczyzApr 21, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. Feel fee to leave it as is.

.map(kv -> KeyValue.newBuilder()
.setKey(kv.getKey())
.setVersion(kv.getVersion())
.build())
.toList();

String nextPageToken = "";
if (!keyVersions.isEmpty()) {
nextPageToken = keyVersions.get(keyVersions.size() - 1).getKey();
}

ListKeyVersionsResponse.Builder responseBuilder = ListKeyVersionsResponse.newBuilder()
.addAllKeyVersions(keyVersions)
.setNextPageToken(nextPageToken);

if (Objects.nonNull(globalVersion)) {
responseBuilder.setGlobalVersion(globalVersion);
}
Comment thread
jkczyz marked this conversation as resolved.

return responseBuilder.build();
}
}
211 changes: 208 additions & 3 deletions app/src/test/java/org/vss/AbstractKVStoreIntegrationTest.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -2,14 +2,22 @@

import com.google.protobuf.ByteString;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
import org.testcontainers.shaded.org.apache.commons.lang3.StringUtils;
import org.vss.exception.ConflictException;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

Expand All@@ -28,6 +36,8 @@ void putShouldSucceedWhenSingleObjectPutOperation() {
assertThat(response.getKey(), is("k1"));
assertThat(response.getVersion(), is(2L));
assertThat(response.getValue().toStringUtf8(), is("k1v2"));

assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
}

@Test
Expand All@@ -50,6 +60,8 @@ void putShouldSucceedWhenMultiObjectPutOperation() {
assertThat(response.getKey(), is("k2"));
assertThat(response.getVersion(), is(2L));
assertThat(response.getValue().toStringUtf8(), is("k2v2"));

assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
}

@Test
Expand All@@ -59,11 +71,13 @@ void putShouldFailWhenKeyVersionMismatched() {
// global_version correctly changed but key-version conflict.
assertThrows(ConflictException.class, () -> putObjects(1L, List.of(kv("k1", "k1v2", 0))));

//Verify that values didn't change
//Verify that values didn't change
KeyValue response = getObject("k1");
assertThat(response.getKey(), is("k1"));
assertThat(response.getVersion(), is(1L));
assertThat(response.getValue().toStringUtf8(), is("k1v1"));

assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(1L));
}

@Test
Expand All@@ -78,7 +92,7 @@ void putMultiObjectShouldFailWhenSingleKeyVersionMismatched() {

assertThrows(ConflictException.class, () -> putObjects(null, second_request));

//Verify that values didn't change
//Verify that values didn't change
KeyValue response = getObject("k1");
assertThat(response.getKey(), is("k1"));
assertThat(response.getVersion(), is(1L));
Expand DownExpand Up@@ -113,6 +127,8 @@ void putShouldSucceedWhenNoGlobalVersionIsGiven() {
assertThat(response.getKey(), is("k1"));
assertThat(response.getVersion(), is(2L));
assertThat(response.getValue().toStringUtf8(), is("k1v2"));

assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(0L));
}

@Test
Expand DownExpand Up@@ -163,6 +179,177 @@ void getShouldReturnCorrectValueWhenKeyExists() {
assertThat(response.getValue().toStringUtf8(), is("k3v1"));
}

@Test
void listShouldReturnPaginatedResponse() {

int totalKvObjects = 1000;
for (int i = 0; i < totalKvObjects; i++) {
putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
}
// Overwrite k1 once and k2 twice.
putObjects(1000L, List.of(kv("k1", "k1v2", 1)));
putObjects(1001L, List.of(kv("k2", "k2v2", 1)));
putObjects(1002L, List.of(kv("k2", "k2v3", 2)));

ListKeyVersionsResponse previousPage = null;
List<KeyValue> allKeyVersions = new ArrayList<>();

while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
ListKeyVersionsResponse currentPage;

if (previousPage == null) {
currentPage = list(null, null, null);
Comment thread
jkczyz marked this conversation as resolved.

// Ensure first page contains correct global version
assertThat(currentPage.getGlobalVersion(), is(1003L));
} else {
String nextPageToken = previousPage.getNextPageToken();
currentPage = list(nextPageToken, null, null);

// Ensure pages after first page dont contain global version.
assertThat(currentPage.hasGlobalVersion(), is(false));
}

allKeyVersions.addAll(currentPage.getKeyVersionsList());
previousPage = currentPage;
}

// Ensure page results don't intersect/duplicate and return complete view.
Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
.collect(Collectors.toSet());
assertThat(uniqueKeys.size(), is(totalKvObjects));

// Ensure that we don't return "vss_global_version" as part of keys.
assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));

// Ensure correct key version for k1
KeyValue k1_response =
allKeyVersions.stream().filter(kv -> "k1".equals(kv.getKey())).findFirst().get();
assertThat(k1_response.getKey(), is("k1"));
assertThat(k1_response.getVersion(), is(2L));
assertThat(k1_response.getValue().toStringUtf8(), is(""));

// Ensure correct key version for k2
KeyValue k2_response =
allKeyVersions.stream().filter(kv -> "k2".equals(kv.getKey())).findFirst().get();
assertThat(k2_response.getKey(), is("k2"));
assertThat(k2_response.getVersion(), is(3L));
assertThat(k2_response.getValue().toStringUtf8(), is(""));
}

@Test
void listShouldHonourPageSizeAndKeyPrefixIfProvided() {
int totalKvObjects = 20;
int pageSize = 5;
for (int i = 0; i < totalKvObjects; i++) {
putObjects((long) i, List.of(kv(i + "k", "k1v1", 0)));
}

ListKeyVersionsResponse previousPage = null;
List<KeyValue> allKeyVersions = new ArrayList<>();
String keyPrefix = "1";

while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
ListKeyVersionsResponse currentPage;

if (previousPage == null) {
currentPage = list(null, pageSize, keyPrefix);
} else {
String nextPageToken = previousPage.getNextPageToken();
currentPage = list(nextPageToken, pageSize, keyPrefix);
}

allKeyVersions.addAll(currentPage.getKeyVersionsList());

// Each page.size() is less than or equal to pageSize in request.
assertThat(currentPage.getKeyVersionsList().size(), lessThanOrEqualTo(pageSize));
previousPage = currentPage;
}

Set<String> uniqueKeys =
allKeyVersions.stream().map(KeyValue::getKey).collect(Collectors.toSet());

// Returns keys only with provided keyPrefix
assertThat(uniqueKeys.size(), is(11));
assertThat(uniqueKeys,
is(Set.of("1k", "10k", "11k", "12k", "13k", "14k", "15k", "16k", "17k", "18k", "19k")));
}

@Test
void listShouldReturnZeroGlobalVersionWhenGlobalVersioningNotEnabled() {
int totalKvObjects = 1000;
for (int i = 0; i < totalKvObjects; i++) {
putObjects(null, List.of(kv("k" + i, "k1v1", 0)));
}

ListKeyVersionsResponse previousPage = null;
List<KeyValue> allKeyVersions = new ArrayList<>();

while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
ListKeyVersionsResponse currentPage;

if (previousPage == null) {
currentPage = list(null, null, null);

// Ensure first page returns global version as ZERO
assertThat(currentPage.getGlobalVersion(), is(0L));
} else {
String nextPageToken = previousPage.getNextPageToken();
currentPage = list(nextPageToken, null, null);

// Ensure pages after first page do not contain global version.
assertThat(currentPage.hasGlobalVersion(), is(false));
}

allKeyVersions.addAll(currentPage.getKeyVersionsList());
previousPage = currentPage;
}
// Returns complete view.
Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
.collect(Collectors.toSet());
assertThat(uniqueKeys.size(), is(totalKvObjects));

// Ensure that we don't return "vss_global_version" as part of keys.
assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));
}

@Test
void listShouldLimitMaxPageSize() {

int totalKvObjects = 10000;

// Each implementation is free to choose its own max_page_size but there should be a reasonable max
// keeping scalability and performance in mind.
// Revisit this test case if some implementation wants to support higher page size.
int vssArbitraryPageSizeMax = 3000;
Comment thread
jkczyz marked this conversation as resolved.

for (int i = 0; i < totalKvObjects; i++) {
putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
}

ListKeyVersionsResponse previousPage = null;
List<KeyValue> allKeyVersions = new ArrayList<>();

while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
ListKeyVersionsResponse currentPage;

if (previousPage == null) {
currentPage = list(null, null, null);
} else {
String nextPageToken = previousPage.getNextPageToken();
currentPage = list(nextPageToken, null, null);
}

allKeyVersions.addAll(currentPage.getKeyVersionsList());

// Each page.size() is less than MAX_PAGE_SIZE
assertThat(currentPage.getKeyVersionsList().size(), lessThan(vssArbitraryPageSizeMax));
previousPage = currentPage;
}

assertThat(allKeyVersions.size(), is(totalKvObjects));
}

private KeyValue getObject(String key) {
GetObjectRequest getRequest = GetObjectRequest.newBuilder()
.setStoreId(STORE_ID)
Expand All@@ -171,7 +358,7 @@ private KeyValue getObject(String key) {
return this.kvStore.get(getRequest).getValue();
}

private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
private void putObjects(@Nullable Long globalVersion, List<KeyValue> keyValues) {
PutObjectRequest.Builder putObjectRequestBuilder = PutObjectRequest.newBuilder()
.setStoreId(STORE_ID)
.addAllTransactionItems(keyValues);
Expand All@@ -183,6 +370,24 @@ private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
this.kvStore.put(putObjectRequestBuilder.build());
}

private ListKeyVersionsResponse list(@Nullable String nextPageToken, @Nullable Integer pageSize,
@Nullable String keyPrefix) {
ListKeyVersionsRequest.Builder listRequestBuilder = ListKeyVersionsRequest.newBuilder()
.setStoreId(STORE_ID);

if (StringUtils.isNotBlank(nextPageToken)) {
listRequestBuilder.setPageToken(nextPageToken);
}
if (pageSize != null) {
listRequestBuilder.setPageSize(pageSize);
}
if (StringUtils.isNotBlank(keyPrefix)) {
listRequestBuilder.setKeyPrefix(keyPrefix);
}

return this.kvStore.listKeyVersions(listRequestBuilder.build());
}

private KeyValue kv(String key, String value, int version) {
return KeyValue.newBuilder().setKey(key).setVersion(version).setValue(
ByteString.copyFrom(value.getBytes(
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

G8XSU
merged 2 commits into

from

Apr 24, 2023
 Merged


Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same
#8
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
79e408f

Add ListKeyVersions Api Implementation


G8XSU Apr 13, 2023
69fb119

Add ListKeyVersions Api AbstractKVStore Integration Tests


G8XSU Apr 21, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


53 changes: 51 additions & 2 deletions
app/src/main/java/org/vss/impl/postgres/PostgresBackendImpl.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.inject.Singleton;
import org.jooq.DSLContext;
import org.jooq.Insert;
Expand All@@ -21,12 +22,12 @@
import org.vss.exception.ConflictException;
import org.vss.postgres.tables.records.VssDbRecord;

import static org.jooq.impl.DSL.val;
import static org.vss.postgres.tables.VssDb.VSS_DB;

@Singleton
public class PostgresBackendImpl implements KVStore {

 private static final int LIST_KEY_VERSIONS_MAX_PAGE_SIZE = 100;
 private final DSLContext context;

 @Inject
Expand DownExpand Up@@ -127,6 +128,54 @@ private VssDbRecord buildVssRecord(String storeId, KeyValue kv) {

 @Override
 public ListKeyVersionsResponse listKeyVersions(ListKeyVersionsRequest request) {
 throw new UnsupportedOperationException("Operation not implemented");
 String storeId = request.getStoreId();
 String keyPrefix = request.getKeyPrefix();
 String pageToken = request.getPageToken();
 int pageSize = request.hasPageSize() ? request.getPageSize() : Integer.MAX_VALUE;

 // Only fetch global_version for first page.
 // Fetch global_version before fetching any key_versions to ensure that,
 // all current key_versions were stored at global_version or later.
 Long globalVersion = null;
 if (!request.hasPageToken()) {
 GetObjectRequest getGlobalVersionRequest = GetObjectRequest.newBuilder()
 .setStoreId(storeId)
 .setKey(GLOBAL_VERSION_KEY)
 .build();
 globalVersion = get(getGlobalVersionRequest).getValue().getVersion();
 }

 List<VssDbRecord> vssDbRecords = context.select(VSS_DB.KEY, VSS_DB.VERSION).from(VSS_DB)
 .where(VSS_DB.STORE_ID.eq(storeId)
 .and(VSS_DB.KEY.startsWith(keyPrefix)))
 .orderBy(VSS_DB.KEY)
 .seek(pageToken)
 .limit(Math.min(pageSize, LIST_KEY_VERSIONS_MAX_PAGE_SIZE))
 .stream()
 .map(record -> record.into(VssDbRecord.class))
 .toList();

 List<KeyValue> keyVersions = vssDbRecords.stream()
 .filter(kv -> !GLOBAL_VERSION_KEY.equals(kv.getKey()))
@jkczyzjkczyzApr 20, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean the number of entries may be one less than pageSize even though there are pageSize matches when the key prefix is empty? Should we filter at the SQL level instead?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can mean number of entries in response can be one less than pageSize.

But that shouldn't be a concern and client should not assume response to contain specific number of entries.

For e.g. max number of results in paginated response can change at anytime with no notice to client.
We already caution against this in api doc in proto:

"Caution: Clients must not assume a specific number of key_versions to be present in a page for paginated response."

Only way to know whether nextPage exists or not is by presence of nextPageToken.
@jkczyzjkczyzApr 21, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't answer my second question. :)
Should we filter at the SQL level instead?
Is there a reason not to?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no big reason in case of SQL, i can do that. (apart from client/api-expectation and precedent)

It is just something to keep in mind that this operation might not be supported by all KV-database i.e. (list along with key not equals).
@jkczyzjkczyzApr 21, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. Feel fee to leave it as is.
 .map(kv -> KeyValue.newBuilder()
 .setKey(kv.getKey())
 .setVersion(kv.getVersion())
 .build())
 .toList();

 String nextPageToken = "";
 if (!keyVersions.isEmpty()) {
 nextPageToken = keyVersions.get(keyVersions.size() - 1).getKey();
 }

 ListKeyVersionsResponse.Builder responseBuilder = ListKeyVersionsResponse.newBuilder()
 .addAllKeyVersions(keyVersions)
 .setNextPageToken(nextPageToken);

 if (Objects.nonNull(globalVersion)) {
 responseBuilder.setGlobalVersion(globalVersion);
 }
Comment thread
jkczyz marked this conversation as resolved.


 return responseBuilder.build();
 }
}

211 changes: 208 additions & 3 deletions
app/src/test/java/org/vss/AbstractKVStoreIntegrationTest.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,14 +2,22 @@

import com.google.protobuf.ByteString;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
import org.testcontainers.shaded.org.apache.commons.lang3.StringUtils;
import org.vss.exception.ConflictException;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

Expand All@@ -28,6 +36,8 @@ void putShouldSucceedWhenSingleObjectPutOperation() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -50,6 +60,8 @@ void putShouldSucceedWhenMultiObjectPutOperation() {
 assertThat(response.getKey(), is("k2"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k2v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -59,11 +71,13 @@ void putShouldFailWhenKeyVersionMismatched() {
 // global_version correctly changed but key-version conflict.
 assertThrows(ConflictException.class, () -> putObjects(1L, List.of(kv("k1", "k1v2", 0))));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
 assertThat(response.getValue().toStringUtf8(), is("k1v1"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(1L));
 }

 @Test
Expand All@@ -78,7 +92,7 @@ void putMultiObjectShouldFailWhenSingleKeyVersionMismatched() {

 assertThrows(ConflictException.class, () -> putObjects(null, second_request));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
Expand DownExpand Up@@ -113,6 +127,8 @@ void putShouldSucceedWhenNoGlobalVersionIsGiven() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(0L));
 }

 @Test
Expand DownExpand Up@@ -163,6 +179,177 @@ void getShouldReturnCorrectValueWhenKeyExists() {
 assertThat(response.getValue().toStringUtf8(), is("k3v1"));
 }

 @Test
 void listShouldReturnPaginatedResponse() {

 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }
 // Overwrite k1 once and k2 twice.
 putObjects(1000L, List.of(kv("k1", "k1v2", 1)));
 putObjects(1001L, List.of(kv("k2", "k2v2", 1)));
 putObjects(1002L, List.of(kv("k2", "k2v3", 2)));

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
Comment thread
jkczyz marked this conversation as resolved.


 // Ensure first page contains correct global version
 assertThat(currentPage.getGlobalVersion(), is(1003L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page dont contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }

 // Ensure page results don't intersect/duplicate and return complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));

 // Ensure correct key version for k1
 KeyValue k1_response =
 allKeyVersions.stream().filter(kv -> "k1".equals(kv.getKey())).findFirst().get();
 assertThat(k1_response.getKey(), is("k1"));
 assertThat(k1_response.getVersion(), is(2L));
 assertThat(k1_response.getValue().toStringUtf8(), is(""));

 // Ensure correct key version for k2
 KeyValue k2_response =
 allKeyVersions.stream().filter(kv -> "k2".equals(kv.getKey())).findFirst().get();
 assertThat(k2_response.getKey(), is("k2"));
 assertThat(k2_response.getVersion(), is(3L));
 assertThat(k2_response.getValue().toStringUtf8(), is(""));
 }

 @Test
 void listShouldHonourPageSizeAndKeyPrefixIfProvided() {
 int totalKvObjects = 20;
 int pageSize = 5;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv(i + "k", "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();
 String keyPrefix = "1";

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, pageSize, keyPrefix);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, pageSize, keyPrefix);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than or equal to pageSize in request.
 assertThat(currentPage.getKeyVersionsList().size(), lessThanOrEqualTo(pageSize));
 previousPage = currentPage;
 }

 Set<String> uniqueKeys =
 allKeyVersions.stream().map(KeyValue::getKey).collect(Collectors.toSet());

 // Returns keys only with provided keyPrefix
 assertThat(uniqueKeys.size(), is(11));
 assertThat(uniqueKeys,
 is(Set.of("1k", "10k", "11k", "12k", "13k", "14k", "15k", "16k", "17k", "18k", "19k")));
 }

 @Test
 void listShouldReturnZeroGlobalVersionWhenGlobalVersioningNotEnabled() {
 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects(null, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);

 // Ensure first page returns global version as ZERO
 assertThat(currentPage.getGlobalVersion(), is(0L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page do not contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }
 // Returns complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));
 }

 @Test
 void listShouldLimitMaxPageSize() {

 int totalKvObjects = 10000;

 // Each implementation is free to choose its own max_page_size but there should be a reasonable max
 // keeping scalability and performance in mind.
 // Revisit this test case if some implementation wants to support higher page size.
 int vssArbitraryPageSizeMax = 3000;
Comment thread
jkczyz marked this conversation as resolved.


 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than MAX_PAGE_SIZE
 assertThat(currentPage.getKeyVersionsList().size(), lessThan(vssArbitraryPageSizeMax));
 previousPage = currentPage;
 }

 assertThat(allKeyVersions.size(), is(totalKvObjects));
 }

 private KeyValue getObject(String key) {
 GetObjectRequest getRequest = GetObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
Expand All@@ -171,7 +358,7 @@ private KeyValue getObject(String key) {
 return this.kvStore.get(getRequest).getValue();
 }

 private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 private void putObjects(@Nullable Long globalVersion, List<KeyValue> keyValues) {
 PutObjectRequest.Builder putObjectRequestBuilder = PutObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
 .addAllTransactionItems(keyValues);
Expand All@@ -183,6 +370,24 @@ private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 this.kvStore.put(putObjectRequestBuilder.build());
 }

 private ListKeyVersionsResponse list(@Nullable String nextPageToken, @Nullable Integer pageSize,
 @Nullable String keyPrefix) {
 ListKeyVersionsRequest.Builder listRequestBuilder = ListKeyVersionsRequest.newBuilder()
 .setStoreId(STORE_ID);

 if (StringUtils.isNotBlank(nextPageToken)) {
 listRequestBuilder.setPageToken(nextPageToken);
 }
 if (pageSize != null) {
 listRequestBuilder.setPageSize(pageSize);
 }
 if (StringUtils.isNotBlank(keyPrefix)) {
 listRequestBuilder.setKeyPrefix(keyPrefix);
 }

 return this.kvStore.listKeyVersions(listRequestBuilder.build());
 }

 private KeyValue kv(String key, String value, int version) {
 return KeyValue.newBuilder().setKey(key).setVersion(version).setValue(
 ByteString.copyFrom(value.getBytes(
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

G8XSU
merged 2 commits into

from

Apr 24, 2023
 Merged


Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same
#8
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
79e408f

Add ListKeyVersions Api Implementation


G8XSU Apr 13, 2023
69fb119

Add ListKeyVersions Api AbstractKVStore Integration Tests


G8XSU Apr 21, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


53 changes: 51 additions & 2 deletions
app/src/main/java/org/vss/impl/postgres/PostgresBackendImpl.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.inject.Singleton;
import org.jooq.DSLContext;
import org.jooq.Insert;
Expand All@@ -21,12 +22,12 @@
import org.vss.exception.ConflictException;
import org.vss.postgres.tables.records.VssDbRecord;

import static org.jooq.impl.DSL.val;
import static org.vss.postgres.tables.VssDb.VSS_DB;

@Singleton
public class PostgresBackendImpl implements KVStore {

 private static final int LIST_KEY_VERSIONS_MAX_PAGE_SIZE = 100;
 private final DSLContext context;

 @Inject
Expand DownExpand Up@@ -127,6 +128,54 @@ private VssDbRecord buildVssRecord(String storeId, KeyValue kv) {

 @Override
 public ListKeyVersionsResponse listKeyVersions(ListKeyVersionsRequest request) {
 throw new UnsupportedOperationException("Operation not implemented");
 String storeId = request.getStoreId();
 String keyPrefix = request.getKeyPrefix();
 String pageToken = request.getPageToken();
 int pageSize = request.hasPageSize() ? request.getPageSize() : Integer.MAX_VALUE;

 // Only fetch global_version for first page.
 // Fetch global_version before fetching any key_versions to ensure that,
 // all current key_versions were stored at global_version or later.
 Long globalVersion = null;
 if (!request.hasPageToken()) {
 GetObjectRequest getGlobalVersionRequest = GetObjectRequest.newBuilder()
 .setStoreId(storeId)
 .setKey(GLOBAL_VERSION_KEY)
 .build();
 globalVersion = get(getGlobalVersionRequest).getValue().getVersion();
 }

 List<VssDbRecord> vssDbRecords = context.select(VSS_DB.KEY, VSS_DB.VERSION).from(VSS_DB)
 .where(VSS_DB.STORE_ID.eq(storeId)
 .and(VSS_DB.KEY.startsWith(keyPrefix)))
 .orderBy(VSS_DB.KEY)
 .seek(pageToken)
 .limit(Math.min(pageSize, LIST_KEY_VERSIONS_MAX_PAGE_SIZE))
 .stream()
 .map(record -> record.into(VssDbRecord.class))
 .toList();

 List<KeyValue> keyVersions = vssDbRecords.stream()
 .filter(kv -> !GLOBAL_VERSION_KEY.equals(kv.getKey()))
@jkczyzjkczyzApr 20, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean the number of entries may be one less than pageSize even though there are pageSize matches when the key prefix is empty? Should we filter at the SQL level instead?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can mean number of entries in response can be one less than pageSize.

But that shouldn't be a concern and client should not assume response to contain specific number of entries.

For e.g. max number of results in paginated response can change at anytime with no notice to client.
We already caution against this in api doc in proto:

"Caution: Clients must not assume a specific number of key_versions to be present in a page for paginated response."

Only way to know whether nextPage exists or not is by presence of nextPageToken.
@jkczyzjkczyzApr 21, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't answer my second question. :)
Should we filter at the SQL level instead?
Is there a reason not to?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no big reason in case of SQL, i can do that. (apart from client/api-expectation and precedent)

It is just something to keep in mind that this operation might not be supported by all KV-database i.e. (list along with key not equals).
@jkczyzjkczyzApr 21, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. Feel fee to leave it as is.
 .map(kv -> KeyValue.newBuilder()
 .setKey(kv.getKey())
 .setVersion(kv.getVersion())
 .build())
 .toList();

 String nextPageToken = "";
 if (!keyVersions.isEmpty()) {
 nextPageToken = keyVersions.get(keyVersions.size() - 1).getKey();
 }

 ListKeyVersionsResponse.Builder responseBuilder = ListKeyVersionsResponse.newBuilder()
 .addAllKeyVersions(keyVersions)
 .setNextPageToken(nextPageToken);

 if (Objects.nonNull(globalVersion)) {
 responseBuilder.setGlobalVersion(globalVersion);
 }
Comment thread
jkczyz marked this conversation as resolved.


 return responseBuilder.build();
 }
}

211 changes: 208 additions & 3 deletions
app/src/test/java/org/vss/AbstractKVStoreIntegrationTest.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,14 +2,22 @@

import com.google.protobuf.ByteString;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
import org.testcontainers.shaded.org.apache.commons.lang3.StringUtils;
import org.vss.exception.ConflictException;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

Expand All@@ -28,6 +36,8 @@ void putShouldSucceedWhenSingleObjectPutOperation() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -50,6 +60,8 @@ void putShouldSucceedWhenMultiObjectPutOperation() {
 assertThat(response.getKey(), is("k2"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k2v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -59,11 +71,13 @@ void putShouldFailWhenKeyVersionMismatched() {
 // global_version correctly changed but key-version conflict.
 assertThrows(ConflictException.class, () -> putObjects(1L, List.of(kv("k1", "k1v2", 0))));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
 assertThat(response.getValue().toStringUtf8(), is("k1v1"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(1L));
 }

 @Test
Expand All@@ -78,7 +92,7 @@ void putMultiObjectShouldFailWhenSingleKeyVersionMismatched() {

 assertThrows(ConflictException.class, () -> putObjects(null, second_request));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
Expand DownExpand Up@@ -113,6 +127,8 @@ void putShouldSucceedWhenNoGlobalVersionIsGiven() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(0L));
 }

 @Test
Expand DownExpand Up@@ -163,6 +179,177 @@ void getShouldReturnCorrectValueWhenKeyExists() {
 assertThat(response.getValue().toStringUtf8(), is("k3v1"));
 }

 @Test
 void listShouldReturnPaginatedResponse() {

 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }
 // Overwrite k1 once and k2 twice.
 putObjects(1000L, List.of(kv("k1", "k1v2", 1)));
 putObjects(1001L, List.of(kv("k2", "k2v2", 1)));
 putObjects(1002L, List.of(kv("k2", "k2v3", 2)));

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
Comment thread
jkczyz marked this conversation as resolved.


 // Ensure first page contains correct global version
 assertThat(currentPage.getGlobalVersion(), is(1003L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page dont contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }

 // Ensure page results don't intersect/duplicate and return complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));

 // Ensure correct key version for k1
 KeyValue k1_response =
 allKeyVersions.stream().filter(kv -> "k1".equals(kv.getKey())).findFirst().get();
 assertThat(k1_response.getKey(), is("k1"));
 assertThat(k1_response.getVersion(), is(2L));
 assertThat(k1_response.getValue().toStringUtf8(), is(""));

 // Ensure correct key version for k2
 KeyValue k2_response =
 allKeyVersions.stream().filter(kv -> "k2".equals(kv.getKey())).findFirst().get();
 assertThat(k2_response.getKey(), is("k2"));
 assertThat(k2_response.getVersion(), is(3L));
 assertThat(k2_response.getValue().toStringUtf8(), is(""));
 }

 @Test
 void listShouldHonourPageSizeAndKeyPrefixIfProvided() {
 int totalKvObjects = 20;
 int pageSize = 5;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv(i + "k", "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();
 String keyPrefix = "1";

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, pageSize, keyPrefix);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, pageSize, keyPrefix);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than or equal to pageSize in request.
 assertThat(currentPage.getKeyVersionsList().size(), lessThanOrEqualTo(pageSize));
 previousPage = currentPage;
 }

 Set<String> uniqueKeys =
 allKeyVersions.stream().map(KeyValue::getKey).collect(Collectors.toSet());

 // Returns keys only with provided keyPrefix
 assertThat(uniqueKeys.size(), is(11));
 assertThat(uniqueKeys,
 is(Set.of("1k", "10k", "11k", "12k", "13k", "14k", "15k", "16k", "17k", "18k", "19k")));
 }

 @Test
 void listShouldReturnZeroGlobalVersionWhenGlobalVersioningNotEnabled() {
 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects(null, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);

 // Ensure first page returns global version as ZERO
 assertThat(currentPage.getGlobalVersion(), is(0L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page do not contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }
 // Returns complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));
 }

 @Test
 void listShouldLimitMaxPageSize() {

 int totalKvObjects = 10000;

 // Each implementation is free to choose its own max_page_size but there should be a reasonable max
 // keeping scalability and performance in mind.
 // Revisit this test case if some implementation wants to support higher page size.
 int vssArbitraryPageSizeMax = 3000;
Comment thread
jkczyz marked this conversation as resolved.


 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than MAX_PAGE_SIZE
 assertThat(currentPage.getKeyVersionsList().size(), lessThan(vssArbitraryPageSizeMax));
 previousPage = currentPage;
 }

 assertThat(allKeyVersions.size(), is(totalKvObjects));
 }

 private KeyValue getObject(String key) {
 GetObjectRequest getRequest = GetObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
Expand All@@ -171,7 +358,7 @@ private KeyValue getObject(String key) {
 return this.kvStore.get(getRequest).getValue();
 }

 private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 private void putObjects(@Nullable Long globalVersion, List<KeyValue> keyValues) {
 PutObjectRequest.Builder putObjectRequestBuilder = PutObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
 .addAllTransactionItems(keyValues);
Expand All@@ -183,6 +370,24 @@ private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 this.kvStore.put(putObjectRequestBuilder.build());
 }

 private ListKeyVersionsResponse list(@Nullable String nextPageToken, @Nullable Integer pageSize,
 @Nullable String keyPrefix) {
 ListKeyVersionsRequest.Builder listRequestBuilder = ListKeyVersionsRequest.newBuilder()
 .setStoreId(STORE_ID);

 if (StringUtils.isNotBlank(nextPageToken)) {
 listRequestBuilder.setPageToken(nextPageToken);
 }
 if (pageSize != null) {
 listRequestBuilder.setPageSize(pageSize);
 }
 if (StringUtils.isNotBlank(keyPrefix)) {
 listRequestBuilder.setKeyPrefix(keyPrefix);
 }

 return this.kvStore.listKeyVersions(listRequestBuilder.build());
 }

 private KeyValue kv(String key, String value, int version) {
 return KeyValue.newBuilder().setKey(key).setVersion(version).setValue(
 ByteString.copyFrom(value.getBytes(
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

G8XSU
merged 2 commits into

from

Apr 24, 2023
 Merged


Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same
#8
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
79e408f

Add ListKeyVersions Api Implementation


G8XSU Apr 13, 2023
69fb119

Add ListKeyVersions Api AbstractKVStore Integration Tests


G8XSU Apr 21, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


53 changes: 51 additions & 2 deletions
app/src/main/java/org/vss/impl/postgres/PostgresBackendImpl.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.inject.Singleton;
import org.jooq.DSLContext;
import org.jooq.Insert;
Expand All@@ -21,12 +22,12 @@
import org.vss.exception.ConflictException;
import org.vss.postgres.tables.records.VssDbRecord;

import static org.jooq.impl.DSL.val;
import static org.vss.postgres.tables.VssDb.VSS_DB;

@Singleton
public class PostgresBackendImpl implements KVStore {

 private static final int LIST_KEY_VERSIONS_MAX_PAGE_SIZE = 100;
 private final DSLContext context;

 @Inject
Expand DownExpand Up@@ -127,6 +128,54 @@ private VssDbRecord buildVssRecord(String storeId, KeyValue kv) {

 @Override
 public ListKeyVersionsResponse listKeyVersions(ListKeyVersionsRequest request) {
 throw new UnsupportedOperationException("Operation not implemented");
 String storeId = request.getStoreId();
 String keyPrefix = request.getKeyPrefix();
 String pageToken = request.getPageToken();
 int pageSize = request.hasPageSize() ? request.getPageSize() : Integer.MAX_VALUE;

 // Only fetch global_version for first page.
 // Fetch global_version before fetching any key_versions to ensure that,
 // all current key_versions were stored at global_version or later.
 Long globalVersion = null;
 if (!request.hasPageToken()) {
 GetObjectRequest getGlobalVersionRequest = GetObjectRequest.newBuilder()
 .setStoreId(storeId)
 .setKey(GLOBAL_VERSION_KEY)
 .build();
 globalVersion = get(getGlobalVersionRequest).getValue().getVersion();
 }

 List<VssDbRecord> vssDbRecords = context.select(VSS_DB.KEY, VSS_DB.VERSION).from(VSS_DB)
 .where(VSS_DB.STORE_ID.eq(storeId)
 .and(VSS_DB.KEY.startsWith(keyPrefix)))
 .orderBy(VSS_DB.KEY)
 .seek(pageToken)
 .limit(Math.min(pageSize, LIST_KEY_VERSIONS_MAX_PAGE_SIZE))
 .stream()
 .map(record -> record.into(VssDbRecord.class))
 .toList();

 List<KeyValue> keyVersions = vssDbRecords.stream()
 .filter(kv -> !GLOBAL_VERSION_KEY.equals(kv.getKey()))
@jkczyzjkczyzApr 20, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean the number of entries may be one less than pageSize even though there are pageSize matches when the key prefix is empty? Should we filter at the SQL level instead?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can mean number of entries in response can be one less than pageSize.

But that shouldn't be a concern and client should not assume response to contain specific number of entries.

For e.g. max number of results in paginated response can change at anytime with no notice to client.
We already caution against this in api doc in proto:

"Caution: Clients must not assume a specific number of key_versions to be present in a page for paginated response."

Only way to know whether nextPage exists or not is by presence of nextPageToken.
@jkczyzjkczyzApr 21, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't answer my second question. :)
Should we filter at the SQL level instead?
Is there a reason not to?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no big reason in case of SQL, i can do that. (apart from client/api-expectation and precedent)

It is just something to keep in mind that this operation might not be supported by all KV-database i.e. (list along with key not equals).
@jkczyzjkczyzApr 21, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. Feel fee to leave it as is.
 .map(kv -> KeyValue.newBuilder()
 .setKey(kv.getKey())
 .setVersion(kv.getVersion())
 .build())
 .toList();

 String nextPageToken = "";
 if (!keyVersions.isEmpty()) {
 nextPageToken = keyVersions.get(keyVersions.size() - 1).getKey();
 }

 ListKeyVersionsResponse.Builder responseBuilder = ListKeyVersionsResponse.newBuilder()
 .addAllKeyVersions(keyVersions)
 .setNextPageToken(nextPageToken);

 if (Objects.nonNull(globalVersion)) {
 responseBuilder.setGlobalVersion(globalVersion);
 }
Comment thread
jkczyz marked this conversation as resolved.


 return responseBuilder.build();
 }
}

211 changes: 208 additions & 3 deletions
app/src/test/java/org/vss/AbstractKVStoreIntegrationTest.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,14 +2,22 @@

import com.google.protobuf.ByteString;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
import org.testcontainers.shaded.org.apache.commons.lang3.StringUtils;
import org.vss.exception.ConflictException;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

Expand All@@ -28,6 +36,8 @@ void putShouldSucceedWhenSingleObjectPutOperation() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -50,6 +60,8 @@ void putShouldSucceedWhenMultiObjectPutOperation() {
 assertThat(response.getKey(), is("k2"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k2v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -59,11 +71,13 @@ void putShouldFailWhenKeyVersionMismatched() {
 // global_version correctly changed but key-version conflict.
 assertThrows(ConflictException.class, () -> putObjects(1L, List.of(kv("k1", "k1v2", 0))));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
 assertThat(response.getValue().toStringUtf8(), is("k1v1"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(1L));
 }

 @Test
Expand All@@ -78,7 +92,7 @@ void putMultiObjectShouldFailWhenSingleKeyVersionMismatched() {

 assertThrows(ConflictException.class, () -> putObjects(null, second_request));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
Expand DownExpand Up@@ -113,6 +127,8 @@ void putShouldSucceedWhenNoGlobalVersionIsGiven() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(0L));
 }

 @Test
Expand DownExpand Up@@ -163,6 +179,177 @@ void getShouldReturnCorrectValueWhenKeyExists() {
 assertThat(response.getValue().toStringUtf8(), is("k3v1"));
 }

 @Test
 void listShouldReturnPaginatedResponse() {

 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }
 // Overwrite k1 once and k2 twice.
 putObjects(1000L, List.of(kv("k1", "k1v2", 1)));
 putObjects(1001L, List.of(kv("k2", "k2v2", 1)));
 putObjects(1002L, List.of(kv("k2", "k2v3", 2)));

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
Comment thread
jkczyz marked this conversation as resolved.


 // Ensure first page contains correct global version
 assertThat(currentPage.getGlobalVersion(), is(1003L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page dont contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }

 // Ensure page results don't intersect/duplicate and return complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));

 // Ensure correct key version for k1
 KeyValue k1_response =
 allKeyVersions.stream().filter(kv -> "k1".equals(kv.getKey())).findFirst().get();
 assertThat(k1_response.getKey(), is("k1"));
 assertThat(k1_response.getVersion(), is(2L));
 assertThat(k1_response.getValue().toStringUtf8(), is(""));

 // Ensure correct key version for k2
 KeyValue k2_response =
 allKeyVersions.stream().filter(kv -> "k2".equals(kv.getKey())).findFirst().get();
 assertThat(k2_response.getKey(), is("k2"));
 assertThat(k2_response.getVersion(), is(3L));
 assertThat(k2_response.getValue().toStringUtf8(), is(""));
 }

 @Test
 void listShouldHonourPageSizeAndKeyPrefixIfProvided() {
 int totalKvObjects = 20;
 int pageSize = 5;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv(i + "k", "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();
 String keyPrefix = "1";

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, pageSize, keyPrefix);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, pageSize, keyPrefix);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than or equal to pageSize in request.
 assertThat(currentPage.getKeyVersionsList().size(), lessThanOrEqualTo(pageSize));
 previousPage = currentPage;
 }

 Set<String> uniqueKeys =
 allKeyVersions.stream().map(KeyValue::getKey).collect(Collectors.toSet());

 // Returns keys only with provided keyPrefix
 assertThat(uniqueKeys.size(), is(11));
 assertThat(uniqueKeys,
 is(Set.of("1k", "10k", "11k", "12k", "13k", "14k", "15k", "16k", "17k", "18k", "19k")));
 }

 @Test
 void listShouldReturnZeroGlobalVersionWhenGlobalVersioningNotEnabled() {
 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects(null, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);

 // Ensure first page returns global version as ZERO
 assertThat(currentPage.getGlobalVersion(), is(0L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page do not contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }
 // Returns complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));
 }

 @Test
 void listShouldLimitMaxPageSize() {

 int totalKvObjects = 10000;

 // Each implementation is free to choose its own max_page_size but there should be a reasonable max
 // keeping scalability and performance in mind.
 // Revisit this test case if some implementation wants to support higher page size.
 int vssArbitraryPageSizeMax = 3000;
Comment thread
jkczyz marked this conversation as resolved.


 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than MAX_PAGE_SIZE
 assertThat(currentPage.getKeyVersionsList().size(), lessThan(vssArbitraryPageSizeMax));
 previousPage = currentPage;
 }

 assertThat(allKeyVersions.size(), is(totalKvObjects));
 }

 private KeyValue getObject(String key) {
 GetObjectRequest getRequest = GetObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
Expand All@@ -171,7 +358,7 @@ private KeyValue getObject(String key) {
 return this.kvStore.get(getRequest).getValue();
 }

 private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 private void putObjects(@Nullable Long globalVersion, List<KeyValue> keyValues) {
 PutObjectRequest.Builder putObjectRequestBuilder = PutObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
 .addAllTransactionItems(keyValues);
Expand All@@ -183,6 +370,24 @@ private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 this.kvStore.put(putObjectRequestBuilder.build());
 }

 private ListKeyVersionsResponse list(@Nullable String nextPageToken, @Nullable Integer pageSize,
 @Nullable String keyPrefix) {
 ListKeyVersionsRequest.Builder listRequestBuilder = ListKeyVersionsRequest.newBuilder()
 .setStoreId(STORE_ID);

 if (StringUtils.isNotBlank(nextPageToken)) {
 listRequestBuilder.setPageToken(nextPageToken);
 }
 if (pageSize != null) {
 listRequestBuilder.setPageSize(pageSize);
 }
 if (StringUtils.isNotBlank(keyPrefix)) {
 listRequestBuilder.setKeyPrefix(keyPrefix);
 }

 return this.kvStore.listKeyVersions(listRequestBuilder.build());
 }

 private KeyValue kv(String key, String value, int version) {
 return KeyValue.newBuilder().setKey(key).setVersion(version).setValue(
 ByteString.copyFrom(value.getBytes(
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

G8XSU
merged 2 commits into

from

Apr 24, 2023
 Merged


Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same
#8
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
79e408f

Add ListKeyVersions Api Implementation


G8XSU Apr 13, 2023
69fb119

Add ListKeyVersions Api AbstractKVStore Integration Tests


G8XSU Apr 21, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


53 changes: 51 additions & 2 deletions
app/src/main/java/org/vss/impl/postgres/PostgresBackendImpl.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.inject.Singleton;
import org.jooq.DSLContext;
import org.jooq.Insert;
Expand All@@ -21,12 +22,12 @@
import org.vss.exception.ConflictException;
import org.vss.postgres.tables.records.VssDbRecord;

import static org.jooq.impl.DSL.val;
import static org.vss.postgres.tables.VssDb.VSS_DB;

@Singleton
public class PostgresBackendImpl implements KVStore {

 private static final int LIST_KEY_VERSIONS_MAX_PAGE_SIZE = 100;
 private final DSLContext context;

 @Inject
Expand DownExpand Up@@ -127,6 +128,54 @@ private VssDbRecord buildVssRecord(String storeId, KeyValue kv) {

 @Override
 public ListKeyVersionsResponse listKeyVersions(ListKeyVersionsRequest request) {
 throw new UnsupportedOperationException("Operation not implemented");
 String storeId = request.getStoreId();
 String keyPrefix = request.getKeyPrefix();
 String pageToken = request.getPageToken();
 int pageSize = request.hasPageSize() ? request.getPageSize() : Integer.MAX_VALUE;

 // Only fetch global_version for first page.
 // Fetch global_version before fetching any key_versions to ensure that,
 // all current key_versions were stored at global_version or later.
 Long globalVersion = null;
 if (!request.hasPageToken()) {
 GetObjectRequest getGlobalVersionRequest = GetObjectRequest.newBuilder()
 .setStoreId(storeId)
 .setKey(GLOBAL_VERSION_KEY)
 .build();
 globalVersion = get(getGlobalVersionRequest).getValue().getVersion();
 }

 List<VssDbRecord> vssDbRecords = context.select(VSS_DB.KEY, VSS_DB.VERSION).from(VSS_DB)
 .where(VSS_DB.STORE_ID.eq(storeId)
 .and(VSS_DB.KEY.startsWith(keyPrefix)))
 .orderBy(VSS_DB.KEY)
 .seek(pageToken)
 .limit(Math.min(pageSize, LIST_KEY_VERSIONS_MAX_PAGE_SIZE))
 .stream()
 .map(record -> record.into(VssDbRecord.class))
 .toList();

 List<KeyValue> keyVersions = vssDbRecords.stream()
 .filter(kv -> !GLOBAL_VERSION_KEY.equals(kv.getKey()))
@jkczyzjkczyzApr 20, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean the number of entries may be one less than pageSize even though there are pageSize matches when the key prefix is empty? Should we filter at the SQL level instead?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can mean number of entries in response can be one less than pageSize.

But that shouldn't be a concern and client should not assume response to contain specific number of entries.

For e.g. max number of results in paginated response can change at anytime with no notice to client.
We already caution against this in api doc in proto:

"Caution: Clients must not assume a specific number of key_versions to be present in a page for paginated response."

Only way to know whether nextPage exists or not is by presence of nextPageToken.
@jkczyzjkczyzApr 21, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't answer my second question. :)
Should we filter at the SQL level instead?
Is there a reason not to?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no big reason in case of SQL, i can do that. (apart from client/api-expectation and precedent)

It is just something to keep in mind that this operation might not be supported by all KV-database i.e. (list along with key not equals).
@jkczyzjkczyzApr 21, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. Feel fee to leave it as is.
 .map(kv -> KeyValue.newBuilder()
 .setKey(kv.getKey())
 .setVersion(kv.getVersion())
 .build())
 .toList();

 String nextPageToken = "";
 if (!keyVersions.isEmpty()) {
 nextPageToken = keyVersions.get(keyVersions.size() - 1).getKey();
 }

 ListKeyVersionsResponse.Builder responseBuilder = ListKeyVersionsResponse.newBuilder()
 .addAllKeyVersions(keyVersions)
 .setNextPageToken(nextPageToken);

 if (Objects.nonNull(globalVersion)) {
 responseBuilder.setGlobalVersion(globalVersion);
 }
Comment thread
jkczyz marked this conversation as resolved.


 return responseBuilder.build();
 }
}

211 changes: 208 additions & 3 deletions
app/src/test/java/org/vss/AbstractKVStoreIntegrationTest.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,14 +2,22 @@

import com.google.protobuf.ByteString;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
import org.testcontainers.shaded.org.apache.commons.lang3.StringUtils;
import org.vss.exception.ConflictException;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

Expand All@@ -28,6 +36,8 @@ void putShouldSucceedWhenSingleObjectPutOperation() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -50,6 +60,8 @@ void putShouldSucceedWhenMultiObjectPutOperation() {
 assertThat(response.getKey(), is("k2"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k2v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -59,11 +71,13 @@ void putShouldFailWhenKeyVersionMismatched() {
 // global_version correctly changed but key-version conflict.
 assertThrows(ConflictException.class, () -> putObjects(1L, List.of(kv("k1", "k1v2", 0))));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
 assertThat(response.getValue().toStringUtf8(), is("k1v1"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(1L));
 }

 @Test
Expand All@@ -78,7 +92,7 @@ void putMultiObjectShouldFailWhenSingleKeyVersionMismatched() {

 assertThrows(ConflictException.class, () -> putObjects(null, second_request));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
Expand DownExpand Up@@ -113,6 +127,8 @@ void putShouldSucceedWhenNoGlobalVersionIsGiven() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(0L));
 }

 @Test
Expand DownExpand Up@@ -163,6 +179,177 @@ void getShouldReturnCorrectValueWhenKeyExists() {
 assertThat(response.getValue().toStringUtf8(), is("k3v1"));
 }

 @Test
 void listShouldReturnPaginatedResponse() {

 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }
 // Overwrite k1 once and k2 twice.
 putObjects(1000L, List.of(kv("k1", "k1v2", 1)));
 putObjects(1001L, List.of(kv("k2", "k2v2", 1)));
 putObjects(1002L, List.of(kv("k2", "k2v3", 2)));

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
Comment thread
jkczyz marked this conversation as resolved.


 // Ensure first page contains correct global version
 assertThat(currentPage.getGlobalVersion(), is(1003L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page dont contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }

 // Ensure page results don't intersect/duplicate and return complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));

 // Ensure correct key version for k1
 KeyValue k1_response =
 allKeyVersions.stream().filter(kv -> "k1".equals(kv.getKey())).findFirst().get();
 assertThat(k1_response.getKey(), is("k1"));
 assertThat(k1_response.getVersion(), is(2L));
 assertThat(k1_response.getValue().toStringUtf8(), is(""));

 // Ensure correct key version for k2
 KeyValue k2_response =
 allKeyVersions.stream().filter(kv -> "k2".equals(kv.getKey())).findFirst().get();
 assertThat(k2_response.getKey(), is("k2"));
 assertThat(k2_response.getVersion(), is(3L));
 assertThat(k2_response.getValue().toStringUtf8(), is(""));
 }

 @Test
 void listShouldHonourPageSizeAndKeyPrefixIfProvided() {
 int totalKvObjects = 20;
 int pageSize = 5;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv(i + "k", "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();
 String keyPrefix = "1";

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, pageSize, keyPrefix);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, pageSize, keyPrefix);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than or equal to pageSize in request.
 assertThat(currentPage.getKeyVersionsList().size(), lessThanOrEqualTo(pageSize));
 previousPage = currentPage;
 }

 Set<String> uniqueKeys =
 allKeyVersions.stream().map(KeyValue::getKey).collect(Collectors.toSet());

 // Returns keys only with provided keyPrefix
 assertThat(uniqueKeys.size(), is(11));
 assertThat(uniqueKeys,
 is(Set.of("1k", "10k", "11k", "12k", "13k", "14k", "15k", "16k", "17k", "18k", "19k")));
 }

 @Test
 void listShouldReturnZeroGlobalVersionWhenGlobalVersioningNotEnabled() {
 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects(null, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);

 // Ensure first page returns global version as ZERO
 assertThat(currentPage.getGlobalVersion(), is(0L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page do not contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }
 // Returns complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));
 }

 @Test
 void listShouldLimitMaxPageSize() {

 int totalKvObjects = 10000;

 // Each implementation is free to choose its own max_page_size but there should be a reasonable max
 // keeping scalability and performance in mind.
 // Revisit this test case if some implementation wants to support higher page size.
 int vssArbitraryPageSizeMax = 3000;
Comment thread
jkczyz marked this conversation as resolved.


 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than MAX_PAGE_SIZE
 assertThat(currentPage.getKeyVersionsList().size(), lessThan(vssArbitraryPageSizeMax));
 previousPage = currentPage;
 }

 assertThat(allKeyVersions.size(), is(totalKvObjects));
 }

 private KeyValue getObject(String key) {
 GetObjectRequest getRequest = GetObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
Expand All@@ -171,7 +358,7 @@ private KeyValue getObject(String key) {
 return this.kvStore.get(getRequest).getValue();
 }

 private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 private void putObjects(@Nullable Long globalVersion, List<KeyValue> keyValues) {
 PutObjectRequest.Builder putObjectRequestBuilder = PutObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
 .addAllTransactionItems(keyValues);
Expand All@@ -183,6 +370,24 @@ private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 this.kvStore.put(putObjectRequestBuilder.build());
 }

 private ListKeyVersionsResponse list(@Nullable String nextPageToken, @Nullable Integer pageSize,
 @Nullable String keyPrefix) {
 ListKeyVersionsRequest.Builder listRequestBuilder = ListKeyVersionsRequest.newBuilder()
 .setStoreId(STORE_ID);

 if (StringUtils.isNotBlank(nextPageToken)) {
 listRequestBuilder.setPageToken(nextPageToken);
 }
 if (pageSize != null) {
 listRequestBuilder.setPageSize(pageSize);
 }
 if (StringUtils.isNotBlank(keyPrefix)) {
 listRequestBuilder.setKeyPrefix(keyPrefix);
 }

 return this.kvStore.listKeyVersions(listRequestBuilder.build());
 }

 private KeyValue kv(String key, String value, int version) {
 return KeyValue.newBuilder().setKey(key).setVersion(version).setValue(
 ByteString.copyFrom(value.getBytes(
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

G8XSU
merged 2 commits into

from

Apr 24, 2023
 Merged


Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same
#8
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
79e408f

Add ListKeyVersions Api Implementation


G8XSU Apr 13, 2023
69fb119

Add ListKeyVersions Api AbstractKVStore Integration Tests


G8XSU Apr 21, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


53 changes: 51 additions & 2 deletions
app/src/main/java/org/vss/impl/postgres/PostgresBackendImpl.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.inject.Singleton;
import org.jooq.DSLContext;
import org.jooq.Insert;
Expand All@@ -21,12 +22,12 @@
import org.vss.exception.ConflictException;
import org.vss.postgres.tables.records.VssDbRecord;

import static org.jooq.impl.DSL.val;
import static org.vss.postgres.tables.VssDb.VSS_DB;

@Singleton
public class PostgresBackendImpl implements KVStore {

 private static final int LIST_KEY_VERSIONS_MAX_PAGE_SIZE = 100;
 private final DSLContext context;

 @Inject
Expand DownExpand Up@@ -127,6 +128,54 @@ private VssDbRecord buildVssRecord(String storeId, KeyValue kv) {

 @Override
 public ListKeyVersionsResponse listKeyVersions(ListKeyVersionsRequest request) {
 throw new UnsupportedOperationException("Operation not implemented");
 String storeId = request.getStoreId();
 String keyPrefix = request.getKeyPrefix();
 String pageToken = request.getPageToken();
 int pageSize = request.hasPageSize() ? request.getPageSize() : Integer.MAX_VALUE;

 // Only fetch global_version for first page.
 // Fetch global_version before fetching any key_versions to ensure that,
 // all current key_versions were stored at global_version or later.
 Long globalVersion = null;
 if (!request.hasPageToken()) {
 GetObjectRequest getGlobalVersionRequest = GetObjectRequest.newBuilder()
 .setStoreId(storeId)
 .setKey(GLOBAL_VERSION_KEY)
 .build();
 globalVersion = get(getGlobalVersionRequest).getValue().getVersion();
 }

 List<VssDbRecord> vssDbRecords = context.select(VSS_DB.KEY, VSS_DB.VERSION).from(VSS_DB)
 .where(VSS_DB.STORE_ID.eq(storeId)
 .and(VSS_DB.KEY.startsWith(keyPrefix)))
 .orderBy(VSS_DB.KEY)
 .seek(pageToken)
 .limit(Math.min(pageSize, LIST_KEY_VERSIONS_MAX_PAGE_SIZE))
 .stream()
 .map(record -> record.into(VssDbRecord.class))
 .toList();

 List<KeyValue> keyVersions = vssDbRecords.stream()
 .filter(kv -> !GLOBAL_VERSION_KEY.equals(kv.getKey()))
@jkczyzjkczyzApr 20, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean the number of entries may be one less than pageSize even though there are pageSize matches when the key prefix is empty? Should we filter at the SQL level instead?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can mean number of entries in response can be one less than pageSize.

But that shouldn't be a concern and client should not assume response to contain specific number of entries.

For e.g. max number of results in paginated response can change at anytime with no notice to client.
We already caution against this in api doc in proto:

"Caution: Clients must not assume a specific number of key_versions to be present in a page for paginated response."

Only way to know whether nextPage exists or not is by presence of nextPageToken.
@jkczyzjkczyzApr 21, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't answer my second question. :)
Should we filter at the SQL level instead?
Is there a reason not to?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no big reason in case of SQL, i can do that. (apart from client/api-expectation and precedent)

It is just something to keep in mind that this operation might not be supported by all KV-database i.e. (list along with key not equals).
@jkczyzjkczyzApr 21, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. Feel fee to leave it as is.
 .map(kv -> KeyValue.newBuilder()
 .setKey(kv.getKey())
 .setVersion(kv.getVersion())
 .build())
 .toList();

 String nextPageToken = "";
 if (!keyVersions.isEmpty()) {
 nextPageToken = keyVersions.get(keyVersions.size() - 1).getKey();
 }

 ListKeyVersionsResponse.Builder responseBuilder = ListKeyVersionsResponse.newBuilder()
 .addAllKeyVersions(keyVersions)
 .setNextPageToken(nextPageToken);

 if (Objects.nonNull(globalVersion)) {
 responseBuilder.setGlobalVersion(globalVersion);
 }
Comment thread
jkczyz marked this conversation as resolved.


 return responseBuilder.build();
 }
}

211 changes: 208 additions & 3 deletions
app/src/test/java/org/vss/AbstractKVStoreIntegrationTest.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,14 +2,22 @@

import com.google.protobuf.ByteString;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
import org.testcontainers.shaded.org.apache.commons.lang3.StringUtils;
import org.vss.exception.ConflictException;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

Expand All@@ -28,6 +36,8 @@ void putShouldSucceedWhenSingleObjectPutOperation() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -50,6 +60,8 @@ void putShouldSucceedWhenMultiObjectPutOperation() {
 assertThat(response.getKey(), is("k2"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k2v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -59,11 +71,13 @@ void putShouldFailWhenKeyVersionMismatched() {
 // global_version correctly changed but key-version conflict.
 assertThrows(ConflictException.class, () -> putObjects(1L, List.of(kv("k1", "k1v2", 0))));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
 assertThat(response.getValue().toStringUtf8(), is("k1v1"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(1L));
 }

 @Test
Expand All@@ -78,7 +92,7 @@ void putMultiObjectShouldFailWhenSingleKeyVersionMismatched() {

 assertThrows(ConflictException.class, () -> putObjects(null, second_request));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
Expand DownExpand Up@@ -113,6 +127,8 @@ void putShouldSucceedWhenNoGlobalVersionIsGiven() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(0L));
 }

 @Test
Expand DownExpand Up@@ -163,6 +179,177 @@ void getShouldReturnCorrectValueWhenKeyExists() {
 assertThat(response.getValue().toStringUtf8(), is("k3v1"));
 }

 @Test
 void listShouldReturnPaginatedResponse() {

 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }
 // Overwrite k1 once and k2 twice.
 putObjects(1000L, List.of(kv("k1", "k1v2", 1)));
 putObjects(1001L, List.of(kv("k2", "k2v2", 1)));
 putObjects(1002L, List.of(kv("k2", "k2v3", 2)));

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
Comment thread
jkczyz marked this conversation as resolved.


 // Ensure first page contains correct global version
 assertThat(currentPage.getGlobalVersion(), is(1003L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page dont contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }

 // Ensure page results don't intersect/duplicate and return complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));

 // Ensure correct key version for k1
 KeyValue k1_response =
 allKeyVersions.stream().filter(kv -> "k1".equals(kv.getKey())).findFirst().get();
 assertThat(k1_response.getKey(), is("k1"));
 assertThat(k1_response.getVersion(), is(2L));
 assertThat(k1_response.getValue().toStringUtf8(), is(""));

 // Ensure correct key version for k2
 KeyValue k2_response =
 allKeyVersions.stream().filter(kv -> "k2".equals(kv.getKey())).findFirst().get();
 assertThat(k2_response.getKey(), is("k2"));
 assertThat(k2_response.getVersion(), is(3L));
 assertThat(k2_response.getValue().toStringUtf8(), is(""));
 }

 @Test
 void listShouldHonourPageSizeAndKeyPrefixIfProvided() {
 int totalKvObjects = 20;
 int pageSize = 5;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv(i + "k", "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();
 String keyPrefix = "1";

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, pageSize, keyPrefix);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, pageSize, keyPrefix);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than or equal to pageSize in request.
 assertThat(currentPage.getKeyVersionsList().size(), lessThanOrEqualTo(pageSize));
 previousPage = currentPage;
 }

 Set<String> uniqueKeys =
 allKeyVersions.stream().map(KeyValue::getKey).collect(Collectors.toSet());

 // Returns keys only with provided keyPrefix
 assertThat(uniqueKeys.size(), is(11));
 assertThat(uniqueKeys,
 is(Set.of("1k", "10k", "11k", "12k", "13k", "14k", "15k", "16k", "17k", "18k", "19k")));
 }

 @Test
 void listShouldReturnZeroGlobalVersionWhenGlobalVersioningNotEnabled() {
 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects(null, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);

 // Ensure first page returns global version as ZERO
 assertThat(currentPage.getGlobalVersion(), is(0L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page do not contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }
 // Returns complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));
 }

 @Test
 void listShouldLimitMaxPageSize() {

 int totalKvObjects = 10000;

 // Each implementation is free to choose its own max_page_size but there should be a reasonable max
 // keeping scalability and performance in mind.
 // Revisit this test case if some implementation wants to support higher page size.
 int vssArbitraryPageSizeMax = 3000;
Comment thread
jkczyz marked this conversation as resolved.


 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than MAX_PAGE_SIZE
 assertThat(currentPage.getKeyVersionsList().size(), lessThan(vssArbitraryPageSizeMax));
 previousPage = currentPage;
 }

 assertThat(allKeyVersions.size(), is(totalKvObjects));
 }

 private KeyValue getObject(String key) {
 GetObjectRequest getRequest = GetObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
Expand All@@ -171,7 +358,7 @@ private KeyValue getObject(String key) {
 return this.kvStore.get(getRequest).getValue();
 }

 private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 private void putObjects(@Nullable Long globalVersion, List<KeyValue> keyValues) {
 PutObjectRequest.Builder putObjectRequestBuilder = PutObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
 .addAllTransactionItems(keyValues);
Expand All@@ -183,6 +370,24 @@ private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 this.kvStore.put(putObjectRequestBuilder.build());
 }

 private ListKeyVersionsResponse list(@Nullable String nextPageToken, @Nullable Integer pageSize,
 @Nullable String keyPrefix) {
 ListKeyVersionsRequest.Builder listRequestBuilder = ListKeyVersionsRequest.newBuilder()
 .setStoreId(STORE_ID);

 if (StringUtils.isNotBlank(nextPageToken)) {
 listRequestBuilder.setPageToken(nextPageToken);
 }
 if (pageSize != null) {
 listRequestBuilder.setPageSize(pageSize);
 }
 if (StringUtils.isNotBlank(keyPrefix)) {
 listRequestBuilder.setKeyPrefix(keyPrefix);
 }

 return this.kvStore.listKeyVersions(listRequestBuilder.build());
 }

 private KeyValue kv(String key, String value, int version) {
 return KeyValue.newBuilder().setKey(key).setVersion(version).setValue(
 ByteString.copyFrom(value.getBytes(
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

G8XSU
merged 2 commits into

from

Apr 24, 2023
 Merged


Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same
#8
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
79e408f

Add ListKeyVersions Api Implementation


G8XSU Apr 13, 2023
69fb119

Add ListKeyVersions Api AbstractKVStore Integration Tests


G8XSU Apr 21, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


53 changes: 51 additions & 2 deletions
app/src/main/java/org/vss/impl/postgres/PostgresBackendImpl.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.inject.Singleton;
import org.jooq.DSLContext;
import org.jooq.Insert;
Expand All@@ -21,12 +22,12 @@
import org.vss.exception.ConflictException;
import org.vss.postgres.tables.records.VssDbRecord;

import static org.jooq.impl.DSL.val;
import static org.vss.postgres.tables.VssDb.VSS_DB;

@Singleton
public class PostgresBackendImpl implements KVStore {

 private static final int LIST_KEY_VERSIONS_MAX_PAGE_SIZE = 100;
 private final DSLContext context;

 @Inject
Expand DownExpand Up@@ -127,6 +128,54 @@ private VssDbRecord buildVssRecord(String storeId, KeyValue kv) {

 @Override
 public ListKeyVersionsResponse listKeyVersions(ListKeyVersionsRequest request) {
 throw new UnsupportedOperationException("Operation not implemented");
 String storeId = request.getStoreId();
 String keyPrefix = request.getKeyPrefix();
 String pageToken = request.getPageToken();
 int pageSize = request.hasPageSize() ? request.getPageSize() : Integer.MAX_VALUE;

 // Only fetch global_version for first page.
 // Fetch global_version before fetching any key_versions to ensure that,
 // all current key_versions were stored at global_version or later.
 Long globalVersion = null;
 if (!request.hasPageToken()) {
 GetObjectRequest getGlobalVersionRequest = GetObjectRequest.newBuilder()
 .setStoreId(storeId)
 .setKey(GLOBAL_VERSION_KEY)
 .build();
 globalVersion = get(getGlobalVersionRequest).getValue().getVersion();
 }

 List<VssDbRecord> vssDbRecords = context.select(VSS_DB.KEY, VSS_DB.VERSION).from(VSS_DB)
 .where(VSS_DB.STORE_ID.eq(storeId)
 .and(VSS_DB.KEY.startsWith(keyPrefix)))
 .orderBy(VSS_DB.KEY)
 .seek(pageToken)
 .limit(Math.min(pageSize, LIST_KEY_VERSIONS_MAX_PAGE_SIZE))
 .stream()
 .map(record -> record.into(VssDbRecord.class))
 .toList();

 List<KeyValue> keyVersions = vssDbRecords.stream()
 .filter(kv -> !GLOBAL_VERSION_KEY.equals(kv.getKey()))
@jkczyzjkczyzApr 20, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean the number of entries may be one less than pageSize even though there are pageSize matches when the key prefix is empty? Should we filter at the SQL level instead?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can mean number of entries in response can be one less than pageSize.

But that shouldn't be a concern and client should not assume response to contain specific number of entries.

For e.g. max number of results in paginated response can change at anytime with no notice to client.
We already caution against this in api doc in proto:

"Caution: Clients must not assume a specific number of key_versions to be present in a page for paginated response."

Only way to know whether nextPage exists or not is by presence of nextPageToken.
@jkczyzjkczyzApr 21, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't answer my second question. :)
Should we filter at the SQL level instead?
Is there a reason not to?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no big reason in case of SQL, i can do that. (apart from client/api-expectation and precedent)

It is just something to keep in mind that this operation might not be supported by all KV-database i.e. (list along with key not equals).
@jkczyzjkczyzApr 21, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. Feel fee to leave it as is.
 .map(kv -> KeyValue.newBuilder()
 .setKey(kv.getKey())
 .setVersion(kv.getVersion())
 .build())
 .toList();

 String nextPageToken = "";
 if (!keyVersions.isEmpty()) {
 nextPageToken = keyVersions.get(keyVersions.size() - 1).getKey();
 }

 ListKeyVersionsResponse.Builder responseBuilder = ListKeyVersionsResponse.newBuilder()
 .addAllKeyVersions(keyVersions)
 .setNextPageToken(nextPageToken);

 if (Objects.nonNull(globalVersion)) {
 responseBuilder.setGlobalVersion(globalVersion);
 }
Comment thread
jkczyz marked this conversation as resolved.


 return responseBuilder.build();
 }
}

211 changes: 208 additions & 3 deletions
app/src/test/java/org/vss/AbstractKVStoreIntegrationTest.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,14 +2,22 @@

import com.google.protobuf.ByteString;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
import org.testcontainers.shaded.org.apache.commons.lang3.StringUtils;
import org.vss.exception.ConflictException;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

Expand All@@ -28,6 +36,8 @@ void putShouldSucceedWhenSingleObjectPutOperation() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -50,6 +60,8 @@ void putShouldSucceedWhenMultiObjectPutOperation() {
 assertThat(response.getKey(), is("k2"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k2v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -59,11 +71,13 @@ void putShouldFailWhenKeyVersionMismatched() {
 // global_version correctly changed but key-version conflict.
 assertThrows(ConflictException.class, () -> putObjects(1L, List.of(kv("k1", "k1v2", 0))));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
 assertThat(response.getValue().toStringUtf8(), is("k1v1"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(1L));
 }

 @Test
Expand All@@ -78,7 +92,7 @@ void putMultiObjectShouldFailWhenSingleKeyVersionMismatched() {

 assertThrows(ConflictException.class, () -> putObjects(null, second_request));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
Expand DownExpand Up@@ -113,6 +127,8 @@ void putShouldSucceedWhenNoGlobalVersionIsGiven() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(0L));
 }

 @Test
Expand DownExpand Up@@ -163,6 +179,177 @@ void getShouldReturnCorrectValueWhenKeyExists() {
 assertThat(response.getValue().toStringUtf8(), is("k3v1"));
 }

 @Test
 void listShouldReturnPaginatedResponse() {

 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }
 // Overwrite k1 once and k2 twice.
 putObjects(1000L, List.of(kv("k1", "k1v2", 1)));
 putObjects(1001L, List.of(kv("k2", "k2v2", 1)));
 putObjects(1002L, List.of(kv("k2", "k2v3", 2)));

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
Comment thread
jkczyz marked this conversation as resolved.


 // Ensure first page contains correct global version
 assertThat(currentPage.getGlobalVersion(), is(1003L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page dont contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }

 // Ensure page results don't intersect/duplicate and return complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));

 // Ensure correct key version for k1
 KeyValue k1_response =
 allKeyVersions.stream().filter(kv -> "k1".equals(kv.getKey())).findFirst().get();
 assertThat(k1_response.getKey(), is("k1"));
 assertThat(k1_response.getVersion(), is(2L));
 assertThat(k1_response.getValue().toStringUtf8(), is(""));

 // Ensure correct key version for k2
 KeyValue k2_response =
 allKeyVersions.stream().filter(kv -> "k2".equals(kv.getKey())).findFirst().get();
 assertThat(k2_response.getKey(), is("k2"));
 assertThat(k2_response.getVersion(), is(3L));
 assertThat(k2_response.getValue().toStringUtf8(), is(""));
 }

 @Test
 void listShouldHonourPageSizeAndKeyPrefixIfProvided() {
 int totalKvObjects = 20;
 int pageSize = 5;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv(i + "k", "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();
 String keyPrefix = "1";

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, pageSize, keyPrefix);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, pageSize, keyPrefix);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than or equal to pageSize in request.
 assertThat(currentPage.getKeyVersionsList().size(), lessThanOrEqualTo(pageSize));
 previousPage = currentPage;
 }

 Set<String> uniqueKeys =
 allKeyVersions.stream().map(KeyValue::getKey).collect(Collectors.toSet());

 // Returns keys only with provided keyPrefix
 assertThat(uniqueKeys.size(), is(11));
 assertThat(uniqueKeys,
 is(Set.of("1k", "10k", "11k", "12k", "13k", "14k", "15k", "16k", "17k", "18k", "19k")));
 }

 @Test
 void listShouldReturnZeroGlobalVersionWhenGlobalVersioningNotEnabled() {
 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects(null, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);

 // Ensure first page returns global version as ZERO
 assertThat(currentPage.getGlobalVersion(), is(0L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page do not contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }
 // Returns complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));
 }

 @Test
 void listShouldLimitMaxPageSize() {

 int totalKvObjects = 10000;

 // Each implementation is free to choose its own max_page_size but there should be a reasonable max
 // keeping scalability and performance in mind.
 // Revisit this test case if some implementation wants to support higher page size.
 int vssArbitraryPageSizeMax = 3000;
Comment thread
jkczyz marked this conversation as resolved.


 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than MAX_PAGE_SIZE
 assertThat(currentPage.getKeyVersionsList().size(), lessThan(vssArbitraryPageSizeMax));
 previousPage = currentPage;
 }

 assertThat(allKeyVersions.size(), is(totalKvObjects));
 }

 private KeyValue getObject(String key) {
 GetObjectRequest getRequest = GetObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
Expand All@@ -171,7 +358,7 @@ private KeyValue getObject(String key) {
 return this.kvStore.get(getRequest).getValue();
 }

 private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 private void putObjects(@Nullable Long globalVersion, List<KeyValue> keyValues) {
 PutObjectRequest.Builder putObjectRequestBuilder = PutObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
 .addAllTransactionItems(keyValues);
Expand All@@ -183,6 +370,24 @@ private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 this.kvStore.put(putObjectRequestBuilder.build());
 }

 private ListKeyVersionsResponse list(@Nullable String nextPageToken, @Nullable Integer pageSize,
 @Nullable String keyPrefix) {
 ListKeyVersionsRequest.Builder listRequestBuilder = ListKeyVersionsRequest.newBuilder()
 .setStoreId(STORE_ID);

 if (StringUtils.isNotBlank(nextPageToken)) {
 listRequestBuilder.setPageToken(nextPageToken);
 }
 if (pageSize != null) {
 listRequestBuilder.setPageSize(pageSize);
 }
 if (StringUtils.isNotBlank(keyPrefix)) {
 listRequestBuilder.setKeyPrefix(keyPrefix);
 }

 return this.kvStore.listKeyVersions(listRequestBuilder.build());
 }

 private KeyValue kv(String key, String value, int version) {
 return KeyValue.newBuilder().setKey(key).setVersion(version).setValue(
 ByteString.copyFrom(value.getBytes(
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same#8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

G8XSU
merged 2 commits into

from

Apr 24, 2023
 Merged


Add Implementation for ListKeyVersions Api and AbstractKVStore tests for the same
#8
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


2 commits


Select commit
Hold shift + click to select a range
79e408f

Add ListKeyVersions Api Implementation


G8XSU Apr 13, 2023
69fb119

Add ListKeyVersions Api AbstractKVStore Integration Tests


G8XSU Apr 21, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


53 changes: 51 additions & 2 deletions
app/src/main/java/org/vss/impl/postgres/PostgresBackendImpl.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,6 +5,7 @@
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import javax.inject.Singleton;
import org.jooq.DSLContext;
import org.jooq.Insert;
Expand All@@ -21,12 +22,12 @@
import org.vss.exception.ConflictException;
import org.vss.postgres.tables.records.VssDbRecord;

import static org.jooq.impl.DSL.val;
import static org.vss.postgres.tables.VssDb.VSS_DB;

@Singleton
public class PostgresBackendImpl implements KVStore {

 private static final int LIST_KEY_VERSIONS_MAX_PAGE_SIZE = 100;
 private final DSLContext context;

 @Inject
Expand DownExpand Up@@ -127,6 +128,54 @@ private VssDbRecord buildVssRecord(String storeId, KeyValue kv) {

 @Override
 public ListKeyVersionsResponse listKeyVersions(ListKeyVersionsRequest request) {
 throw new UnsupportedOperationException("Operation not implemented");
 String storeId = request.getStoreId();
 String keyPrefix = request.getKeyPrefix();
 String pageToken = request.getPageToken();
 int pageSize = request.hasPageSize() ? request.getPageSize() : Integer.MAX_VALUE;

 // Only fetch global_version for first page.
 // Fetch global_version before fetching any key_versions to ensure that,
 // all current key_versions were stored at global_version or later.
 Long globalVersion = null;
 if (!request.hasPageToken()) {
 GetObjectRequest getGlobalVersionRequest = GetObjectRequest.newBuilder()
 .setStoreId(storeId)
 .setKey(GLOBAL_VERSION_KEY)
 .build();
 globalVersion = get(getGlobalVersionRequest).getValue().getVersion();
 }

 List<VssDbRecord> vssDbRecords = context.select(VSS_DB.KEY, VSS_DB.VERSION).from(VSS_DB)
 .where(VSS_DB.STORE_ID.eq(storeId)
 .and(VSS_DB.KEY.startsWith(keyPrefix)))
 .orderBy(VSS_DB.KEY)
 .seek(pageToken)
 .limit(Math.min(pageSize, LIST_KEY_VERSIONS_MAX_PAGE_SIZE))
 .stream()
 .map(record -> record.into(VssDbRecord.class))
 .toList();

 List<KeyValue> keyVersions = vssDbRecords.stream()
 .filter(kv -> !GLOBAL_VERSION_KEY.equals(kv.getKey()))
@jkczyzjkczyzApr 20, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this mean the number of entries may be one less than pageSize even though there are pageSize matches when the key prefix is empty? Should we filter at the SQL level instead?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it can mean number of entries in response can be one less than pageSize.

But that shouldn't be a concern and client should not assume response to contain specific number of entries.

For e.g. max number of results in paginated response can change at anytime with no notice to client.
We already caution against this in api doc in proto:

"Caution: Clients must not assume a specific number of key_versions to be present in a page for paginated response."

Only way to know whether nextPage exists or not is by presence of nextPageToken.
@jkczyzjkczyzApr 21, 2023

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That doesn't answer my second question. :)
Should we filter at the SQL level instead?
Is there a reason not to?
@G8XSUG8XSUApr 21, 2023

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no big reason in case of SQL, i can do that. (apart from client/api-expectation and precedent)

It is just something to keep in mind that this operation might not be supported by all KV-database i.e. (list along with key not equals).
@jkczyzjkczyzApr 21, 2023•

edited

Loading

Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's ok. Feel fee to leave it as is.
 .map(kv -> KeyValue.newBuilder()
 .setKey(kv.getKey())
 .setVersion(kv.getVersion())
 .build())
 .toList();

 String nextPageToken = "";
 if (!keyVersions.isEmpty()) {
 nextPageToken = keyVersions.get(keyVersions.size() - 1).getKey();
 }

 ListKeyVersionsResponse.Builder responseBuilder = ListKeyVersionsResponse.newBuilder()
 .addAllKeyVersions(keyVersions)
 .setNextPageToken(nextPageToken);

 if (Objects.nonNull(globalVersion)) {
 responseBuilder.setGlobalVersion(globalVersion);
 }
Comment thread
jkczyz marked this conversation as resolved.


 return responseBuilder.build();
 }
}

211 changes: 208 additions & 3 deletions
app/src/test/java/org/vss/AbstractKVStoreIntegrationTest.java

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -2,14 +2,22 @@

import com.google.protobuf.ByteString;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
import javax.annotation.Nullable;
import org.junit.jupiter.api.Test;
import org.testcontainers.shaded.org.apache.commons.lang3.StringUtils;
import org.vss.exception.ConflictException;

import static org.hamcrest.MatcherAssert.assertThat;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.Matchers.lessThanOrEqualTo;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

Expand All@@ -28,6 +36,8 @@ void putShouldSucceedWhenSingleObjectPutOperation() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -50,6 +60,8 @@ void putShouldSucceedWhenMultiObjectPutOperation() {
 assertThat(response.getKey(), is("k2"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k2v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(2L));
 }

 @Test
Expand All@@ -59,11 +71,13 @@ void putShouldFailWhenKeyVersionMismatched() {
 // global_version correctly changed but key-version conflict.
 assertThrows(ConflictException.class, () -> putObjects(1L, List.of(kv("k1", "k1v2", 0))));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
 assertThat(response.getValue().toStringUtf8(), is("k1v1"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(1L));
 }

 @Test
Expand All@@ -78,7 +92,7 @@ void putMultiObjectShouldFailWhenSingleKeyVersionMismatched() {

 assertThrows(ConflictException.class, () -> putObjects(null, second_request));

 //Verify that values didn't change
 //Verify that values didn't change
 KeyValue response = getObject("k1");
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(1L));
Expand DownExpand Up@@ -113,6 +127,8 @@ void putShouldSucceedWhenNoGlobalVersionIsGiven() {
 assertThat(response.getKey(), is("k1"));
 assertThat(response.getVersion(), is(2L));
 assertThat(response.getValue().toStringUtf8(), is("k1v2"));

 assertThat(getObject(KVStore.GLOBAL_VERSION_KEY).getVersion(), is(0L));
 }

 @Test
Expand DownExpand Up@@ -163,6 +179,177 @@ void getShouldReturnCorrectValueWhenKeyExists() {
 assertThat(response.getValue().toStringUtf8(), is("k3v1"));
 }

 @Test
 void listShouldReturnPaginatedResponse() {

 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }
 // Overwrite k1 once and k2 twice.
 putObjects(1000L, List.of(kv("k1", "k1v2", 1)));
 putObjects(1001L, List.of(kv("k2", "k2v2", 1)));
 putObjects(1002L, List.of(kv("k2", "k2v3", 2)));

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
Comment thread
jkczyz marked this conversation as resolved.


 // Ensure first page contains correct global version
 assertThat(currentPage.getGlobalVersion(), is(1003L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page dont contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }

 // Ensure page results don't intersect/duplicate and return complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));

 // Ensure correct key version for k1
 KeyValue k1_response =
 allKeyVersions.stream().filter(kv -> "k1".equals(kv.getKey())).findFirst().get();
 assertThat(k1_response.getKey(), is("k1"));
 assertThat(k1_response.getVersion(), is(2L));
 assertThat(k1_response.getValue().toStringUtf8(), is(""));

 // Ensure correct key version for k2
 KeyValue k2_response =
 allKeyVersions.stream().filter(kv -> "k2".equals(kv.getKey())).findFirst().get();
 assertThat(k2_response.getKey(), is("k2"));
 assertThat(k2_response.getVersion(), is(3L));
 assertThat(k2_response.getValue().toStringUtf8(), is(""));
 }

 @Test
 void listShouldHonourPageSizeAndKeyPrefixIfProvided() {
 int totalKvObjects = 20;
 int pageSize = 5;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv(i + "k", "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();
 String keyPrefix = "1";

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, pageSize, keyPrefix);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, pageSize, keyPrefix);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than or equal to pageSize in request.
 assertThat(currentPage.getKeyVersionsList().size(), lessThanOrEqualTo(pageSize));
 previousPage = currentPage;
 }

 Set<String> uniqueKeys =
 allKeyVersions.stream().map(KeyValue::getKey).collect(Collectors.toSet());

 // Returns keys only with provided keyPrefix
 assertThat(uniqueKeys.size(), is(11));
 assertThat(uniqueKeys,
 is(Set.of("1k", "10k", "11k", "12k", "13k", "14k", "15k", "16k", "17k", "18k", "19k")));
 }

 @Test
 void listShouldReturnZeroGlobalVersionWhenGlobalVersioningNotEnabled() {
 int totalKvObjects = 1000;
 for (int i = 0; i < totalKvObjects; i++) {
 putObjects(null, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);

 // Ensure first page returns global version as ZERO
 assertThat(currentPage.getGlobalVersion(), is(0L));
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);

 // Ensure pages after first page do not contain global version.
 assertThat(currentPage.hasGlobalVersion(), is(false));
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());
 previousPage = currentPage;
 }
 // Returns complete view.
 Set<String> uniqueKeys = allKeyVersions.stream().map(KeyValue::getKey).distinct()
 .collect(Collectors.toSet());
 assertThat(uniqueKeys.size(), is(totalKvObjects));

 // Ensure that we don't return "vss_global_version" as part of keys.
 assertFalse(uniqueKeys.contains(KVStore.GLOBAL_VERSION_KEY));
 }

 @Test
 void listShouldLimitMaxPageSize() {

 int totalKvObjects = 10000;

 // Each implementation is free to choose its own max_page_size but there should be a reasonable max
 // keeping scalability and performance in mind.
 // Revisit this test case if some implementation wants to support higher page size.
 int vssArbitraryPageSizeMax = 3000;
Comment thread
jkczyz marked this conversation as resolved.


 for (int i = 0; i < totalKvObjects; i++) {
 putObjects((long) i, List.of(kv("k" + i, "k1v1", 0)));
 }

 ListKeyVersionsResponse previousPage = null;
 List<KeyValue> allKeyVersions = new ArrayList<>();

 while (previousPage == null || !previousPage.getKeyVersionsList().isEmpty()) {
 ListKeyVersionsResponse currentPage;

 if (previousPage == null) {
 currentPage = list(null, null, null);
 } else {
 String nextPageToken = previousPage.getNextPageToken();
 currentPage = list(nextPageToken, null, null);
 }

 allKeyVersions.addAll(currentPage.getKeyVersionsList());

 // Each page.size() is less than MAX_PAGE_SIZE
 assertThat(currentPage.getKeyVersionsList().size(), lessThan(vssArbitraryPageSizeMax));
 previousPage = currentPage;
 }

 assertThat(allKeyVersions.size(), is(totalKvObjects));
 }

 private KeyValue getObject(String key) {
 GetObjectRequest getRequest = GetObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
Expand All@@ -171,7 +358,7 @@ private KeyValue getObject(String key) {
 return this.kvStore.get(getRequest).getValue();
 }

 private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 private void putObjects(@Nullable Long globalVersion, List<KeyValue> keyValues) {
 PutObjectRequest.Builder putObjectRequestBuilder = PutObjectRequest.newBuilder()
 .setStoreId(STORE_ID)
 .addAllTransactionItems(keyValues);
Expand All@@ -183,6 +370,24 @@ private void putObjects(Long globalVersion, List<KeyValue> keyValues) {
 this.kvStore.put(putObjectRequestBuilder.build());
 }

 private ListKeyVersionsResponse list(@Nullable String nextPageToken, @Nullable Integer pageSize,
 @Nullable String keyPrefix) {
 ListKeyVersionsRequest.Builder listRequestBuilder = ListKeyVersionsRequest.newBuilder()
 .setStoreId(STORE_ID);

 if (StringUtils.isNotBlank(nextPageToken)) {
 listRequestBuilder.setPageToken(nextPageToken);
 }
 if (pageSize != null) {
 listRequestBuilder.setPageSize(pageSize);
 }
 if (StringUtils.isNotBlank(keyPrefix)) {
 listRequestBuilder.setKeyPrefix(keyPrefix);
 }

 return this.kvStore.listKeyVersions(listRequestBuilder.build());
 }

 private KeyValue kv(String key, String value, int version) {
 return KeyValue.newBuilder().setKey(key).setVersion(version).setValue(
 ByteString.copyFrom(value.getBytes(
Expand Down