Filestore Improvements by mitsuhiko · Pull Request #108 · getsentry/rfcs

mitsuhiko · 2023-07-13T13:47:24Z

This is a meta RFC to cover some of the potential improvements to our filestore system.

markstory · 2023-07-13T20:27:51Z

+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)


We also use files outside of organization contexts in control silo. Currently we've cloned the File model relations into control silo models so that we would have similar storage/interfaces for user & sentry app avatars.
Would you want to align file usage in control silo as well?

User avatars are indeed an interesting problem. Do we have different limits on non-debug-files?
As in: A debug-file can be up to 2G right now, and it is internally chunked.
Can we get away with using a different model for avatars altogether? I would argue they are a lot smaller (limited to 1M maybe?), and it does not make sense to chunk those.

Swatinem · 2023-07-20T09:33:28Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

How would you manage the migration from the old system to the new one? Will there ever be a cut-off date at which you can just hard-drop the old tables and GCS storage?

As this whole blob-related discussion started off with my discovery of a race-condition between blob upload and blob deletion, would this be solved by splitting off the staging area for uploads as you suggested from the long term storage?

As a reminder, the race condition is actually two separate TOCTOU (Time-of-Check-to-Time-of-Use) problems:

Before even uploading, sentry-cli asks the backend server which chunks are missing based on the chunk-hash.
Between this check and the final file assembly, the blob can be deleted, failing the assemble.
When assembling the final File, it first queries all the blobs based on their chunk-hash.
Between this check, and actually inserting a reference into the BlobIndex table, the blob is being deleted, failing the assemble.

I believe the first problem can be solved by a dedicated per-org staging area, one that will refresh a chunks TTL on query by chunk-hash.

The second problem can be either solved by not storing blobs deduplicated, like suggested.
Or I believe an epoch-based reclamation could also be used while still keeping deduplication:

Deletion would schedule to delete a chunk for epoch N
When assembling, we can use an UPSERT to increment the epoch in the database. (time of check)
In between, the deletion job would delete records with a matching epoch N, but deletion would not do anything as the epoch was already bumped to N+1
In the next assemble step, when creating BlobIndex entries, the blob is still there, yay (time of use)

Not sure if that complexity would be worth it, or we can just store duplicated blobs.

Deletions would be trivial, and also possible for older files and blobs correctly if we do not have concurrent writes and deletes.

Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko · 2023-07-20T14:20:12Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

I don't know. I think I would allow chunking as part of the system but I would force the chunk to be associated with the offset. But honestly for most of the stuff we probably want to do, one huge chunk for the entirety of the file is probably preferable in practical terms.

Swatinem · 2023-07-20T14:24:16Z

Can we get a histogram of chunk reuse reasonably? I would love to have some real data on how that reuse looks like.
Maybe the "empty chunk" is potentially shared a ton.

mitsuhiko · 2023-07-13T13:47:24Z

This is a meta RFC to cover some of the potential improvements to our filestore system.

Rendered RFC

markstory · 2023-07-13T20:27:51Z

+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)


We also use files outside of organization contexts in control silo. Currently we've cloned the File model relations into control silo models so that we would have similar storage/interfaces for user & sentry app avatars.
Would you want to align file usage in control silo as well?

User avatars are indeed an interesting problem. Do we have different limits on non-debug-files?
As in: A debug-file can be up to 2G right now, and it is internally chunked.
Can we get away with using a different model for avatars altogether? I would argue they are a lot smaller (limited to 1M maybe?), and it does not make sense to chunk those.

Swatinem · 2023-07-20T09:33:28Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

How would you manage the migration from the old system to the new one? Will there ever be a cut-off date at which you can just hard-drop the old tables and GCS storage?

As this whole blob-related discussion started off with my discovery of a race-condition between blob upload and blob deletion, would this be solved by splitting off the staging area for uploads as you suggested from the long term storage?

As a reminder, the race condition is actually two separate TOCTOU (Time-of-Check-to-Time-of-Use) problems:

Before even uploading, sentry-cli asks the backend server which chunks are missing based on the chunk-hash.
Between this check and the final file assembly, the blob can be deleted, failing the assemble.
When assembling the final File, it first queries all the blobs based on their chunk-hash.
Between this check, and actually inserting a reference into the BlobIndex table, the blob is being deleted, failing the assemble.

I believe the first problem can be solved by a dedicated per-org staging area, one that will refresh a chunks TTL on query by chunk-hash.

The second problem can be either solved by not storing blobs deduplicated, like suggested.
Or I believe an epoch-based reclamation could also be used while still keeping deduplication:

Deletion would schedule to delete a chunk for epoch N
When assembling, we can use an UPSERT to increment the epoch in the database. (time of check)
In between, the deletion job would delete records with a matching epoch N, but deletion would not do anything as the epoch was already bumped to N+1
In the next assemble step, when creating BlobIndex entries, the blob is still there, yay (time of use)

Not sure if that complexity would be worth it, or we can just store duplicated blobs.

Deletions would be trivial, and also possible for older files and blobs correctly if we do not have concurrent writes and deletes.

Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko · 2023-07-20T14:20:12Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

I don't know. I think I would allow chunking as part of the system but I would force the chunk to be associated with the offset. But honestly for most of the stuff we probably want to do, one huge chunk for the entirety of the file is probably preferable in practical terms.

Swatinem · 2023-07-20T14:24:16Z

Can we get a histogram of chunk reuse reasonably? I would love to have some real data on how that reuse looks like.
Maybe the "empty chunk" is potentially shared a ton.

mitsuhiko · 2023-07-13T13:47:24Z

This is a meta RFC to cover some of the potential improvements to our filestore system.

Rendered RFC

markstory · 2023-07-13T20:27:51Z

+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)


We also use files outside of organization contexts in control silo. Currently we've cloned the File model relations into control silo models so that we would have similar storage/interfaces for user & sentry app avatars.
Would you want to align file usage in control silo as well?

User avatars are indeed an interesting problem. Do we have different limits on non-debug-files?
As in: A debug-file can be up to 2G right now, and it is internally chunked.
Can we get away with using a different model for avatars altogether? I would argue they are a lot smaller (limited to 1M maybe?), and it does not make sense to chunk those.

Swatinem · 2023-07-20T09:33:28Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

How would you manage the migration from the old system to the new one? Will there ever be a cut-off date at which you can just hard-drop the old tables and GCS storage?

As this whole blob-related discussion started off with my discovery of a race-condition between blob upload and blob deletion, would this be solved by splitting off the staging area for uploads as you suggested from the long term storage?

As a reminder, the race condition is actually two separate TOCTOU (Time-of-Check-to-Time-of-Use) problems:

Before even uploading, sentry-cli asks the backend server which chunks are missing based on the chunk-hash.
Between this check and the final file assembly, the blob can be deleted, failing the assemble.
When assembling the final File, it first queries all the blobs based on their chunk-hash.
Between this check, and actually inserting a reference into the BlobIndex table, the blob is being deleted, failing the assemble.

I believe the first problem can be solved by a dedicated per-org staging area, one that will refresh a chunks TTL on query by chunk-hash.

The second problem can be either solved by not storing blobs deduplicated, like suggested.
Or I believe an epoch-based reclamation could also be used while still keeping deduplication:

Deletion would schedule to delete a chunk for epoch N
When assembling, we can use an UPSERT to increment the epoch in the database. (time of check)
In between, the deletion job would delete records with a matching epoch N, but deletion would not do anything as the epoch was already bumped to N+1
In the next assemble step, when creating BlobIndex entries, the blob is still there, yay (time of use)

Not sure if that complexity would be worth it, or we can just store duplicated blobs.

Deletions would be trivial, and also possible for older files and blobs correctly if we do not have concurrent writes and deletes.

Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko · 2023-07-20T14:20:12Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

I don't know. I think I would allow chunking as part of the system but I would force the chunk to be associated with the offset. But honestly for most of the stuff we probably want to do, one huge chunk for the entirety of the file is probably preferable in practical terms.

Swatinem · 2023-07-20T14:24:16Z

Can we get a histogram of chunk reuse reasonably? I would love to have some real data on how that reuse looks like.
Maybe the "empty chunk" is potentially shared a ton.

mitsuhiko · 2023-07-13T13:47:24Z

This is a meta RFC to cover some of the potential improvements to our filestore system.

Rendered RFC

markstory · 2023-07-13T20:27:51Z

+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)


We also use files outside of organization contexts in control silo. Currently we've cloned the File model relations into control silo models so that we would have similar storage/interfaces for user & sentry app avatars.
Would you want to align file usage in control silo as well?

User avatars are indeed an interesting problem. Do we have different limits on non-debug-files?
As in: A debug-file can be up to 2G right now, and it is internally chunked.
Can we get away with using a different model for avatars altogether? I would argue they are a lot smaller (limited to 1M maybe?), and it does not make sense to chunk those.

Swatinem · 2023-07-20T09:33:28Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

How would you manage the migration from the old system to the new one? Will there ever be a cut-off date at which you can just hard-drop the old tables and GCS storage?

As this whole blob-related discussion started off with my discovery of a race-condition between blob upload and blob deletion, would this be solved by splitting off the staging area for uploads as you suggested from the long term storage?

As a reminder, the race condition is actually two separate TOCTOU (Time-of-Check-to-Time-of-Use) problems:

Before even uploading, sentry-cli asks the backend server which chunks are missing based on the chunk-hash.
Between this check and the final file assembly, the blob can be deleted, failing the assemble.
When assembling the final File, it first queries all the blobs based on their chunk-hash.
Between this check, and actually inserting a reference into the BlobIndex table, the blob is being deleted, failing the assemble.

I believe the first problem can be solved by a dedicated per-org staging area, one that will refresh a chunks TTL on query by chunk-hash.

The second problem can be either solved by not storing blobs deduplicated, like suggested.
Or I believe an epoch-based reclamation could also be used while still keeping deduplication:

Deletion would schedule to delete a chunk for epoch N
When assembling, we can use an UPSERT to increment the epoch in the database. (time of check)
In between, the deletion job would delete records with a matching epoch N, but deletion would not do anything as the epoch was already bumped to N+1
In the next assemble step, when creating BlobIndex entries, the blob is still there, yay (time of use)

Not sure if that complexity would be worth it, or we can just store duplicated blobs.

Deletions would be trivial, and also possible for older files and blobs correctly if we do not have concurrent writes and deletes.

Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko · 2023-07-20T14:20:12Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

I don't know. I think I would allow chunking as part of the system but I would force the chunk to be associated with the offset. But honestly for most of the stuff we probably want to do, one huge chunk for the entirety of the file is probably preferable in practical terms.

Swatinem · 2023-07-20T14:24:16Z

Can we get a histogram of chunk reuse reasonably? I would love to have some real data on how that reuse looks like.
Maybe the "empty chunk" is potentially shared a ton.

mitsuhiko · 2023-07-13T13:47:24Z

This is a meta RFC to cover some of the potential improvements to our filestore system.

Rendered RFC

markstory · 2023-07-13T20:27:51Z

+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)


We also use files outside of organization contexts in control silo. Currently we've cloned the File model relations into control silo models so that we would have similar storage/interfaces for user & sentry app avatars.
Would you want to align file usage in control silo as well?

User avatars are indeed an interesting problem. Do we have different limits on non-debug-files?
As in: A debug-file can be up to 2G right now, and it is internally chunked.
Can we get away with using a different model for avatars altogether? I would argue they are a lot smaller (limited to 1M maybe?), and it does not make sense to chunk those.

Swatinem · 2023-07-20T09:33:28Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

How would you manage the migration from the old system to the new one? Will there ever be a cut-off date at which you can just hard-drop the old tables and GCS storage?

As this whole blob-related discussion started off with my discovery of a race-condition between blob upload and blob deletion, would this be solved by splitting off the staging area for uploads as you suggested from the long term storage?

As a reminder, the race condition is actually two separate TOCTOU (Time-of-Check-to-Time-of-Use) problems:

Before even uploading, sentry-cli asks the backend server which chunks are missing based on the chunk-hash.
Between this check and the final file assembly, the blob can be deleted, failing the assemble.
When assembling the final File, it first queries all the blobs based on their chunk-hash.
Between this check, and actually inserting a reference into the BlobIndex table, the blob is being deleted, failing the assemble.

I believe the first problem can be solved by a dedicated per-org staging area, one that will refresh a chunks TTL on query by chunk-hash.

The second problem can be either solved by not storing blobs deduplicated, like suggested.
Or I believe an epoch-based reclamation could also be used while still keeping deduplication:

Deletion would schedule to delete a chunk for epoch N
When assembling, we can use an UPSERT to increment the epoch in the database. (time of check)
In between, the deletion job would delete records with a matching epoch N, but deletion would not do anything as the epoch was already bumped to N+1
In the next assemble step, when creating BlobIndex entries, the blob is still there, yay (time of use)

Not sure if that complexity would be worth it, or we can just store duplicated blobs.

Deletions would be trivial, and also possible for older files and blobs correctly if we do not have concurrent writes and deletes.

Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko · 2023-07-20T14:20:12Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

I don't know. I think I would allow chunking as part of the system but I would force the chunk to be associated with the offset. But honestly for most of the stuff we probably want to do, one huge chunk for the entirety of the file is probably preferable in practical terms.

Swatinem · 2023-07-20T14:24:16Z

Can we get a histogram of chunk reuse reasonably? I would love to have some real data on how that reuse looks like.
Maybe the "empty chunk" is potentially shared a ton.

mitsuhiko · 2023-07-13T13:47:24Z

This is a meta RFC to cover some of the potential improvements to our filestore system.

Rendered RFC

markstory · 2023-07-13T20:27:51Z

+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)


We also use files outside of organization contexts in control silo. Currently we've cloned the File model relations into control silo models so that we would have similar storage/interfaces for user & sentry app avatars.
Would you want to align file usage in control silo as well?

User avatars are indeed an interesting problem. Do we have different limits on non-debug-files?
As in: A debug-file can be up to 2G right now, and it is internally chunked.
Can we get away with using a different model for avatars altogether? I would argue they are a lot smaller (limited to 1M maybe?), and it does not make sense to chunk those.

Swatinem · 2023-07-20T09:33:28Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

How would you manage the migration from the old system to the new one? Will there ever be a cut-off date at which you can just hard-drop the old tables and GCS storage?

As this whole blob-related discussion started off with my discovery of a race-condition between blob upload and blob deletion, would this be solved by splitting off the staging area for uploads as you suggested from the long term storage?

As a reminder, the race condition is actually two separate TOCTOU (Time-of-Check-to-Time-of-Use) problems:

Before even uploading, sentry-cli asks the backend server which chunks are missing based on the chunk-hash.
Between this check and the final file assembly, the blob can be deleted, failing the assemble.
When assembling the final File, it first queries all the blobs based on their chunk-hash.
Between this check, and actually inserting a reference into the BlobIndex table, the blob is being deleted, failing the assemble.

I believe the first problem can be solved by a dedicated per-org staging area, one that will refresh a chunks TTL on query by chunk-hash.

The second problem can be either solved by not storing blobs deduplicated, like suggested.
Or I believe an epoch-based reclamation could also be used while still keeping deduplication:

Deletion would schedule to delete a chunk for epoch N
When assembling, we can use an UPSERT to increment the epoch in the database. (time of check)
In between, the deletion job would delete records with a matching epoch N, but deletion would not do anything as the epoch was already bumped to N+1
In the next assemble step, when creating BlobIndex entries, the blob is still there, yay (time of use)

Not sure if that complexity would be worth it, or we can just store duplicated blobs.

Deletions would be trivial, and also possible for older files and blobs correctly if we do not have concurrent writes and deletes.

Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko · 2023-07-20T14:20:12Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

I don't know. I think I would allow chunking as part of the system but I would force the chunk to be associated with the offset. But honestly for most of the stuff we probably want to do, one huge chunk for the entirety of the file is probably preferable in practical terms.

Swatinem · 2023-07-20T14:24:16Z

Can we get a histogram of chunk reuse reasonably? I would love to have some real data on how that reuse looks like.
Maybe the "empty chunk" is potentially shared a ton.

mitsuhiko · 2023-07-13T13:47:24Z

This is a meta RFC to cover some of the potential improvements to our filestore system.

Rendered RFC

markstory · 2023-07-13T20:27:51Z

+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)


We also use files outside of organization contexts in control silo. Currently we've cloned the File model relations into control silo models so that we would have similar storage/interfaces for user & sentry app avatars.
Would you want to align file usage in control silo as well?

User avatars are indeed an interesting problem. Do we have different limits on non-debug-files?
As in: A debug-file can be up to 2G right now, and it is internally chunked.
Can we get away with using a different model for avatars altogether? I would argue they are a lot smaller (limited to 1M maybe?), and it does not make sense to chunk those.

Swatinem · 2023-07-20T09:33:28Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

How would you manage the migration from the old system to the new one? Will there ever be a cut-off date at which you can just hard-drop the old tables and GCS storage?

As this whole blob-related discussion started off with my discovery of a race-condition between blob upload and blob deletion, would this be solved by splitting off the staging area for uploads as you suggested from the long term storage?

As a reminder, the race condition is actually two separate TOCTOU (Time-of-Check-to-Time-of-Use) problems:

Before even uploading, sentry-cli asks the backend server which chunks are missing based on the chunk-hash.
Between this check and the final file assembly, the blob can be deleted, failing the assemble.
When assembling the final File, it first queries all the blobs based on their chunk-hash.
Between this check, and actually inserting a reference into the BlobIndex table, the blob is being deleted, failing the assemble.

I believe the first problem can be solved by a dedicated per-org staging area, one that will refresh a chunks TTL on query by chunk-hash.

The second problem can be either solved by not storing blobs deduplicated, like suggested.
Or I believe an epoch-based reclamation could also be used while still keeping deduplication:

Deletion would schedule to delete a chunk for epoch N
When assembling, we can use an UPSERT to increment the epoch in the database. (time of check)
In between, the deletion job would delete records with a matching epoch N, but deletion would not do anything as the epoch was already bumped to N+1
In the next assemble step, when creating BlobIndex entries, the blob is still there, yay (time of use)

Not sure if that complexity would be worth it, or we can just store duplicated blobs.

Deletions would be trivial, and also possible for older files and blobs correctly if we do not have concurrent writes and deletes.

Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko · 2023-07-20T14:20:12Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

I don't know. I think I would allow chunking as part of the system but I would force the chunk to be associated with the offset. But honestly for most of the stuff we probably want to do, one huge chunk for the entirety of the file is probably preferable in practical terms.

Swatinem · 2023-07-20T14:24:16Z

Can we get a histogram of chunk reuse reasonably? I would love to have some real data on how that reuse looks like.
Maybe the "empty chunk" is potentially shared a ton.

mitsuhiko · 2023-07-13T13:47:24Z

This is a meta RFC to cover some of the potential improvements to our filestore system.

Rendered RFC

markstory · 2023-07-13T20:27:51Z

+
+```python
+class FileBlob2(Model):
+ organization_id = BoundedBigIntegerField(db_index=True)


We also use files outside of organization contexts in control silo. Currently we've cloned the File model relations into control silo models so that we would have similar storage/interfaces for user & sentry app avatars.
Would you want to align file usage in control silo as well?

User avatars are indeed an interesting problem. Do we have different limits on non-debug-files?
As in: A debug-file can be up to 2G right now, and it is internally chunked.
Can we get away with using a different model for avatars altogether? I would argue they are a lot smaller (limited to 1M maybe?), and it does not make sense to chunk those.

Swatinem · 2023-07-20T09:33:28Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

How would you manage the migration from the old system to the new one? Will there ever be a cut-off date at which you can just hard-drop the old tables and GCS storage?

As this whole blob-related discussion started off with my discovery of a race-condition between blob upload and blob deletion, would this be solved by splitting off the staging area for uploads as you suggested from the long term storage?

As a reminder, the race condition is actually two separate TOCTOU (Time-of-Check-to-Time-of-Use) problems:

Before even uploading, sentry-cli asks the backend server which chunks are missing based on the chunk-hash.
Between this check and the final file assembly, the blob can be deleted, failing the assemble.
When assembling the final File, it first queries all the blobs based on their chunk-hash.
Between this check, and actually inserting a reference into the BlobIndex table, the blob is being deleted, failing the assemble.

I believe the first problem can be solved by a dedicated per-org staging area, one that will refresh a chunks TTL on query by chunk-hash.

The second problem can be either solved by not storing blobs deduplicated, like suggested.
Or I believe an epoch-based reclamation could also be used while still keeping deduplication:

Deletion would schedule to delete a chunk for epoch N
When assembling, we can use an UPSERT to increment the epoch in the database. (time of check)
In between, the deletion job would delete records with a matching epoch N, but deletion would not do anything as the epoch was already bumped to N+1
In the next assemble step, when creating BlobIndex entries, the blob is still there, yay (time of use)

Not sure if that complexity would be worth it, or we can just store duplicated blobs.

Deletions would be trivial, and also possible for older files and blobs correctly if we do not have concurrent writes and deletes.

Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko · 2023-07-20T14:20:12Z

So if I read this correctly, you would still want to chunk files, but not saving those chunks deduplicated, thus avoiding the atomic reference counting problem?

I don't know. I think I would allow chunking as part of the system but I would force the chunk to be associated with the offset. But honestly for most of the stuff we probably want to do, one huge chunk for the entirety of the file is probably preferable in practical terms.

Swatinem · 2023-07-20T14:24:16Z

Can we get a histogram of chunk reuse reasonably? I would love to have some real data on how that reuse looks like.
Maybe the "empty chunk" is potentially shared a ton.

mitsuhiko added 3 commits July 13, 2023 12:39

WIP'
1e34e28

More text changes
b0a1340

Update file
432ff7f

mitsuhiko changed the title ~~Filestore new~~Filestore ImprovementsJul 13, 2023

markstory reviewed Jul 13, 2023
View reviewed changes

Apply suggestions from code review
6edfdcf
Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko added 3 commits July 13, 2023 12:39

WIP'
1e34e28

More text changes
b0a1340

Update file
432ff7f

mitsuhiko changed the title ~~Filestore new~~Filestore ImprovementsJul 13, 2023

markstory reviewed Jul 13, 2023
View reviewed changes

Apply suggestions from code review
6edfdcf
Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko added 3 commits July 13, 2023 12:39

WIP'
1e34e28

More text changes
b0a1340

Update file
432ff7f

mitsuhiko changed the title ~~Filestore new~~Filestore ImprovementsJul 13, 2023

markstory reviewed Jul 13, 2023
View reviewed changes

Apply suggestions from code review
6edfdcf
Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko added 3 commits July 13, 2023 12:39

WIP'
1e34e28

More text changes
b0a1340

Update file
432ff7f

mitsuhiko changed the title ~~Filestore new~~Filestore ImprovementsJul 13, 2023

markstory reviewed Jul 13, 2023
View reviewed changes

Apply suggestions from code review
6edfdcf
Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko added 3 commits July 13, 2023 12:39

WIP'
1e34e28

More text changes
b0a1340

Update file
432ff7f

mitsuhiko changed the title ~~Filestore new~~Filestore ImprovementsJul 13, 2023

markstory reviewed Jul 13, 2023
View reviewed changes

Apply suggestions from code review
6edfdcf
Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko added 3 commits July 13, 2023 12:39

WIP'
1e34e28

More text changes
b0a1340

Update file
432ff7f

mitsuhiko changed the title ~~Filestore new~~Filestore ImprovementsJul 13, 2023

markstory reviewed Jul 13, 2023
View reviewed changes

Apply suggestions from code review
6edfdcf
Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko added 3 commits July 13, 2023 12:39

WIP'
1e34e28

More text changes
b0a1340

Update file
432ff7f

mitsuhiko changed the title ~~Filestore new~~Filestore ImprovementsJul 13, 2023

markstory reviewed Jul 13, 2023
View reviewed changes

Apply suggestions from code review
6edfdcf
Co-authored-by: Mark Story <mark@mark-story.com>

mitsuhiko added 3 commits July 13, 2023 12:39

WIP'
1e34e28

More text changes
b0a1340

Update file
432ff7f

mitsuhiko changed the title ~~Filestore new~~Filestore ImprovementsJul 13, 2023

markstory reviewed Jul 13, 2023
View reviewed changes

Apply suggestions from code review
6edfdcf
Co-authored-by: Mark Story <mark@mark-story.com>

Uh oh!

Uh oh!

Conversation

mitsuhiko commented Jul 13, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swatinem commented Jul 20, 2023

Uh oh!

mitsuhiko commented Jul 20, 2023

Uh oh!

Swatinem commented Jul 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

mitsuhiko commented Jul 13, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swatinem commented Jul 20, 2023

Uh oh!

mitsuhiko commented Jul 20, 2023

Uh oh!

Swatinem commented Jul 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

mitsuhiko commented Jul 13, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Swatinem commented Jul 20, 2023

Uh oh!

mitsuhiko commented Jul 20, 2023

Uh oh!

Swatinem commented Jul 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

mitsuhiko commented Jul 13, 2023•
edited
Loading

mitsuhiko commented Jul 13, 2023•
edited
Loading

mitsuhiko commented Jul 13, 2023•
edited
Loading

mitsuhiko commented Jul 13, 2023•
edited
Loading

mitsuhiko commented Jul 13, 2023•
edited
Loading