Uh oh!
There was an error while loading. Please reload this page.
Core: Validate concurrently added delete files in RowDelta - #3195
Conversation
| * @param conflictDetectionFilter an expression on rows in the table | ||
| * @return this for method chaining | ||
| */ | ||
| RowDelta conflictDetectionFilter(Expression conflictDetectionFilter); |
There was a problem hiding this comment.
This is actually different from the implementation in PR #3069. I think this one is slightly better as we set the conflict detection filter only once and then enable data/delete file validation.
There was a problem hiding this comment.
Otherwise, we would need to reason about what happens if the filters are different, which one to use for validating referenced data files, etc.
aokolnychyi
commented
Sep 28, 2021
| return (globalDeletes == null || globalDeletes.length == 0) && sortedDeletesByPartition.isEmpty(); | ||
| } | ||
| public List<DeleteFile> referencedDeleteFiles() { |
There was a problem hiding this comment.
Would Iterable<DeleteFile> work instead? That would make this lazy.
| protected void validateNoNewDeleteFiles(TableMetadata base, Long startingSnapshotId, | ||
| Expression dataFilter, boolean caseSensitive) { | ||
| // if there is no current table state, no files have been added | ||
| if (base.currentSnapshot() == null) { |
There was a problem hiding this comment.
Can you add the check for base.formatVersion() < 2?
There was a problem hiding this comment.
Added to both methods. Could you check, @rdblue?
rdblue
commented
Sep 28, 2021
Looks good. Thanks for adding that check! |
aokolnychyi
commented
Sep 28, 2021
Thanks for reviewing, @rdblue! |
This PR validates concurrently added delete files in
BaseRowDelta.