HADOOP-18012. ABFS: Using Source Path eTags for Rename Idemptonency checks - #5488
Merged
Merged
Conversation
Loading
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RenameFilePath on its first try receives a Request timed out error with code 500. On retrying the same operation, a Source file not found (404) error is received.
This change brings in the following mitigation:
It checks whether etags remain the same before and after the retry and accordingly send an Operation Successful result, instead of source file not found. That is, if the rename actually succeeds at the backend, the original source eTag (before the operation) and the destination eTag (after the operation) should match each other. However, this logic works only in the case of HNS configurations, and files. For directories or flat namespaces, rename recovery is not attempted.
The check for whether the eTags match or not is enabled by a configuration:
fs.azure.enable.rename.resilience.The tests introduced mock the retry scenario and check correct behavior (failure/recovery) depending on whether it is HNS, FNS, directory or file, both at the FileSystem level and by unit testing the renamePath API at AbfsClient level.