Uh oh!
There was an error while loading. Please reload this page.
HBASE-22877 WebHDFS based export snapshot will fail if hfile is in archive directory - #509
HBASE-22877 WebHDFS based export snapshot will fail if hfile is in archive directory#509VicoWu wants to merge 2 commits into
Conversation
…chive directory FileLink.tryOpen() depends on fs.open(path, bufferSize) to throw FileNotFoundException to try the next file location; But when we use WebHDFS, no exception was thrown even when the file didn't exist; We should add an addition code to handle the WebHDFS case;
jojochuang
commented
Aug 19, 2019
Does the patch try to fail early? Or does it try to support webhdfs based snapshot export? |
@jojochuang
|
Apache-HBase
commented
Aug 19, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 19, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 20, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 20, 2019
💔 -1 overall
This message was automatically generated. |
@VicoWu Could you elaborate why you think it doesn't throw FileNotfound exception. In the stack trace you pasted, it did throw FileNotFound Exception but it is wrapped in RemoteException. You just need to unwrap RemoteException to see underlying exception. Maybe I am missing something. Please correct me if I am wrong. |
Apache-HBase
commented
Aug 20, 2019
💔 -1 overall
This message was automatically generated. |
jojochuang
commented
Aug 20, 2019
Interesting to know this webhdfs behavior. |
@jojochuang |
Apache-HBase
commented
Aug 21, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 21, 2019
💔 -1 overall
This message was automatically generated. |
I am confused whether you are talking about WebHdfsFileSystem or HttpFSFileSystem. These two are different. Refer to class org.apache.hadoop.fs.http.client.HttpFSFileSystem for latter. The further comment is assuming that you are talking about WebHdfsFileSystem since you mentioned that multiple times.
This is not correct. When you call WebHdfsFileSystem#open(), it does create an http connection to namenode and gets the list of datanodes where the blocks for that file resides. When you call read on the input stream, it directly goes to datanode. |
Apache-HBase
commented
Aug 21, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 21, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 21, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 22, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 22, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 22, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 22, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 22, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 23, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 24, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 24, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 24, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 24, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 24, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 24, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 24, 2019
💔 -1 overall
This message was automatically generated. |
@shahrs87
Yes, I previous make mistakes for the differences between
To be simple, in my code base And this patch has been applied to only So that's why your experiment cannot reproduce this problem and that you think the So that's my investigation for this problem; I think everything is much more clear now; |
Apache-HBase
commented
Aug 25, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 25, 2019
🎊 +1 overall
This message was automatically generated. |
This bug does not exist in branch-2.6.0 also. |
Apache-HBase
commented
Aug 27, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 27, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 28, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 28, 2019
💔 -1 overall
This message was automatically generated. |
@shahrs87 And I find the first occurrence for the |
Apache-HBase
commented
Aug 29, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 30, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 30, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 31, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 31, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 1, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 2, 2019
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 2, 2019
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Sep 2, 2019
💔 -1 overall
This message was automatically generated. |
saintstack
commented
Sep 14, 2019
Ok. Really good back and forth here. Good info in the above dialog. Seems like a CDH5 issue. Closing. Shout if I have it wrong. |
The corresponding issue: HBASE-22877
The problem happened in the method
tryOpen:FileLink.tryOpen()depends onfs.open(path, bufferSize)to throwFileNotFoundExceptionto try the next file location; When we are using traditional HDFS, the
fsis implements ofDistributedFileSystemORViewFileSystem, but when we are use webhdfs, thefsis implement ofWebHdfsFileSystem, in this case, no exception will be thrown even whenthe file didn't exist when we are calling
fs.open(path, bufferSize); so,ExportMapperwill think that hfile exists in the tmp directory, the FileLink will use this directory as the hfile directory by fault; Then finally, when mapper task trying to read this hfile, it found that this file doesn't exist in fact and mapper will fail:We should add an addition code to handle the
WebHDFScase;