Uh oh!
There was an error while loading. Please reload this page.
HBASE-27522 Use reflect to read HDFS ReadStatistics - #4917
Closed
cxzl25 wants to merge 1 commit into
Closed
Conversation
Apache-HBase
commented
Dec 8, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 8, 2022
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 8, 2022
🎊 +1 overall
This message was automatically generated. |
Apache9
commented
Dec 9, 2022
Contributor
Have you tried to depend on the 2.5.2-hadoop3 artifacts? It is compiled with hadoop 3.2.4 which should have fixed the problem. https://mvnrepository.com/artifact/org.apache.hbase/hbase-client/2.5.2-hadoop3 |
Contributor
It looks like your HBase was build with Hadoop 2, but ran with Hadoop 3 runtime. Make sure your build and runtime dependency are the same. For what is worth, I specifically wrote this part of the code to avoid using reflection. |
jojochuang
commented
Dec 9, 2022
Contributor
IMO this PR is not needed for master branch, because master branch is supposed to depend on Hadoop 3 library only. It would make more sense for branch-2.x |
cxzl25
commented
Dec 9, 2022
ContributorAuthor
Thank you for your reminder, I tested it and it works with |
cxzl25
commented
Dec 9, 2022
ContributorAuthor
Yes, Hudi depends on You can find issue 5765 in the Hudi GitHub repository.
Reflection does have a slight performance penalty.
Thank you for the reminder, I now noticed that the master branch only has hadoop3. |
cxzl25
commented
Dec 9, 2022
ContributorAuthor
CurrentExceptioninthread"main"java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
atorg.apache.hadoop.hbase.io.FSDataInputStreamWrapper.updateInputStreamStatistics(FSDataInputStreamWrapper.java:249)
atorg.apache.hadoop.hbase.io.FSDataInputStreamWrapper.close(FSDataInputStreamWrapper.java:296)
atorg.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.closeStreams(HFileBlock.java:1825)
atorg.apache.hadoop.hbase.io.hfile.HFilePreadReader.close(HFilePreadReader.java:107)
atorg.apache.hadoop.hbase.io.hfile.HFileReaderImpl.close(HFileReaderImpl.java:1421)
atorg.test.TestHFile.main(TestHFile.java:39)TestHFilepackageorg.test;
importorg.apache.hadoop.conf.Configuration;
importorg.apache.hadoop.fs.FileSystem;
importorg.apache.hadoop.fs.Path;
importorg.apache.hadoop.hbase.KeyValue;
importorg.apache.hadoop.hbase.io.hfile.CacheConfig;
importorg.apache.hadoop.hbase.io.hfile.HFile;
importorg.apache.hadoop.hbase.io.hfile.HFileContext;
importorg.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
importorg.apache.hadoop.hdfs.MiniDFSCluster;
importjava.io.IOException;
publicclassTestHFile {
publicstaticvoidmain(String[] args) throwsIOException {
MiniDFSClusterdfsCluster = newMiniDFSCluster.Builder(newConfiguration()).build();
FileSystemfs = dfsCluster.getFileSystem();
Configurationconf = fs.getConf();
Pathfile = newPath("/tmp/test_hbase_file.hfile");
HFileContextfileContext = newHFileContextBuilder().build();
CacheConfigcacheConfig = newCacheConfig(conf);
HFile.Writerwriter = HFile.getWriterFactory(conf, cacheConfig)
.withPath(fs, file)
.withFileContext(fileContext)
.create();
KeyValuekv = newKeyValue("k1".getBytes(), null, null, "v1".getBytes());
writer.append(kv);
writer.close();
HFile.Readerreader = HFile.createReader(fs, file, conf);
try {
reader.close();
} finally {
dfsCluster.shutdown();
}
}
}pom.xml<projectxmlns="http://maven.apache.org/POM/4.0.0"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>hbase_hfile_test</artifactId>
<version>1.0-SNAPSHOT</version>
<packaging>jar</packaging>
<name>hbase_hfile_test</name>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<hbase.version>2.4.9</hbase.version>
<!-- <hbase.version>2.5.2-hadoop3</hbase.version>-->
</properties>
<profiles>
<profile>
<id>hadoop-2.10</id>
<properties>
<hadoop.version>2.10.0</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
<classifier>tests</classifier>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
<classifier>tests</classifier>
</dependency>
</dependencies>
</profile>
<profile>
<id>hadoop-3.0</id>
<properties>
<hadoop.version>3.0.0</hadoop.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-minicluster</artifactId>
<version>${hadoop.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-api</artifactId>
</exclusion>
</exclusions>
</dependency>
</dependencies>
</profile>
</profiles>
<dependencies>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>${hbase.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.12</version>
</dependency>
</dependencies>
</project>
|
Apache-HBase
commented
Dec 12, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 12, 2022
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 12, 2022
🎊 +1 overall
This message was automatically generated. |
Apache9
commented
Dec 19, 2022
Contributor
As I said above, if you want to use hadoop3, now you can depend on hbase-client 2.5.2-hadoop3. Let's just bump the hbase version in Hudi to solve the problem? Thanks. |
cxzl25
commented
Dec 20, 2022
ContributorAuthor
Thank you for comments. @Apache9@jojochuang |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-27522
Use reflection in this PR for better compatibility.
In Hadoop 3.0.0 HDFS-8905, the return value of the
getReadStatisticsmethod of classHdfsDataInputStreamis modified.Add the metrics for reading
getReadStatisticsin HBASE-8868, but use HBase to read HFILE in the Hadoop 3 environment, and an error will be reported in the close reader.