Uh oh!
There was an error while loading. Please reload this page.
HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method - #2167
Merged
Conversation
ContributorAuthor
This can imporve a lot A simple test case before improve cost: 8566118066 7954087533 7990385173 8121662923 8094065619 8107938198 8045528125 8116824329 8122912440 8108117395 before all cost: 81227.639801 ms before vs after: 29198.60908098455:1 The test code in |
utf7
commented
Jul 29, 2020
ContributorAuthor
why don't trigger the related link to HBASE JIRA ? |
utf7
commented
Jul 29, 2020
ContributorAuthor
mind have a look ? thanks @Apache9@infraio@guangxuCheng@pankaj72981 |
| public final static LongAdder tot_mgr_log_split_batch_start = new LongAdder(); | ||
| public final static LongAdder tot_mgr_log_split_batch_success = new LongAdder(); | ||
| public final static LongAdder tot_mgr_log_split_batch_err = new LongAdder(); | ||
| public final static LongAdder tot_mgr_new_unexpected_wals = new LongAdder(); |
ContributorAuthor
There was a problem hiding this comment.
Thanks for view @pankaj72981
I will remove the HBASE-24790 related change in this PR
Apache-HBase
commented
Jul 29, 2020
🎊 +1 overall
This message was automatically generated. |
| tableNameBytes = Bytes.toBytes(writeTableNames); | ||
| } | ||
| String tableName = Bytes.toString(tableNameBytes); | ||
| Path tableRelPath = getTableRelativePath(tableNameBytes); |
Contributor
There was a problem hiding this comment.
That is a nice find.
BTW reading this part of code, I think so much of optimization we can do. It might not be relevant as this change. Though good to do IMO
...
} else {
tableNameBytes = Bytes.toBytes(writeTableNames);
}
String tableName = Bytes.toString(tableNameBytes);
...
Only in case of multiTable, we need create tableName and tableNameBytes again and again for every write. So we can make sure this is created at 1st and do not create every time for not multiTable case.
private Path getTableRelativePath(byte[] tableNameBytes) {
String tableName = Bytes.toString(tableNameBytes);
String[] tableNameParts = tableName.split(":");
Path tableRelPath = new Path(tableName.split(":")[0]);
if (tableNameParts.length > 1) {
tableRelPath = new Path(tableRelPath, tableName.split(":")[1]);
}
return tableRelPath;
}
split is done 3 times. We can just refer tableNameParts[0], tableNameParts[1] and other places.
Can u pls include these kind of fixes also in PR?
ContributorAuthor
There was a problem hiding this comment.
tableName split 3 times int getTableRelativePath method has fixed in this PR
i will do some other fixes in next commit , the code there is a little mess
…elativePath method
utf7
commented
Jul 29, 2020
ContributorAuthor
remove the HBASE-24790 code and do more code clean by review @anoopsjohn@pankaj72981 please have a look, thanks |
Apache-HBase
commented
Jul 29, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 29, 2020
🎊 +1 overall
This message was automatically generated. |
| private final Map<byte[], WriterLength> writers = new TreeMap<>(Bytes.BYTES_COMPARATOR); | ||
| private final Map<byte[], byte[]> previousRows = new TreeMap<>(Bytes.BYTES_COMPARATOR); | ||
| private final long now = EnvironmentEdgeManager.currentTime(); | ||
| private byte[] tableNameBytes = Bytes.toBytes(writeTableNames);; |
Contributor
There was a problem hiding this comment.
Remove extra semicolon at end.
Apache-HBase
commented
Jul 29, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 29, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 29, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 29, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 29, 2020
🎊 +1 overall
This message was automatically generated. |
tedyu
reviewed
Jul 29, 2020
| LOG.debug("Use favored nodes writer: {}", initialIsa.getHostString()); | ||
| wl = getNewWriter(tableNameBytes, family, conf, new InetSocketAddress[] { initialIsa | ||
| }); | ||
| favoredNodes = new InetSocketAddress[] { initialIsa}; |
Contributor
There was a problem hiding this comment.
Does this apply when LOCALITY_SENSITIVE_CONF_KEY check on line 274 is false ?
ContributorAuthor
There was a problem hiding this comment.
Yes, if the LOCALITY_SENSITIVE_CONF_KEY is false or get locality failed,the favoredNodes will be null
if LOCALITY_SENSITIVE_CONF_KEY = true and get localicy success ,the favoredNodes will be not null
same logic with before , just code clean
before this pr, too much wl = getNewWriter in the code
tedyu
commented
Jul 30, 2020
Contributor
Can you see why javadoc for hbase-mapreduce didn't pass ? Thanks |
utf7
commented
Jul 30, 2020
ContributorAuthor
thanks ,@tedyu in my local env
all passed how to trigger a retry ? |
utf7
commented
Jul 30, 2020
ContributorAuthor
rery qa |
tedyu
commented
Jul 30, 2020
Contributor
Apache9
commented
Jul 30, 2020
Contributor
The failure of javadoc for jdk11 is expected, we still have problems building javadoc with java11, that's why we just issue a -0, not -1. |
utf7
commented
Jul 30, 2020
ContributorAuthor
Apache9
commented
Jul 30, 2020
Contributor
utf7
commented
Jul 30, 2020
ContributorAuthor
tedyu
commented
Jul 30, 2020
Contributor
+1 |
utf7
commented
Aug 3, 2020
ContributorAuthor
The code hash finished , mind have a look, thanks @anoopsjohn@pankaj72981 |
anoopsjohn
approved these changes
Aug 3, 2020
anoopsjohn
left a comment
Contributor
There was a problem hiding this comment.
Looks good. Nice work
Added few minor comments.
| if (conf.getBoolean(LOCALITY_SENSITIVE_CONF_KEY, DEFAULT_LOCALITY_SENSITIVE)) { | ||
| HRegionLocation loc = null; | ||
| String tableName = Bytes.toString(tableNameBytes); |
Contributor
There was a problem hiding this comment.
Looks like a format issue here?
| } } | ||
| } | ||
| } |
| .withCompression(compression).withChecksumType(HStore.getChecksumType(conf)) | ||
| .withBytesPerCheckSum(HStore.getBytesPerChecksum(conf)).withBlockSize(blockSize) | ||
| .withColumnFamily(family).withTableName(tableName); | ||
| HFileContextBuilder contextBuilder = new HFileContextBuilder().withCompression(compression) |
Contributor
There was a problem hiding this comment.
Pls check format issue at these changed/added lines once.
ContributorAuthor
There was a problem hiding this comment.
i move the compression set config to here and has been use hbase-eclipse-format format the code here
| private final Map<byte[], WriterLength> writers = new TreeMap<>(Bytes.BYTES_COMPARATOR); | ||
| private final Map<byte[], byte[]> previousRows = new TreeMap<>(Bytes.BYTES_COMPARATOR); | ||
| private final long now = EnvironmentEdgeManager.currentTime(); | ||
| private byte[] tableNameBytes = Bytes.toBytes(writeTableNames); |
Contributor
There was a problem hiding this comment.
Should we do this under writeMultipleTables check?
private byte[] tableNameBytes = (writeMultipleTables)? null: Bytes.toBytes(writeTableNames);
ContributorAuthor
There was a problem hiding this comment.
yes，i want to make code simple, so just keep init no matter multiple or not
It is a good ponit to use use writeMultipleTables check,will address it
Apache-HBase
commented
Aug 3, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Aug 3, 2020
🎊 +1 overall
This message was automatically generated. |
anoopsjohn
approved these changes
Aug 3, 2020
vinayakphegde pushed a commit
to vinayakphegde/hbase
that referenced
this pull request
Apr 4, 2024
…elativePath method (apache#2167) Signed-off-by: Anoop <anoopsamjohn@apache.org> Signed-off-by: Ted Yu <tyu@apache.org> Change-Id: I684e0b79d2eb7d8cc10d2cdd3ef9ececc1cfd49b
szucsvillo pushed a commit
to szucsvillo/hbase
that referenced
this pull request
May 10, 2024
…elativePath method (apache#2167) Signed-off-by: Anoop <anoopsamjohn@apache.org> Signed-off-by: Ted Yu <tyu@apache.org> Change-Id: Ibf946930c20b15595a846d7b62e8b59cee42be07
szucsvillo pushed a commit
to szucsvillo/hbase
that referenced
this pull request
May 10, 2024
…elativePath method (apache#2167) Signed-off-by: Anoop <anoopsamjohn@apache.org> Signed-off-by: Ted Yu <tyu@apache.org> Change-Id: Ief23ecd94cba02af177a8187ad27a1d13fd0243e
szucsvillo pushed a commit
to szucsvillo/hbase
that referenced
this pull request
May 10, 2024
…elativePath method (apache#2167) Signed-off-by: Anoop <anoopsamjohn@apache.org> Signed-off-by: Ted Yu <tyu@apache.org> Change-Id: Ic1b21413b73c43d5591ae37800a2432b8a2ca688
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HBASE-24791 Improve HFileOutputFormat2 to avoid always call getTableRelativePath method