Skip to content

HBASE-23095 Reuse FileStatus in StoreFileInfo - #674

Merged
Apache9 merged 4 commits into
apache:masterfrom
karthikhw:HBASE-Snapshot
Oct 8, 2019
Merged

HBASE-23095 Reuse FileStatus in StoreFileInfo#674
Apache9 merged 4 commits into
apache:masterfrom
karthikhw:HBASE-Snapshot

Conversation

@karthikhw

Copy link
Copy Markdown
Contributor

We found this performance issue when taking a snapshot on large MOB table where MOB compaction is disabled more than 3 months, is due to MOB data loss issue HBASE-22075.

*/
public StoreFileInfo(final Configuration conf, final FileSystem fs, final FileStatus fileStatus)
throws IOException {
this(conf, fs, fileStatus.getPath());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, this is the problem...

}

private StoreFileInfo(final Configuration conf, final FileSystem fs, final Path initialPath,
final Long createdTimestamp, final Long size) throws IOException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just passing a FileStatus in?

@karthikhwkarthikhwSep 30, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially thought the same Duo like directly passing "FileStatus" in constructor but what I feel that "Path" is also in same constructor and even that Path can be constructed from FileStatus. So that looks to me not right. :) Another important is Path can not be removed from constructor because many callers pass only Path object, not FileStatus.

I am fine Duo if you still want to change?
public StoreFileInfo(final Configuration conf, final FileSystem fs, final Path initialPath, final FileStatus filestatus)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is fine, as the FileStatus could be null. And maybe you could add a @nullable annotation to indicate this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing the individual params Ts and size looks cleaner IMO. Anyways explained why selected that path.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this requires extra boxing/unboxing? And what if we need more parameters in the future...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any way is ok. As this was not in a hot path, even that Autoboxing thing was also ok. That 2 params passing looked more clean from the calling stand point. Anyway not so big concern. This looks clear too.

@karthikhwkarthikhwOct 1, 2019

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @anoopsjohn for the confirmation.

@Apache9 Duo, I already did new commit for your suggestions. Please confirm which one you would like to keep the old or new change?
6f4baae

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its ok.. Keep it as the way u have pushed as of now.. Its ok only.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure @anoopsjohn. Thanks again!!

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate Duo @Apache9 if you get some time for review :)

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
💙reexec0m 33sDocker mode activated.
_ Prechecks _
💚dupname0m 0sNo case conflicting files found.
💚hbaseanti0m 0sPatch does not have any anti-patterns.
💚@author0m 0sThe patch does not contain any @author tags.
💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚mvninstall5m 24smaster passed
💚compile0m 55smaster passed
💚checkstyle1m 21smaster passed
💚shadedjars4m 33sbranch has no errors when building our shaded downstream artifacts.
💚javadoc0m 37smaster passed
💙spotbugs4m 6sUsed deprecated FindBugs config; considering switching to SpotBugs.
💚findbugs4m 5smaster passed
_ Patch Compile Tests _
💚mvninstall4m 54sthe patch passed
💚compile0m 53sthe patch passed
💚javac0m 53sthe patch passed
💔checkstyle1m 16shbase-server: The patch generated 6 new + 19 unchanged - 0 fixed = 25 total (was 19)
💚whitespace0m 0sThe patch has no whitespace issues.
💚shadedjars4m 36spatch has no errors when building our shaded downstream artifacts.
💚hadoopcheck15m 38sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚javadoc0m 35sthe patch passed
💚findbugs4m 7sthe patch passed
_ Other Tests _
💚unit160m 20shbase-server in the patch passed.
💚asflicense0m 31sThe patch does not generate ASF License warnings.
216m 42s
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/1/artifact/out/Dockerfile
GITHUB PR#674
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 0974af5462ef 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh
git revisionmaster / ca0d9f3
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/1/artifact/out/diff-checkstyle-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/1/testReport/
Max. process+thread count4805 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/1/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
💙reexec3m 46sDocker mode activated.
_ Prechecks _
💚dupname0m 0sNo case conflicting files found.
💚hbaseanti0m 0sPatch does not have any anti-patterns.
💚@author0m 0sThe patch does not contain any @author tags.
💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚mvninstall6m 4smaster passed
💚compile0m 59smaster passed
💚checkstyle1m 29smaster passed
💚shadedjars5m 6sbranch has no errors when building our shaded downstream artifacts.
💚javadoc0m 38smaster passed
💙spotbugs4m 26sUsed deprecated FindBugs config; considering switching to SpotBugs.
💚findbugs4m 23smaster passed
_ Patch Compile Tests _
💚mvninstall5m 28sthe patch passed
💚compile0m 58sthe patch passed
💚javac0m 58sthe patch passed
💔checkstyle1m 29shbase-server: The patch generated 2 new + 19 unchanged - 0 fixed = 21 total (was 19)
💚whitespace0m 0sThe patch has no whitespace issues.
💚shadedjars5m 2spatch has no errors when building our shaded downstream artifacts.
💚hadoopcheck17m 30sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚javadoc0m 35sthe patch passed
💚findbugs4m 30sthe patch passed
_ Other Tests _
💔unit259m 48shbase-server in the patch failed.
💚asflicense0m 41sThe patch does not generate ASF License warnings.
325m 30s
ReasonTests
Failed junit testshadoop.hbase.util.TestFromClientSide3WoUnsafe
hadoop.hbase.master.assignment.TestMergeTableRegionsProcedure
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/artifact/out/Dockerfile
GITHUB PR#674
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux b7b1f92f219e 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh
git revisionmaster / 5217618
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/artifact/out/diff-checkstyle-hbase-server.txt
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/testReport/
Max. process+thread count4879 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/2/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@anoopsjohnanoopsjohn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1
Nice catch and analysis. Good job.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
💙reexec2m 5sDocker mode activated.
_ Prechecks _
💚dupname0m 0sNo case conflicting files found.
💚hbaseanti0m 0sPatch does not have any anti-patterns.
💚@author0m 0sThe patch does not contain any @author tags.
💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚mvninstall5m 31smaster passed
💚compile0m 54smaster passed
💚checkstyle1m 20smaster passed
💚shadedjars4m 38sbranch has no errors when building our shaded downstream artifacts.
💚javadoc0m 40smaster passed
💙spotbugs4m 3sUsed deprecated FindBugs config; considering switching to SpotBugs.
💚findbugs4m 1smaster passed
_ Patch Compile Tests _
💚mvninstall4m 49sthe patch passed
💚compile0m 57sthe patch passed
💚javac0m 57sthe patch passed
💔checkstyle1m 18shbase-server: The patch generated 1 new + 19 unchanged - 0 fixed = 20 total (was 19)
💚whitespace0m 0sThe patch has no whitespace issues.
💚shadedjars4m 37spatch has no errors when building our shaded downstream artifacts.
💚hadoopcheck15m 42sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚javadoc0m 36sthe patch passed
💚findbugs4m 12sthe patch passed
_ Other Tests _
💚unit160m 32shbase-server in the patch passed.
💚asflicense0m 44sThe patch does not generate ASF License warnings.
219m 17s
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/3/artifact/out/Dockerfile
GITHUB PR#674
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 5f28b0bbf8da 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh
git revisionmaster / 5217618
Default Java1.8.0_181
checkstylehttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/3/artifact/out/diff-checkstyle-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/3/testReport/
Max. process+thread count4663 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/3/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
💙reexec1m 9sDocker mode activated.
_ Prechecks _
💚dupname0m 0sNo case conflicting files found.
💚hbaseanti0m 0sPatch does not have any anti-patterns.
💚@author0m 0sThe patch does not contain any @author tags.
💚test4tests0m 0sThe patch appears to include 1 new or modified test files.
_ master Compile Tests _
💚mvninstall5m 54smaster passed
💚compile0m 58smaster passed
💚checkstyle1m 30smaster passed
💚shadedjars5m 4sbranch has no errors when building our shaded downstream artifacts.
💚javadoc0m 38smaster passed
💙spotbugs4m 27sUsed deprecated FindBugs config; considering switching to SpotBugs.
💚findbugs4m 24smaster passed
_ Patch Compile Tests _
💚mvninstall5m 28sthe patch passed
💚compile0m 57sthe patch passed
💚javac0m 57sthe patch passed
💚checkstyle1m 28sthe patch passed
💚whitespace0m 0sThe patch has no whitespace issues.
💚shadedjars5m 2spatch has no errors when building our shaded downstream artifacts.
💚hadoopcheck17m 25sPatch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.
💚javadoc0m 34sthe patch passed
💚findbugs4m 29sthe patch passed
_ Other Tests _
💔unit248m 7shbase-server in the patch failed.
💚asflicense0m 28sThe patch does not generate ASF License warnings.
310m 30s
ReasonTests
Failed junit testshadoop.hbase.master.TestSplitWALManager
SubsystemReport/Notes
DockerClient=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/artifact/out/Dockerfile
GITHUB PR#674
Optional Testsdupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile
unameLinux 71e4e1221f58 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 GNU/Linux
Build toolmaven
Personality/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-674/out/precommit/personality/provided.sh
git revisionmaster / 2ebdcbc
Default Java1.8.0_181
unithttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/artifact/out/patch-unit-hbase-server.txt
Test Resultshttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/testReport/
Max. process+thread count5080 (vs. ulimit of 10000)
modulesC: hbase-server U: hbase-server
Console outputhttps://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-674/4/console
versionsgit=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11
Powered byApache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

@karthikhw

Copy link
Copy Markdown
ContributorAuthor

@Apache9 Can you please approve? It still be point "change requested".

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@Apache9
Apache9 merged commit ff52994 into apache:masterOct 8, 2019
asfgit pushed a commit that referenced this pull request Oct 8, 2019
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
asfgit pushed a commit that referenced this pull request Oct 8, 2019
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
asfgit pushed a commit that referenced this pull request Oct 8, 2019
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
infraio pushed a commit to infraio/hbase that referenced this pull request Aug 17, 2020
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
symat pushed a commit to symat/hbase that referenced this pull request Feb 17, 2021
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 66506a6)
Change-Id: Idd7edd87d0d29024518b8928c1684106ec4f8cc3
kgeisz pushed a commit to kgeisz/hbase that referenced this pull request Jan 5, 2026
…ple CVEs (apache#674)
CDPD-94965: Backport HBASE-29651: Bump jruby to 9.4.14.0 to fix multiple CVEs
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@karthikhw@Apache-HBase@anoopsjohn@Apache9