HBASE-28456 HBase Restore restores old data if data for the same timestamp is in different hfiles by bbeaudreault · Pull Request #5775 · apache/hbase

bbeaudreault · 2024-03-25T21:05:36Z

Unifies _SeqId_ handling
Adds necessary config in MapReduceHFileSplliter so that sorting can include sequenceId
Updates HFileInputFormat to handle sequenceIds of bulkloaded files. This part is mirrored after similar logic in StoreFileScanner

…stamp is in different hfiles

Apache-HBase · 2024-03-25T21:57:06Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	2m 29s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 2s	master passed
+1 💚	compile	3m 19s	master passed
+1 💚	checkstyle	0m 52s	master passed
+1 💚	spotless	0m 42s	branch has no errors when running spotless:check.
+1 💚	spotbugs	2m 21s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 46s	the patch passed
+1 💚	compile	3m 16s	the patch passed
+1 💚	javac	3m 16s	the patch passed
-0 ⚠️	checkstyle	0m 7s	hbase-backup: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 13s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 40s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 41s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 24s	The patch does not generate ASF License warnings.
		36m 0s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux de657f2b134b 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-11.0.17+8
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-backup.txt
Max. process+thread count	78 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T01:50:29Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 57s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 57s	master passed
+1 💚	compile	1m 34s	master passed
+1 💚	shadedjars	5m 14s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 57s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 49s	the patch passed
+1 💚	compile	1m 37s	the patch passed
+1 💚	javac	1m 37s	the patch passed
+1 💚	shadedjars	5m 10s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 57s	the patch passed
		_ Other Tests _
+1 💚	unit	218m 16s	hbase-server in the patch passed.
+1 💚	unit	13m 55s	hbase-mapreduce in the patch passed.
+1 💚	unit	9m 46s	hbase-backup in the patch passed.
		269m 54s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 126cf8120fd5 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-17.0.10+7
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/testReport/
Max. process+thread count	4606 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T02:12:57Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 39s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 57s	master passed
+1 💚	compile	1m 38s	master passed
+1 💚	shadedjars	6m 2s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 2s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 41s	the patch passed
+1 💚	compile	1m 30s	the patch passed
+1 💚	javac	1m 30s	the patch passed
+1 💚	shadedjars	5m 18s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 56s	the patch passed
		_ Other Tests _
+1 💚	unit	234m 54s	hbase-server in the patch passed.
+1 💚	unit	16m 7s	hbase-mapreduce in the patch passed.
+1 💚	unit	11m 12s	hbase-backup in the patch passed.
		292m 22s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 54e0f2b41f5a 5.4.0-163-generic #180-Ubuntu SMP Tue Sep 5 13:21:23 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/testReport/
Max. process+thread count	4937 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T02:39:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 33s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 15s	master passed
+1 💚	compile	1m 26s	master passed
+1 💚	shadedjars	6m 23s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 58s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 0s	the patch passed
+1 💚	compile	1m 26s	the patch passed
+1 💚	javac	1m 26s	the patch passed
+1 💚	shadedjars	6m 29s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 50s	the patch passed
		_ Other Tests _
-1 ❌	unit	260m 41s	hbase-server in the patch failed.
+1 💚	unit	18m 15s	hbase-mapreduce in the patch passed.
+1 💚	unit	10m 34s	hbase-backup in the patch passed.
		319m 5s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux b9e272bc90a5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/testReport/
Max. process+thread count	4821 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/1/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T03:19:54Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	3m 43s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 10s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 56s	master passed
+1 💚	compile	3m 27s	master passed
+1 💚	checkstyle	1m 1s	master passed
+1 💚	spotless	0m 46s	branch has no errors when running spotless:check.
+1 💚	spotbugs	2m 33s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 48s	the patch passed
+1 💚	compile	3m 26s	the patch passed
+1 💚	javac	3m 26s	the patch passed
-0 ⚠️	checkstyle	0m 11s	hbase-backup: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	4m 52s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	0m 42s	patch has no errors when running spotless:check.
+1 💚	spotbugs	2m 57s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		37m 39s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 9c04c38c254e 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-11.0.17+8
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-backup.txt
Max. process+thread count	82 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T07:11:09Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 18s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 3s	master passed
+1 💚	compile	1m 19s	master passed
+1 💚	shadedjars	5m 34s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 49s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 46s	the patch passed
+1 💚	compile	1m 18s	the patch passed
+1 💚	javac	1m 18s	the patch passed
+1 💚	shadedjars	5m 33s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 47s	the patch passed
		_ Other Tests _
-1 ❌	unit	218m 9s	hbase-server in the patch failed.
+1 💚	unit	14m 19s	hbase-mapreduce in the patch passed.
+1 💚	unit	9m 30s	hbase-backup in the patch passed.
		268m 42s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 7c894cb1c895 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-11.0.17+8
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/testReport/
Max. process+thread count	4646 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T07:16:27Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 38s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for branch
+1 💚	mvninstall	4m 1s	master passed
+1 💚	compile	2m 8s	master passed
+1 💚	shadedjars	6m 4s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 9s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 25s	the patch passed
+1 💚	compile	1m 47s	the patch passed
+1 💚	javac	1m 47s	the patch passed
+1 💚	shadedjars	5m 26s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 57s	the patch passed
		_ Other Tests _
-1 ❌	unit	218m 53s	hbase-server in the patch failed.
+1 💚	unit	13m 59s	hbase-mapreduce in the patch passed.
+1 💚	unit	10m 1s	hbase-backup in the patch passed.
		274m 9s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 49e769fb4895 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-17.0.10+7
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/testReport/
Max. process+thread count	4546 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T08:22:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 30s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 16s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 58s	master passed
+1 💚	compile	1m 28s	master passed
+1 💚	shadedjars	6m 31s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 58s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 41s	the patch passed
+1 💚	compile	1m 24s	the patch passed
+1 💚	javac	1m 24s	the patch passed
+1 💚	shadedjars	6m 40s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	1m 15s	the patch passed
		_ Other Tests _
-1 ❌	unit	273m 6s	hbase-server in the patch failed.
+1 💚	unit	22m 43s	hbase-mapreduce in the patch passed.
+1 💚	unit	13m 26s	hbase-backup in the patch passed.
		340m 11s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux c3f38f403494 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/testReport/
Max. process+thread count	4881 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/2/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ndimiduk · 2024-03-26T12:49:31Z

 * ExtendedCell.
 */
- static final String EXTENDED_CELL_SERIALIZATION_ENABLED_KEY =
+public static final String EXTENDED_CELL_SERIALIZATION_ENABLED_KEY =


This field is becoming part of the public API. It's worth updating the comment about its usage.

@ndimiduk I decided to annotate that as IA.Private. I know we generally prefer not to do that for fields/methods, but for better or worse there's already a strong convention of doing it in this class (11 methods annotated IA.Private). It's probably worth a larger refactor/cleanup of HFileOutputFormat2
Let me know if you strongly disagree with this approach

Good by me.

ndimiduk · 2024-03-26T12:51:57Z


+ /**
+ * Cells in a bulkloaded file don't have a sequenceId since they don't go through memstore. When a
+ * bulkload file is committed, the current memstore ts is stamped onto the file name as the


Yikes! Seems like we need a seqId field in the file header instead of relying on the file name...

I think the reason why we don't do this already is that a bulkloaded HFile is typically created outside the cluster. At that point, we don't know what the memstore seq id is at. Only at the time of bulkload file commit, when we've locked and flushed the memstore, do we know the memstore seq id.
Since hfiles are immutable, we can't open it at that point and change a metadata. So we need to add it to the filename I guess. I agree relying on the file name seems brittle. A more robust solution might be tricky given the above tho

Apache-HBase · 2024-03-26T14:02:45Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 42s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
		_ master Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 46s	master passed
+1 💚	compile	4m 22s	master passed
+1 💚	checkstyle	1m 16s	master passed
+1 💚	spotless	0m 55s	branch has no errors when running spotless:check.
+1 💚	spotbugs	3m 26s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	3m 34s	the patch passed
+1 💚	compile	3m 47s	the patch passed
+1 💚	javac	3m 47s	the patch passed
-0 ⚠️	checkstyle	0m 13s	hbase-backup: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	hadoopcheck	5m 30s	Patch does not cause any errors with Hadoop 3.3.6.
+1 💚	spotless	1m 0s	patch has no errors when running spotless:check.
+1 💚	spotbugs	3m 51s	the patch passed
		_ Other Tests _
+1 💚	asflicense	0m 31s	The patch does not generate ASF License warnings.
		41m 58s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	dupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
uname	Linux 2d82baf71c74 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-11.0.17+8
checkstyle	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-backup.txt
Max. process+thread count	79 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/console
versions	git=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache9 · 2024-03-26T14:16:01Z

+
+import static org.apache.hadoop.hbase.backup.BackupInfo.BackupState.COMPLETE;
+import static org.apache.hadoop.hbase.backup.BackupType.FULL;
+import static org.junit.Assert.*;


Avoid star imports.

Apache9 · 2024-03-26T14:16:06Z

+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Instant;
+import java.util.*;


Apache9 · 2024-03-26T14:18:31Z

 }
 value = scanner.getCell();
+ if (value != null && bulkloadSeqId.isPresent()) {
+ PrivateCellUtil.setSequenceId(value, bulkloadSeqId.getAsLong());


We can not use bulkloadSeqId.ifPresent because PrivateCellUtil.setSequenceId will throw exceptions?

Yea setSequenceId throws IOException. Figured it looks cleaner this way

…stamp is in different hfiles (#5775) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

Apache-HBase · 2024-03-26T17:51:00Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 37s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 49s	master passed
+1 💚	compile	1m 32s	master passed
+1 💚	shadedjars	5m 15s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 56s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 53s	the patch passed
+1 💚	compile	1m 33s	the patch passed
+1 💚	javac	1m 33s	the patch passed
+1 💚	shadedjars	5m 13s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 57s	the patch passed
		_ Other Tests _
-1 ❌	unit	219m 24s	hbase-server in the patch failed.
+1 💚	unit	13m 49s	hbase-mapreduce in the patch passed.
+1 💚	unit	9m 42s	hbase-backup in the patch passed.
		270m 14s

Subsystem	Report/Notes
Docker	ClientAPI=1.45 ServerAPI=1.45 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux ebddb03258ee 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-17.0.10+7
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk17-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/testReport/
Max. process+thread count	4460 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T17:58:26Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 29s	Docker mode activated.
-0 ⚠️	yetus	0m 3s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for branch
+1 💚	mvninstall	2m 45s	master passed
+1 💚	compile	1m 9s	master passed
+1 💚	shadedjars	5m 34s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 44s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 27s	the patch passed
+1 💚	compile	1m 9s	the patch passed
+1 💚	javac	1m 9s	the patch passed
+1 💚	shadedjars	5m 36s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 45s	the patch passed
		_ Other Tests _
-1 ❌	unit	227m 45s	hbase-server in the patch failed.
+1 💚	unit	14m 13s	hbase-mapreduce in the patch passed.
+1 💚	unit	10m 24s	hbase-backup in the patch passed.
		277m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux a953142d17d5 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Temurin-1.8.0_352-b08
unit	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/testReport/
Max. process+thread count	5653 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Apache-HBase · 2024-03-26T18:02:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 35s	Docker mode activated.
-0 ⚠️	yetus	0m 2s	Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
		_ Prechecks _
		_ master Compile Tests _
+0 🆗	mvndep	0m 13s	Maven dependency ordering for branch
+1 💚	mvninstall	3m 9s	master passed
+1 💚	compile	1m 17s	master passed
+1 💚	shadedjars	5m 36s	branch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 49s	master passed
		_ Patch Compile Tests _
+0 🆗	mvndep	0m 12s	Maven dependency ordering for patch
+1 💚	mvninstall	2m 47s	the patch passed
+1 💚	compile	1m 19s	the patch passed
+1 💚	javac	1m 19s	the patch passed
+1 💚	shadedjars	5m 34s	patch has no errors when building our shaded downstream artifacts.
+1 💚	javadoc	0m 48s	the patch passed
		_ Other Tests _
+1 💚	unit	229m 30s	hbase-server in the patch passed.
+1 💚	unit	14m 52s	hbase-mapreduce in the patch passed.
+1 💚	unit	10m 16s	hbase-backup in the patch passed.
		281m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR	#5775
Optional Tests	javac javadoc unit shadedjars compile
uname	Linux 301ff65adad1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	master / `38aef80`
Default Java	Eclipse Adoptium-11.0.17+8
Test Results	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/testReport/
Max. process+thread count	4467 (vs. ulimit of 30000)
modules	C: hbase-server hbase-mapreduce hbase-backup U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5775/3/console
versions	git=2.34.1 maven=3.8.6
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

… for the same timestamp is in different hfiles (apache#5775) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault force-pushed the HBASE-28456 branch from e788c26 to cdef073Compare March 25, 2024 21:09

HBASE-28456 HBase Restore restores old data if data for the same time…
5c744b1
…stamp is in different hfiles

bbeaudreault force-pushed the HBASE-28456 branch from cdef073 to 5c744b1Compare March 25, 2024 21:27

bbeaudreault requested review from Apache9 and ndimiduk March 26, 2024 10:53

ndimiduk approved these changes Mar 26, 2024
View reviewed changes

review comments
2669a33

Apache9 approved these changes Mar 26, 2024
View reviewed changes

checkstyle
6048c4d

bbeaudreault merged commit feef744 into apache:masterMar 26, 2024

bbeaudreault deleted the HBASE-28456 branch March 26, 2024 14:31

bbeaudreault added a commit that referenced this pull request Mar 26, 2024
HBASE-28456HBase Restore restores old data if data for the same time…
cea2439
…stamp is in different hfiles (#5775) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault added a commit that referenced this pull request Mar 26, 2024
HBASE-28456HBase Restore restores old data if data for the same time…
a53008b
…stamp is in different hfiles (#5775) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

bbeaudreault added a commit that referenced this pull request Mar 26, 2024
HBASE-28456HBase Restore restores old data if data for the same time…
f6091b8
…stamp is in different hfiles (#5775) Signed-off-by: Nick Dimiduk <ndimiduk@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org>

Uh oh!

Conversation

bbeaudreault commented Mar 25, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Apache-HBase commented Mar 25, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Apache-HBase commented Mar 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bbeaudreault commented Mar 25, 2024•
edited
Loading