Uh oh!
There was an error while loading. Please reload this page.
HDDS-10422. Fix some warnings about exposing internal representation in hdds-common - #6351
Merged
Conversation
szetszwo
left a comment
Contributor
There was a problem hiding this comment.
@adoroszlai , Thanks for working on this! Please see the comments inlined and also https://issues.apache.org/jira/secure/attachment/13068960/6351_review.patch
BTW, this change is quite big. How about separating it into at least two JIRAs?
Comment on lines
-100
to
-102
| if (info.hasStripeChecksum()) { | ||
| chunkInfo.setStripeChecksum(info.getStripeChecksum()); | ||
| } |
Contributor
There was a problem hiding this comment.
Is it the case that info.hasStripeChecksum() is always false? If yes, how about adding a precondition?
ContributorAuthor
There was a problem hiding this comment.
No, ChunkInfo proto may have stripe checksum. However, getStripeChecksum of the non-proto ChunkInfo is never called. I'll remove this part of the change.
| */ | ||
| public synchronized List<Port> getPorts() { | ||
| return ports; | ||
| return new ArrayList<>(ports); |
Contributor
There was a problem hiding this comment.
We should also change the callers in order to reduce copying:
- NodeDecommissionManager
//NodeDecommissionManagerprivatebooleanvalidateDNPortMatch(intport, DatanodeDetailsdn) {
returndn.containPort(port);
}//DatanodeDetailspublicsynchronizedbooleancontainPort(intport) {
for (Portp : ports) {
if (p.getValue() == port) {
returntrue;
}
}
returnfalse;
}- DatanodeIdYaml
@@ -238,8 +239,9@@privatestaticDatanodeDetailsYamlgetDatanodeDetailsYaml(
= newDatanodeLayoutStorage(conf, datanodeDetails.getUuidString());
Map<String, Integer> portDetails = newLinkedHashMap<>();
- if (!CollectionUtils.isEmpty(datanodeDetails.getPorts())) {
- for (DatanodeDetails.Portport : datanodeDetails.getPorts()) {
+ finalList<DatanodeDetails.Port> datanodePorts = datanodeDetails.getPorts();
+ if (!CollectionUtils.isEmpty(datanodePorts)) {
+ for (DatanodeDetails.Portport : datanodePorts) {
Fieldf = null;
try {
f = DatanodeDetails.Port.Name.classContributorAuthor
There was a problem hiding this comment.
Thanks for the suggestion, updated.
| */ | ||
| public List<String> getRatisPeerRoles() { | ||
| return peerRoles; | ||
| return Collections.unmodifiableList(peerRoles); |
Contributor
There was a problem hiding this comment.
Do it in constructor.
privateScmInfo(StringclusterId, StringscmId, List<String> peerRoles) {
this.clusterId = clusterId;
this.scmId = scmId;
this.peerRoles = Collections.unmodifiableList(peerRoles);
}| this.suggestedLeaderId = pipeline.getSuggestedLeaderId(); | ||
| if (nodeStatus != null) { | ||
| replicaIndexes = new HashMap<>(); | ||
| Map<DatanodeDetails, Integer> indexes = new HashMap<>(); |
Contributor
There was a problem hiding this comment.
Use builder:
finalImmutableMap.Builder<DatanodeDetails, Integer> b = ImmutableMap.builder();
for (DatanodeDetailsdn : nodeStatus.keySet()) {
intindex = pipeline.getReplicaIndex(dn);
if (index > 0) {
b.put(dn, index);
}
}
replicaIndexes = b.build();| public Builder setNodeOrder(List<Integer> orders) { | ||
| // for build from ProtoBuf | ||
| this.nodeOrder = orders; | ||
| this.nodeOrder = orders == null ? ImmutableList.of() : ImmutableList.copyOf(orders); |
Contributor
There was a problem hiding this comment.
Since the orders is from a proto, let's use this.nodeOrder = Collections.unmodifiableList(orders); to avoid copying.
| this.type = checksumType; | ||
| this.bytesPerChecksum = bytesPerChecksum; | ||
| this.checksums = checksums; | ||
| this.checksums = checksums == null ? ImmutableList.of() : ImmutableList.copyOf(checksums); |
Contributor
There was a problem hiding this comment.
Copy is not needed here since it is from proto or a new list.
this.checksums = Collections.unmodifiableList(checksums);| public LeaseCallbackExecutor(T resource, List<Callable<Void>> callbacks) { | ||
| this.resource = resource; | ||
| this.callbacks = callbacks; | ||
| this.callbacks = callbacks == null ? ImmutableList.of() : ImmutableList.copyOf(callbacks); |
Contributor
There was a problem hiding this comment.
callbacks always has zero or one element. Let's remove the List<>?
| quotaList.addQuotaList(OZONE_QUOTA_KB, Units.KB, KB); | ||
| quotaList.addQuotaList(OZONE_QUOTA_B, Units.B, 1L); | ||
| } | ||
| private static final QuotaList QUOTA_LIST = new QuotaList(); |
Contributor
There was a problem hiding this comment.
OzoneQuota used to initalize contents of QuotaList, moved to the latter
The QuotaList class is inefficient. Let's remove it.
adoroszlai
commented
May 22, 2024
ContributorAuthor
Thanks @szetszwo for the review. Deferred some changes that can be logically grouped to separate tasks: HDDS-10896, HDDS-10897 and HDDS-10899. |
This was referenced May 22, 2024
szetszwo
approved these changes
May 22, 2024
szetszwo
left a comment
Contributor
There was a problem hiding this comment.
+1 the change looks good.
adoroszlai
commented
May 23, 2024
ContributorAuthor
Thanks @szetszwo for reviewing and merging this. |
jojochuang pushed a commit
to jojochuang/ozone
that referenced
this pull request
May 23, 2024
errose28 added a commit
to errose28/ozone
that referenced
this pull request
May 28, 2024
…concile-cli * HDDS-10239-container-reconciliation: (296 commits) HDDS-10897. Refactor OzoneQuota (apache#6714) HDDS-10422. Fix some warnings about exposing internal representation in hdds-common (apache#6351) HDDS-10899. Refactor Lease callbacks (apache#6715) HDDS-10890. Increase default value for hdds.container.ratis.log.appender.queue.num-elements (apache#6711) HDDS-10832. Client should switch to streaming based on OpenKeySession replication (apache#6683) HDDS-10435. Support S3 object tags for existing requests (apache#6607) HDDS-10883. Improve logging in Recon for finalising DN logic. (apache#6704) HDDS-8752. Enable TestOzoneRpcClientAbstract#testOverWriteKeyWithAndWithOutVersioning (apache#6702) HDDS-10875. XceiverRatisServer#getRaftPeersInPipeline should be called before XceiverRatisServer#removeGroup (apache#6696) HDDS-10514. Recon - Provide DN decommissioning detailed status and info inline with current CLI command output. (apache#6376) HDDS-10878. Bump zstd-jni to 1.5.6-3 (apache#6701) HDDS-10877. Bump Dropwizard metrics to 3.2.6 (apache#6699) HDDS-10876. Bump jackson to 2.16.2 (apache#6697) HDDS-6116. Remove flaky tag from TestSCMInstallSnapshot (apache#6695) HDDS-2643. TestOzoneDelegationTokenSecretManager#testRenewTokenFailureRenewalTime fails intermittently. HDDS-10699. Refactor ContainerBalancerTask and TestContainerBalancerTask (apache#6537) HDDS-10861. Ozone cli supports default ozone.om.service.id (apache#6680) HDDS-10859. Improve error messages when decommission and maintenance fail-early (apache#6678) HDDS-9031. Upgrade acceptance tests to Docker Compose v2 (apache#6667) HDDS-10559. Add a warning or a check to run repair tool as System user (apache#6574) ... Conflicts: hadoop-ozone/dist/src/main/smoketest/admincli/container.robot
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix some Spotbugs (4.x) warnings that code
may expose internal representation. Notable changes:OzoneQuotaused to initalize contents ofQuotaList, moved to the lattermove creation of mutable metrics objects to(deferred to HDDS-10897)PerformanceMetricsAlso remove unused
hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/utils/BatchOperation.java(seems to be outdated duplicate ofhadoop-hdds/framework/src/main/java/org/apache/hadoop/hdds/utils/db/BatchOperation.java).https://issues.apache.org/jira/browse/HDDS-10422
How was this patch tested?
CI:
https://github.com/adoroszlai/ozone/actions/runs/8200024023