HDDS-9883. Recon - Improve the performance of processing IncrementalContainerReport from DN by devmadhuu · Pull Request #5793 · apache/ozone

devmadhuu · 2023-12-14T13:40:45Z

What changes were proposed in this pull request?

There is a difference between SCM's IncrementalContainerReportHandler and Recon's IncrementalContainerReportHandler , Recon always connects to SCM and verify each container before adding new container to its own containerStateManager cache. This could be a bottle neck if SCM may respond slow and frequent ICR requests may pile up in queue. So as of now, this PR will improve below multiple things:

Recon to verify containers in batches from SCM on receive of ICR request from DNs.

Reduce the scmClient configs for Recon before connecting to SCM:
- `hdds.scmclient.rpc.timeout` - 1 min (Default value is 15 mins)
- `hdds.scmclient.failover.max.retry` - 3 (Default value is dynamic and computed, but based on default values, computed value is 15)
Above 2 SCM client configs will be updated to respective new values as mentioned for recon to connect to SCM. These 2 SCM client configs will be exposed and mapped with new recon configs to be able to adjust independently in recon.
**New configs in Recon:**
- `ozone.recon.scmclient.rpc.timeout`
- `ozone.recon.scmclient.failover.max.retry`

Merge the Incremental container report (ICR) to existing list of ICR reports.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-9883

How was this patch tested?

Tested using existing Junit tests after updating existing test - TestReconIncrementalContainerReportHandler

…alContainerReport requests from DN.

devmadhuu · 2023-12-14T13:41:46Z

@sumitagrawl Pls review.

…alContainerReport requests from DN.

kerneltime · 2023-12-18T17:16:48Z

cc @tanvipenumudy

sumitagrawl

@devmadhuu Thanks for working over this, have few comments

…rectly.

sumitagrawl

@devmadhuu LGTM

jojochuang · 2024-02-23T02:27:02Z

I'm getting this exception which loosk related to this PR

2024-02-22 22:33:39,568 WARN [IPC Server handler 22 on 9891]-org.apache.hadoop.ipc.Server: IPC Server handler 22 on 9891, call Call#36 Retry#0 org.apache.hadoop.ozone.protocol.ReconDatanodeProtocol.submitRequest from 10.140.112.130:34368
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.addAll(Collections.java:1067)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher$IncrementalContainerReportFromDatanode.mergeReport(SCMDatanodeHeartbeatDispatcher.java:384)
at org.apache.hadoop.ozone.recon.scm.ReconContainerReportQueue.mergeIcr(ReconContainerReportQueue.java:41)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.addIncrementalReport(ContainerReportQueue.java:115)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.addValue(ContainerReportQueue.java:173)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.add(ContainerReportQueue.java:187)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.add(ContainerReportQueue.java:42)
at org.apache.hadoop.hdds.server.events.FixedThreadPoolWithAffinityExecutor.onMessage(FixedThreadPoolWithAffinityExecutor.java:178)
at org.apache.hadoop.hdds.server.events.EventQueue.fireEvent(EventQueue.java:220)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher.dispatch(SCMDatanodeHeartbeatDispatcher.java:159)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.sendHeartbeat(SCMDatanodeProtocolServer.java:282)
at org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.processMessage(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:113)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
at org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.submitRequest(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:92)
at org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$StorageContainerDatanodeProtocolService$2.callBlockingMethod(StorageContainerDatanodeProtocolProtos.java:45283)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)

devmadhuu · 2024-02-23T06:26:45Z

I'm getting this exception which loosk related to this PR
2024-02-22 22:33:39,568 WARN [IPC Server handler 22 on 9891]-org.apache.hadoop.ipc.Server: IPC Server handler 22 on 9891, call Call#36 Retry#0 org.apache.hadoop.ozone.protocol.ReconDatanodeProtocol.submitRequest from 10.140.112.130:34368
java.lang.UnsupportedOperationException
at java.util.Collections$UnmodifiableCollection.addAll(Collections.java:1067)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher$IncrementalContainerReportFromDatanode.mergeReport(SCMDatanodeHeartbeatDispatcher.java:384)
at org.apache.hadoop.ozone.recon.scm.ReconContainerReportQueue.mergeIcr(ReconContainerReportQueue.java:41)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.addIncrementalReport(ContainerReportQueue.java:115)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.addValue(ContainerReportQueue.java:173)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.add(ContainerReportQueue.java:187)
at org.apache.hadoop.hdds.scm.server.ContainerReportQueue.add(ContainerReportQueue.java:42)
at org.apache.hadoop.hdds.server.events.FixedThreadPoolWithAffinityExecutor.onMessage(FixedThreadPoolWithAffinityExecutor.java:178)
at org.apache.hadoop.hdds.server.events.EventQueue.fireEvent(EventQueue.java:220)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeHeartbeatDispatcher.dispatch(SCMDatanodeHeartbeatDispatcher.java:159)
at org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer.sendHeartbeat(SCMDatanodeProtocolServer.java:282)
at org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.processMessage(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:113)
at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
at org.apache.hadoop.ozone.protocolPB.StorageContainerDatanodeProtocolServerSideTranslatorPB.submitRequest(StorageContainerDatanodeProtocolServerSideTranslatorPB.java:92)
at org.apache.hadoop.hdds.protocol.proto.StorageContainerDatanodeProtocolProtos$StorageContainerDatanodeProtocolService$2.callBlockingMethod(StorageContainerDatanodeProtocolProtos.java:45283)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)

This issue is being tracked at HDDS-10413

…IncrementalContainerReport from DN (apache#5793) (cherry picked from commit 6021582) Change-Id: Ic9fa1c235a2c347cf9375f6a1da9a47f5dcba213

…ontainerReport from DN (apache#5793) (cherry picked from commit 6021582)

HDDS-9883. Recon - Improve the performance of processing of Increment…
7af329e
…alContainerReport requests from DN.

adoroszlai added the recon label Dec 14, 2023

HDDS-9883. Recon - Improve the performance of processing of Increment…
67ef32b
…alContainerReport requests from DN.

devmadhuu marked this pull request as ready for review December 15, 2023 06:07

adoroszlai changed the title ~~HDDS-9883. Recon - Improve the performance of processing of IncrementalContainerReport requests from DN.~~HDDS-9883. Recon - Improve the performance of processing IncrementalContainerReport from DNDec 15, 2023

kerneltime requested review from duongkame, kerneltime and sumitagrawl December 18, 2023 17:16

sumitagrawl reviewed Jan 3, 2024
View reviewed changes

Comment thread...p-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconContainerReportQueue.java Outdated
Comment thread...econ/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconStorageContainerManagerFacade.java Outdated

HDDS-9883. Fixed review comments to set update hdds property value di…
db63424
…rectly.

sumitagrawl approved these changes Jan 4, 2024
View reviewed changes

HDDS-9883. Fixed failure of TestOzoneConfigurationFields test case.
e6af3e8

sumitagrawl merged commit 6021582 into apache:masterJan 4, 2024

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024
HDDS-9883. Recon - Improve the performance of processing IncrementalC…
c3cedd0
…ontainerReport from DN (apache#5793) (cherry picked from commit 6021582)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024
HDDS-9883. Recon - Improve the performance of processing IncrementalC…
4554b36
…ontainerReport from DN (apache#5793) (cherry picked from commit 6021582)

xichen01 pushed a commit to xichen01/ozone that referenced this pull request Jul 17, 2024
HDDS-9883. Recon - Improve the performance of processing IncrementalC…
82c6118
…ontainerReport from DN (apache#5793) (cherry picked from commit 6021582)

xichen01 mentioned this pull request Jul 18, 2024
[DO NOT MERGE] Backport some fixes, performance optimizations from master to ozone-1.4 #6929 #6964
Merged

Navigation Menu

Uh oh!

HDDS-9883. Recon - Improve the performance of processing IncrementalContainerReport from DN - #5793

HDDS-9883. Recon - Improve the performance of processing IncrementalContainerReport from DN#5793
sumitagrawl merged 4 commits into
apache:masterfrom
devmadhuu:HDDS-9883

devmadhuu commented Dec 14, 2023

Uh oh!

devmadhuu commented Dec 14, 2023

Uh oh!

kerneltime commented Dec 18, 2023

Uh oh!

sumitagrawl left a comment

Uh oh!

Uh oh!

Uh oh!

sumitagrawl left a comment

Uh oh!

jojochuang commented Feb 23, 2024

Uh oh!

devmadhuu commented Feb 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

devmadhuu commented Dec 14, 2023

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

devmadhuu commented Dec 14, 2023

Uh oh!

kerneltime commented Dec 18, 2023

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sumitagrawl left a comment

Choose a reason for hiding this comment

Uh oh!

jojochuang commented Feb 23, 2024

Uh oh!

devmadhuu commented Feb 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants