Externalize kvm agent storage timeout configuration

[GutoVeronezi]

Description

On KVMHAMonitor, the heartbeat timeout is hardcoded to 60000 ms; therefore, operators do not have options to choose which timeout they want/need.

This PR intends to externalize the heartbeat timeout configuration on KVMHAMonitor.

Depending on the use case, this timeout might need increasing/decreasing. Therefore, it is interesting to externalize such configurations for cloud operators. The hardcoded value is taken as the default one. Therefore, the current behavior is maintained.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functioanlity)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

How Has This Been Tested?

It has been tested locally on a test lab.

1. I changed my log4j to debug mode.
2. I added and changed hearbeat.update.timeout setting to agent.properties file several times.
3. I restarted agent and watched the log.

[GabrielBrascher]

Thanks for the PR @GutoVeronezi.

Is there any reason for not adding the new variable at agent.properties?

Normally, a PR adding a new variable would also add it on the properties file.

Example:

# commented lines describing variable
# commented (or not) line with the property name and default value

[GutoVeronezi]

@GabrielBrascher sorry, my bad.

I forgot to add the new variable to the file.

I'll add it later.

[ravening]

@ustcweizhou do we have this param already?

[GabrielBrascher]

@ravening this parameter exists already but it is hardcoded in KVMHABase.java, it is set as:
protected long _heartBeatUpdateTimeout = 60000;

[ravening]

@ravening this parameter exists already but it is hardcoded in KVMHABase.java, it is set as:
protected long _heartBeatUpdateTimeout = 60000;

@GabrielBrascher true but I know that Wei added several global settings to configure the values as well the preferred action as well in a much simpler way. I see in this PR that some new files are added to handle the properties which are not needed

[GutoVeronezi]

@ravening this parameter exists already but it is hardcoded in KVMHABase.java, it is set as:
protected long _heartBeatUpdateTimeout = 60000;

@GabrielBrascher true but I know that Wei added several global settings to configure the values as well the preferred action as well in a much simpler way. I see in this PR that some new files are added to handle the properties which are not needed

@ravening can you show me where this is externalized (if it is)? As far as we have seen in the code, there is no way for operators to configure this right now. Therefore, we need this patch to allow operators to configure the storage check timeout.

[GutoVeronezi]

@RodrigoDLopez@GabrielBrascher@DaanHoogland@sureshanaparti I'm pinging you here because you all reviewed PR 4586.

First I want to apologize for this inconvenience. This PR and 4586 are some of my first PRs and I messed up they a little bit.

Both PRs proposes differents changes, but they are linked by the new classes AgentPropertiesFileHandler and AgentProperties. These new classes were introducted in this PR and 4586 was proposed based on it; I should have waited for this to be merged and then propose that.

I marked 4586 as draft and applied yours suggestions in this PR. Also I added unit tests to the handler. If it be merged, I will rebase 4586.

Could you review this PR?

[DaanHoogland]

looks good, one question and one remark towards design;
q: properties are always read from file. Is that always what we want?
r: the use of the AgentProperties<> (plural) is to instantiate a single property. I think it makes sense to split the class in a container AgentProperties, and an element AgentProperty<>

code looks good

[DaanHoogland]

I suppose 60000 is the default (1 min) wouldn't hurt to make that explicit in here for operator convenience.

[GutoVeronezi]

@DaanHoogland done.

[DaanHoogland]

👍 useful javadoc, nice to see

[DaanHoogland]

👍

[GutoVeronezi]

looks good, one question and one remark towards design;
q: properties are always read from file. Is that always what we want?
r: the use of the AgentProperties<> (plural) is to instantiate a single property. I think it makes sense to split the class in a container AgentProperties, and an element AgentProperty<>

code looks good

@DaanHoogland

About the remark:
As you suggested, AgentProperties now is a container of AgentProperty<>. I turned AgentProperty<> an inner class of AgentProperties to keep accessibility policy (private constructor).

About the question:
This PR intends to externalize the heartbeat timeout configuration on KVMHAMonitor via agent.properties; And the purpose of these classes is to retrieve the properties from file; So in the situation we are working, we always want the value from file.
Is there any other situation that I should consider?

[GabrielBrascher]

Thanks for the PR @GutoVeronezi, code LGTM.

[DaanHoogland]

About the question:
This PR intends to externalize the heartbeat timeout configuration on KVMHAMonitor via agent.properties; And the purpose of these classes is to retrieve the properties from file; So in the situation we are working, we always want the value from file.
Is there any other situation that I should consider?

I don't know but I could immagine there could be properties that require a restart on change and forbid a re-read from file. things like connection settings for instance. I am not sure. It was a legitimate question.

[GutoVeronezi]

About the question:
This PR intends to externalize the heartbeat timeout configuration on KVMHAMonitor via agent.properties; And the purpose of these classes is to retrieve the properties from file; So in the situation we are working, we always want the value from file.
Is there any other situation that I should consider?

I don't know but I could immagine there could be properties that require a restart on change and forbid a re-read from file. things like connection settings for instance. I am not sure. It was a legitimate question.

@DaanHoogland It's a good use case.
In both cases, in some point of time, we will have to retrieve the property from the file; Maybe the question is when implement the call to getPropertyValue. static final would call it only once and, for example, in this PR, the method getPropertyValue is called once in constructor of KVMHAMonitor, which is instantiated only once in LibvirtComputingResource.configure.

[DaanHoogland]

About the question:
This PR intends to externalize the heartbeat timeout configuration on KVMHAMonitor via agent.properties; And the purpose of these classes is to retrieve the properties from file; So in the situation we are working, we always want the value from file.
Is there any other situation that I should consider?

I don't know but I could immagine there could be properties that require a restart on change and forbid a re-read from file. things like connection settings for instance. I am not sure. It was a legitimate question.

@DaanHoogland It's a good use case.
In both cases, in some point of time, we will have to retrieve the property from the file; Maybe the question is when implement the call to getPropertyValue. static final would call it only once and, for example, in this PR, the method getPropertyValue is called once in constructor of KVMHAMonitor, which is instantiated only once in LibvirtComputingResource.configure.

You are right and also, let's solve the issue when it really arises. this is good for now at least.

[blueorangutan]

Packaging result: ✔️ centos7 ✔️ centos8 ✔️ debian. SL-JID 188

[blueorangutan]

Trillian test result (tid-897)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 54700 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4585-t897-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 82 look OK, 6 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestResetVmOnReboot>:setup	Error	0.00	test_reset_vm_on_reboot.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup	Error	0.00	test_resource_accounting.py
ContextSuite context=TestRouterDHCPHosts>:setup	Error	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup	Error	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDns>:setup	Error	0.00	test_router_dns.py
test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true	Failure	334.77	test_routers_network_ops.py
test_02_RVR_Network_FW_PF_SSH_default_routes_egress_false	Failure	334.37	test_routers_network_ops.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Failure	458.49	test_vpc_redundant.py

[GabrielBrascher]

There are some errors on Marvin's tests. Not sure if they are related to this PR, though.

[DaanHoogland]

There are some errors on Marvin's tests. Not sure if they are related to this PR, though.

started a second run, we'll see if the errors are consistent.

[blueorangutan]

Trillian test result (tid-915)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 46008 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4585-t915-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 87 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_07_deploy_kubernetes_ha_cluster	Failure	3664.49	test_kubernetes_clusters.py
test_08_deploy_and_upgrade_kubernetes_ha_cluster	Failure	0.05	test_kubernetes_clusters.py
test_09_delete_kubernetes_ha_cluster	Failure	0.04	test_kubernetes_clusters.py
ContextSuite context=TestKubernetesCluster>:teardown	Error	77.38	test_kubernetes_clusters.py

[GutoVeronezi]

@DaanHoogland@GabrielBrascher is there anything else to do?

[DaanHoogland]

@GabrielBrascher regression tests look ok, did you test?

[yadvr]

LGTM do we need any manual testing? (automated tests LGTM but they may not cover the specific case)

[DaanHoogland]

Note that the threading model has changed @rhtyd , so yes manual testing would be needed, i'd say. I do not know any if automated tests exist for storage timeouts.

[GutoVeronezi]

Any update about this?

[RodrigoDLopez]

Hey @GutoVeronezi
Thank you for your contribution. You're doing an pretty job.

@DaanHoogland@GabrielBrascher @rhtyd

I ran some manual tests on this branch, and as far as I could verify everything works correctly.
Before upgrading to this branch, the agents log was always the same. Something like:

DEBUG [kvm.resource.KVMHAMonitor] Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 201.200.102.10 -p /export/primary2 -m /mnt/8c8fbcc3-eeef-33c1-adf6-51b915df61ca -h 201.200.101.4 DEBUG [kvm.resource.KVMHAMonitor] Executing while with timeout : 60000
DEBUG [kvm.resource.KVMHAMonitor] Execution is successful.

After updating, I ran some tests changing the parameter value and checking if the default value would be used.

• with the default value of heartbeat.update.timeout (commented or missing parameter)

DEBUG [kvm.resource.KVMHAMonitor] Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 201.200.102.10 -p /export/primary2 -m /mnt/8c8fbcc3-eeef-33c1-adf6-51b915df61ca -h 201.200.101.4 DEBUG [kvm.resource.KVMHAMonitor] Executing while with timeout : 60000
DEBUG [kvm.resource.KVMHAMonitor] Execution is successful.

• heartbeat.update.timeout=30000

DEBUG [kvm.resource.KVMHAMonitor] Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 201.200.102.10 -p /export/primary2 -m /mnt/8c8fbcc3-eeef-33c1-adf6-51b915df61ca -h 201.200.101.4 DEBUG [kvm.resource.KVMHAMonitor] Executing while with timeout : 30000
DEBUG [kvm.resource.KVMHAMonitor] Execution is successful.

• heartbeat.update.timeout=10000

DEBUG [kvm.resource.KVMHAMonitor] Executing: /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 201.200.102.10 -p /export/primary2 -m /mnt/8c8fbcc3-eeef-33c1-adf6-51b915df61ca -h 201.200.101.4 DEBUG [kvm.resource.KVMHAMonitor] Executing while with timeout : 10000
DEBUG [kvm.resource.KVMHAMonitor] Execution is successful.

[DaanHoogland]

I found an issue after merging this. addHost fails in several environments:

I think we'll need to revert (which I'm testing now in a lab env)
the smoke tests succeeded so the problem is not this PR in isolation, but something got merged that conflicted with this probably.

cc @GutoVeronezi@GabrielBrascher@RodrigoDLopez@nvazquez

[nvazquez]

Thanks @DaanHoogland seems like reverting on the internal branch made addHost work again. Please update once tests are completed so this can be reverted

[blueorangutan]

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian. SL-JID 589

[DaanHoogland]

the revert indeed works, I re-tested this PR with this ^^ latest build but am reverting this as other work is being blocked. @GutoVeronezi please rebase your code and submit a new PR after testing it. sorry about this.

2021-07-19 15:00:12,092 DEBUG [c.c.a.ApiServlet] (qtp1233705144-19:ctx-27a6578a) (logid:4d64198f) ===START=== 10.0.35.79 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&clusterid=fead03fb-7b43-4b88-9996-1213244c666b&command=addHost&hypervisor=KVM&podid=6b29cd4a-67c7-4dad-b388-146d4db00283&response=json&url=http%3A%2F%2F10.0.32.159&username=root&zoneid=333cd252-9808-4573-9a3e-f8133cd80024&signature=Soqqt4PW5siERQw%2FLaWG%2B27pAL0%3D
2021-07-19 15:00:12,095 DEBUG [c.c.a.ApiServer] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605) (logid:4d64198f) CIDRs from which account 'Acct[36d8ca2c-e8a1-11eb-8bd8-1e008a0003e2-admin] -- Account {"id": 2, "name": "admin", "uuid": "36d8ca2c-e8a1-11eb-8bd8-1e008a0003e2"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-07-19 15:00:12,111 INFO [c.c.r.ResourceManagerImpl] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Trying to add a new host at http://10.0.32.159 in data center 1
2021-07-19 15:00:12,328 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Executing cmd: ls /dev/kvm
2021-07-19 15:00:13,544 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) SSH command: ls /dev/kvm
SSH command output:/dev/kvm
2021-07-19 15:00:13,545 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Executing cmd: rpm -qa | grep -i ovmf
2021-07-19 15:00:13,593 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-5:ctx-d5f0f4cc) (logid:f64b89cf) HA health check task is running...
2021-07-19 15:00:15,224 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Executing cmd: rpm -qa | grep -i ovmf
2021-07-19 15:00:15,333 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d0938266) (logid:7fcaea1f) AutoScaling Monitor is running...
2021-07-19 15:00:15,334 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-ebeff815) (logid:797af13f) HostStatsCollector is running...
2021-07-19 15:00:15,341 DEBUG [c.c.s.StatsCollector] (StatsCollector-5:ctx-c89bff8b) (logid:80c2d343) StorageCollector is running...
2021-07-19 15:00:16,906 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Executing cmd: rpm -qa | grep -i ovmf
2021-07-19 15:00:17,594 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-615114d4) (logid:491f8fc6) HA health check task is running...
2021-07-19 15:00:18,618 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2021-07-19 15:00:18,818 DEBUG [c.c.a.ApiServlet] (qtp1233705144-18:ctx-05850c1a) (logid:ad1635fe) ===START=== 10.0.35.79 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=listHosts&response=json&signature=BYVU5RhMdLyIIO33ThMJS7BwqWk%3D
2021-07-19 15:00:18,821 DEBUG [c.c.a.ApiServer] (qtp1233705144-18:ctx-05850c1a ctx-7f427059) (logid:ad1635fe) CIDRs from which account 'Acct[36d8ca2c-e8a1-11eb-8bd8-1e008a0003e2-admin] -- Account {"id": 2, "name": "admin", "uuid": "36d8ca2c-e8a1-11eb-8bd8-1e008a0003e2"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2021-07-19 15:00:18,824 DEBUG [c.c.a.q.QueryManagerImpl] (qtp1233705144-18:ctx-05850c1a ctx-7f427059 ctx-1a31f824) (logid:ad1635fe) >>>Searching for hosts>>>
2021-07-19 15:00:18,828 DEBUG [c.c.a.q.QueryManagerImpl] (qtp1233705144-18:ctx-05850c1a ctx-7f427059 ctx-1a31f824) (logid:ad1635fe) >>>Generating Response>>>
2021-07-19 15:00:18,829 DEBUG [c.c.a.ApiServlet] (qtp1233705144-18:ctx-05850c1a ctx-7f427059 ctx-1a31f824) (logid:ad1635fe) ===END=== 10.0.35.79 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&command=listHosts&response=json&signature=BYVU5RhMdLyIIO33ThMJS7BwqWk%3D
2021-07-19 15:00:21,231 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
2021-07-19 15:00:21,314 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2021-07-19 15:00:21,595 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-6:ctx-bcbc4ac3) (logid:8233c09d) HA health check task is running...
2021-07-19 15:00:21,900 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-26411329) (logid:d1390140) Skip capacity scan as there is no Primary Storage in 'Up' state
2021-07-19 15:00:23,717 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or directory
2021-07-19 15:00:23,717 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2021-07-19 15:00:25,596 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-5:ctx-a24575f0) (logid:38c28e87) HA health check task is running...
2021-07-19 15:00:26,194 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or directory
2021-07-19 15:00:26,195 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2021-07-19 15:00:28,584 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or directory
2021-07-19 15:00:28,585 WARN [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
2021-07-19 15:00:28,588 WARN [c.c.r.ResourceManagerImpl] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Unable to find the server resources at http://10.0.32.159
2021-07-19 15:00:28,588 INFO [c.c.u.e.CSExceptionErrorCode] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2021-07-19 15:00:28,588 WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Exception: com.cloud.exception.DiscoveryException: Unable to add the host
at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:845)
at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:631)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy197.discoverHosts(Unknown Source)
at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:764)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:588)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:93)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:766)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.base/java.lang.Thread.run(Thread.java:829)
2021-07-19 15:00:28,590 INFO [c.c.a.ApiServer] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) Unable to add the host
2021-07-19 15:00:28,590 DEBUG [c.c.a.ApiServlet] (qtp1233705144-19:ctx-27a6578a ctx-7dc75605 ctx-d460a62f) (logid:4d64198f) ===END=== 10.0.35.79 -- GET apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&clusterid=fead03fb-7b43-4b88-9996-1213244c666b&command=addHost&hypervisor=KVM&podid=6b29cd4a-67c7-4dad-b388-146d4db00283&response=json&url=http%3A%2F%2F10.0.32.159&username=root&zoneid=333cd252-9808-4573-9a3e-f8133cd80024&signature=Soqqt4PW5siERQw%2FLaWG%2B27pAL0%3D

[blueorangutan]

Trillian Build Failed (tid-1318)

[GutoVeronezi]

@DaanHoogland np, let's get this working 😊.