CLOUDSTACK-9305: Cloudstack Usage Breaks if DB HA enabled

[Slair1]

With DB HA enabled in db.properties, the cloudstack-usage service restarts every 10 seconds. Making the suggested change has fixed it for me. Cloudstack 4.8 on Centos7

[Slair1]

@kiwiflyer Very good point, let me make a different change here so it is more generic!

[Slair1]

If anyone still has issues, i also had to make the change from this PR: #1409

[kishankavala]

LGTM

[kiwiflyer]

Tested patch on a 4.7.1 lab.

Prior to patch:
INFO cloud.usage.UsageServer (logid:) UsageServer ready...
INFO context.support.ClassPathXmlApplicationContext (logid:) Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@70ea0c2a: startup date [Thu Mar 17 06:51:40 CDT 2016]; root of context hierarchy
INFO factory.xml.XmlBeanDefinitionReader (logid:) Loading XML bean definitions from class path resource [usageApplicationContext.xml]
INFO factory.annotation.AutowiredAnnotationBeanPostProcessor (logid:) JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
INFO context.support.ClassPathXmlApplicationContext (logid:) Bean 'transactionContextBuilder' of type [class com.cloud.utils.db.TransactionContextBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
INFO factory.support.DefaultListableBeanFactory (logid:) Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@33037ca5: defining beans [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,vmDiskUsageParser,networkOfferingUsageParser,VPNUserUsageParser,IPAddressUsageParser,VMSnapshotUsageParser,loadBalancerUsageParser,networkUsageParser,securityGroupUsageParser,storageUsageParser,portForwardingUsageParser,VMInstanceUsageParser,volumeUsageParser,usageManagerImpl,usageAlertManagerImpl,usageVmDiskDaoImpl,usageIPAddressDaoImpl,usageStorageDaoImpl,usageVolumeDaoImpl,usageDaoImpl,usagePortForwardingRuleDaoImpl,usageVPNUserDaoImpl,usageJobDaoImpl,usageLoadBalancerPolicyDaoImpl,usageNetworkDaoImpl,externalPublicIpStatisticsDaoImpl,usageSecurityGroupDaoImpl,usageVMInstanceDaoImpl,usageNetworkOfferingDaoImpl,usageVMSnapshotDaoImpl,usageEventDetailsDaoImpl,usageEventDaoImpl,eventDaoImpl,SSHKeyPairDaoImpl,userDaoImpl,vmDiskStatisticsDaoImpl,userAccountDaoImpl,accountDaoImpl,userStatsLogDaoImpl,userStatisticsDaoImpl,resourceCountDaoImpl,resourceLimitDaoImpl,alertDaoImpl,domainDaoImpl,configurationDaoImpl,quotaEmailTemplatesDaoImpl,userVmDetailsDaoImpl,quotaUsageDaoImpl,quotaBalanceDaoImpl,quotaTariffDaoImpl,quotaAccountDaoImpl,serviceOfferingDaoImpl,quotaCreditsDaoImpl,quotaAlertManagerImpl,quotaStatementImpl,quotaManagerImpl,transactionContextBuilder,instantiatePostProcessor,ComponentContext,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy
DEBUG utils.crypt.EncryptionSecretKeyChecker (logid:) Encryption Type: file
INFO db.Transaction.Transaction (logid:) Is Data Base High Availiability enabled? Ans : true
INFO db.Transaction.Transaction (logid:) The slaves configured for Cloud Data base is/are : 10.103.0.154,10.103.0.155
--then service tries to restart every few seconds--

After patch:
INFO utils.component.ComponentContext (logid:) Starting com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c20890bf
INFO utils.component.ComponentContext (logid:) Starting org.apache.cloudstack.quota.QuotaStatementImpl_EnhancerByCloudStack_5d0529b4
INFO cloudstack.quota.QuotaStatementImpl (logid:) Starting Statement Manager
INFO utils.component.ComponentContext (logid:) Starting org.apache.cloudstack.quota.QuotaManagerImpl_EnhancerByCloudStack_3ffc3462
INFO cloudstack.quota.QuotaManagerImpl (logid:) Starting Quota Manager
INFO utils.component.ComponentContext (logid:) Starting com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_93e4d889
INFO cloud.usage.UsageManagerImpl (logid:) Starting Usage Manager
INFO utils.component.ComponentContext (logid:) Starting com.cloud.usage.UsageAlertManagerImpl_EnhancerByCloudStack_8fa0a2a9
INFO utils.component.ComponentContext (logid:) Starting org.apache.cloudstack.quota.QuotaAlertManagerImpl_EnhancerByCloudStack_9535c670
INFO cloudstack.quota.QuotaAlertManagerImpl (logid:) Starting Alert Manager
INFO cloud.usage.UsageServer (logid:) UsageServer ready...
--service starts correctly--

LGTM on e98f445

[swill]

@Slair1 can you please squash your commits. I will get this into my CI queue so we can get this merged... Thanks...

[kiwiflyer]

@swill Just FYI that this PR conflicts with #1409 (comment).

They fix different issues, but the patch conflicts.

[swill]

@kiwiflyer they conflict in what sense? They change different lines of code, so I don't expect git to have an issue with them. Can you clarify?

[kiwiflyer]

I applied 1433 and then tried to apply 1409, but 1433 changes prevent the later patch from matching previous lines. One or the other probably needs to be based on top of the other patch so it applies cleanly.

[Slair1]

@swill , i apologize, but this is my first PR. I squashed it in Github Desktop on my local copy and did a push, but it sees like it it just added additional commits at github.com. If could you let me know how to correct it i'll do so as soon as I can.

[swill]

No worries. :) This should work: https://github.com/ginatrapani/todo.txt-android/wiki/Squash-All-Commits-Related-to-a-Single-Issue-into-a-Single-Commit

Also described here: https://ariejan.net/2011/07/05/git-squash-your-latests-commits-into-one/

I have used this method in the past as well: http://stackoverflow.com/questions/616556/how-do-you-squash-commits-into-one-patch-with-git-format-patch?answertab=votes#tab-top

Let me know if you have problems and I will try to help you out.

[Slair1]

@swill i seem to get an access denied when pushing my change:
C:\slair\Source\Github\cloudstack [pr/1433]> git push origin pr/1433 --force
remote: Permission to apache/cloudstack.git denied to Slair1.
fatal: unable to access 'https://github.com/apache/cloudstack.git/': The requested URL returned error: 403

[swill]

No one is allowed to push code to apache/cloudstack, you have to push it to your fork where your PR is coming from. Does that make sense?

So you will be pushing to https://github.com/Slair1/cloudstack on branch patch-1. This will likely be setup as a different git remote It looks like your origin is pointing at the apache/cloudstack repo instead of your Slair1/cloudstack repo.

[Slair1]

@swill thanks, that makes sense. I did it to my branch the first time, but for some reason when i tried to save to here it just created two more commits. i'll try again

[swill]

Once you get your local branch setup correctly with the squashed commit, you should be able to do a push - f to replace what is in the PR with what you have locally. Let me know if you have problems and I can do a test to validate the steps you need to take.

[Slair1]

thanks @swill , it looks much better now, i needed the -f on the git push

[swill]

No problem. Ya it took me a couple pull requests to figure out how they work. Thanks for taking the time to do this for me. :)

[swill]

Running CI now. Will have results later tonight or tomorrow...

[swill]

CI RESULTS

Tests Run: 103
Skipped: 0
Failed: 3
Errors: 2

Summary of the problem(s):

ERROR: Test Create/Delete a manual snap shot and verify
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs2/cloudstack/test/integration/component/test_project_usage.py", line 1376, in test_01_snapshot_usage
snapshot = Snapshot.create(self.apiclient, volumes[0].id)
File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 1018, in create
return Snapshot(apiclient.createSnapshot(cmd).__dict__)
File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 797, in createSnapshot
response = self.connection.marvinRequest(command, response_type=response, method=method)
File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest
raise e
CloudstackAPIException: Execute cmd: createsnapshot failed, due to: errorCode: 530, errorText:KVM Snapshot is not supported: 1
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_usage_A3LTTO/results.txt

ERROR: Test Create/Delete a manual snap shot and verify
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs2/cloudstack/test/integration/component/test_usage.py", line 1300, in test_01_snapshot_usage
snapshot = Snapshot.create(self.apiclient, volumes[0].id)
File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 1018, in create
return Snapshot(apiclient.createSnapshot(cmd).__dict__)
File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 797, in createSnapshot
response = self.connection.marvinRequest(command, response_type=response, method=method)
File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest
raise e
CloudstackAPIException: Execute cmd: createsnapshot failed, due to: errorCode: 530, errorText:KVM Snapshot is not supported: 1
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_usage_A3LTTO/results.txt

FAIL: Test redundant router internals
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs2/cloudstack/test/integration/smoke/test_routers_network_ops.py", line 290, in test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true
"Attempt to retrieve google.com index page should be successful!"
AssertionError: Attempt to retrieve google.com index page should be successful!
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_4ISAIJ/results.txt

FAIL: test_02_vpc_privategw_static_routes (integration.smoke.test_privategw_acl.TestPrivateGwACL)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 253, in test_02_vpc_privategw_static_routes
self.performVPCTests(vpc_off)
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 324, in performVPCTests
self.check_pvt_gw_connectivity(vm1, public_ip_1, vm2.nic[0].ipaddress)
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 559, in check_pvt_gw_connectivity
"Ping to outside world from VM should be successful"
AssertionError: Ping to outside world from VM should be successful
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_4ISAIJ/results.txt

FAIL: test_03_vpc_privategw_restart_vpc_cleanup (integration.smoke.test_privategw_acl.TestPrivateGwACL)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 265, in test_03_vpc_privategw_restart_vpc_cleanup
self.performVPCTests(vpc_off, True)
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 324, in performVPCTests
self.check_pvt_gw_connectivity(vm1, public_ip_1, vm2.nic[0].ipaddress)
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 559, in check_pvt_gw_connectivity
"Ping to outside world from VM should be successful"
AssertionError: Ping to outside world from VM should be successful
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_4ISAIJ/results.txt

Associated Uploads

/tmp/MarvinLogs/DeployDataCenter__Apr_28_2016_22_51_48_T2XTTQ:

• dc_entries.obj
• failed_plus_exceptions.txt
• runinfo.txt

/tmp/MarvinLogs/test_network_4ISAIJ:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

/tmp/MarvinLogs/test_usage_A3LTTO:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

/tmp/MarvinLogs/test_vpc_routers_LS4J5A:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

Uploads will be available until 2016-06-29 02:00:00 +0200 CEST

Comment created by upr comment.

[swill]

None of the failures above are related to this PR, they are all issues I see periodically in my environment. All of the supported usage tests passed, so CI has passed for this PR. Sorry the output makes that hard to understand.

[Slair1]

That's ok, thanks!!

[kiwiflyer]

2 x LGTMs and CI complete. This is Ready to Merge.

[swill]

Sorry, I missed the LGTMs. :) I will add this to my merge queue. Thanks @kiwiflyer for following up.

[yadvr]

@Slair1 if dh ha jar is missing for example there is no db-ha deb package, what happens?

[Slair1]

Hi @rhtyd since it uses "ls" to get a list of packages, if the directory or .jar file does not exist it is no big deal. It will output a No such file or directory message but still loads any packages that exist without issue. the UCP variable still looks good when that occurs.

[yadvr]

fair enough, LGTM

tag:mergeready