[BLOCKER][FIX] CLOUDSTACK-9409: Add role_id to cloud_usage.account

[yadvr]

Adds role_id column to cloud_usage.account, fixes UsageDaoImpl to insert
Accounts with role_id from account table. Without the fix, usage server fails to work.

This fixes a blocker for 4.9.0 reported by @nvazquez (thanks!)

/cc @swill@nvazquez for review and merge thanks.

[yadvr]

@nvazquez I've tested and with this fix unable to reproduce your issue, can you help review and test this PR? Thanks.

[swill]

CI RESULTS

Tests Run: 103
Skipped: 0
Failed: 1
Errors: 2
Duration: 4h 39m 38s

Summary of the problem(s):

FAIL: test_02_vpc_privategw_static_routes (integration.smoke.test_privategw_acl.TestPrivateGwACL)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 262, in test_02_vpc_privategw_static_routes
self.performVPCTests(vpc_off)
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 325, in performVPCTests
privateGw_1 = self.createPvtGw(vpc_1, "10.0.3.100", "10.0.3.101", acl1.id, vlan_1)
File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 595, in createPvtGw
self.fail("Failed to create Private Gateway ==> %s" % e)
AssertionError: Failed to create Private Gateway ==> Execute cmd: createprivategateway failed, due to: errorCode: 431, errorText:Network with vlan vlan://100 already exists in zone 1
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_AVQEMB/results.txt

ERROR: Test Create/Delete a manual snap shot and verify
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs2/cloudstack/test/integration/component/test_project_usage.py", line 1376, in test_01_snapshot_usage
snapshot = Snapshot.create(self.apiclient, volumes[0].id)
File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 1106, in create
return Snapshot(apiclient.createSnapshot(cmd).__dict__)
File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 2857, in createSnapshot
response = self.connection.marvinRequest(command, response_type=response, method=method)
File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest
raise e
CloudstackAPIException: Execute cmd: createsnapshot failed, due to: errorCode: 530, errorText:KVM Snapshot is not supported: 1
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_usage_AUEI82/results.txt

ERROR: Test Create/Delete a manual snap shot and verify
----------------------------------------------------------------------
Traceback (most recent call last):
File "/data/git/cs2/cloudstack/test/integration/component/test_usage.py", line 1300, in test_01_snapshot_usage
snapshot = Snapshot.create(self.apiclient, volumes[0].id)
File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 1106, in create
return Snapshot(apiclient.createSnapshot(cmd).__dict__)
File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 2857, in createSnapshot
response = self.connection.marvinRequest(command, response_type=response, method=method)
File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest
raise e
CloudstackAPIException: Execute cmd: createsnapshot failed, due to: errorCode: 530, errorText:KVM Snapshot is not supported: 1
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_usage_AUEI82/results.txt

Associated Uploads

/tmp/MarvinLogs/DeployDataCenter__Jun_08_2016_18_27_57_AIBI32:

• dc_entries.obj
• failed_plus_exceptions.txt
• runinfo.txt

/tmp/MarvinLogs/test_network_AVQEMB:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

/tmp/MarvinLogs/test_usage_AUEI82:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

/tmp/MarvinLogs/test_vpc_routers_TN1YTG:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

Uploads will be available until 2016-08-08 02:00:00 +0200 CEST

Comment created by upr comment.

[swill]

I ran all of the usage tests against this PR. The last two errors have always happened on this setup for me, so I don't think they are related to this test. The first one looks like a cleanup issue. I will run it again to verify...

[yadvr]

Thanks @swill

[swill]

CI RESULTS

Tests Run: 4
Skipped: 0
Failed: 0
Errors: 0
Duration: 0h 18m 33s

Associated Uploads

/tmp/MarvinLogs/DeployDataCenter__Jun_08_2016_18_27_57_AIBI32:

• dc_entries.obj
• failed_plus_exceptions.txt
• runinfo.txt

/tmp/MarvinLogs/test_privategw_acl_0IG3FA:

• failed_plus_exceptions.txt
• results.txt
• runinfo.txt

Uploads will be available until 2016-08-09 02:00:00 +0200 CEST

Comment created by upr comment.

[swill]

Reran the test that failed that was not expected to fail and it passed now. I think we are good to go here...

[swill]

I would like to get at least one review of this before we merge it. @nvazquez would you be able to review this?

[syed]

LGTM based on the code review. 👍

[nvazquez]

Thanks @rhtyd for this fix!

Just for the record, this was error before this fix:

2016-06-07 11:04:51,233 ERROR [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Exception in usage manager
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@3202d58a: SELECT account.id, account.account_name, account.type, account.role_id, account.domain_id, account.state, account.removed, account.cleanup_needed, account.network_domain, account.uuid, account.default_zone_id, account.default FROM account WHERE account.id = 2
...
Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column 'account.role_id' in 'field list' 

And now I can see usage jobs work properly:

2016-06-09 00:00:52,561 DEBUG [cloud.usage.UsageManagerImpl] (Usage-HB-1:null) (logid:) Scheduling Usage job...
2016-06-09 00:00:52,561 INFO [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) starting usage job...
2016-06-09 00:00:52,568 INFO [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) Parsing usage records between Wed Jun 08 00:00:00 PDT 2016 and Wed Jun 08 23:59:59 PDT 2016
....
2016-06-09 00:00:53,436 INFO [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) (logid:) usage job complete

LGTM

[swill]

Ok, thanks guys. I will get this merged into 4.9. Thank you for the bug report and fix...

[nvazquez]

Cool, thanks @swill!