CLOUDSTACK-10012: Load SQL schema scripts from JAR

[marcaurele]

Load the SQL schema files from the fatjar directly to simply file path discovery.

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✖centos6 ✖centos7 ✔debian. JID-1025

[yadvr]

@marcaurele it might be an issue as cloudstack-setup-databases reads from that folder, and also it will become an issue to verify the changes in various older upgrade paths. Let me review and get back to you on this next week.

[borisstoyanov]

@marcaurele can you please edit the title to start with the JIRA number

[marcaurele]

From our experience on this change that is running in production for us, we can deploy a new SQL schema/lab with the usual mvn developer script, as well as upgrading the DB on schema changes. If a customization is needed, we should invert the way the upgrades are executed and do a package scan on classes rather than declaring them "statically" in DatabaseUpgradeChecker. It should help anyone to extend an upgrade class to add extra changes. The upgrade classes should declare themselves the version from which they should be run, and the DB version after the upgrade. With the help of a Comparable, we could sort them in the correct order, and even skip over unneeded ones if a class declares a bigger upgrade path.

[yadvr]

Thanks @marcaurele there are several conflicts and large changes, I'll work with you on this after the original fat-jar PR is tested and merged.

[yadvr]

@marcaurele with the fatjar PR merged, please rebase and switch the base branch of this PR to master. Let's get this reviewed, tested and merged as next step. Thanks.

[marcaurele]

@rhtyd rebase done + target branch changed

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[yadvr]

@marcaurele this will require two additional changes:

• Fix cloud(stack)-setup-databases script to use com.cloud.upgrade.DatabaseUpgradeChecker and remove assumptions that a setup folder will be available to read/run the sql scripts from. We'll need to do this while making sure cloudstack-setup-databases is backward compatible.
• Fix various packaging scripts to not copy/bundle setup as a separate directory

I don't have time this week, but can collaborate with you next week. I think I certainly can fix the packging bits.

[blueorangutan]

Packaging result: ✖centos6 ✖centos7 ✔debian. JID-1219

[marcaurele]

@rhtyd cloud-setup-databases is good to go as it is since I left the SQL files in setup/db directory to create the schema. Only the migration files are moved into the JAR. I found one trace of this sub directory in the centos packaging and remove the lines.

[marcaurele]

@rhtyd all checks are green, can you have a look or push further testing on this branch.

[yadvr]

Thanks @marcaurele - I'm wondering if it's still a good idea to keep the subdirectory with initial seed schema files while having the upgrade paths shaded/included in jar? I'll kick some tests.
@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1353

[yadvr]

@marcaurele can you fix the conflicts? I'll help run tests then.

[marcaurele]

@rhtyd rebase done

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1404

[yadvr]

@blueorangutan test

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-1818)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31511 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2247-t1818-kvm-centos7.zip
Smoke tests completed. 63 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_vpc_privategw_acl	Failure	61.22	test_privategw_acl.py
test_02_vpc_privategw_static_routes	Failure	287.69	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	Failure	162.67	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	Failure	378.41	test_privategw_acl.py
test_02_create_template_with_checksum_sha1	Error	5.14	test_templates.py
test_03_create_template_with_checksum_sha256	Error	5.14	test_templates.py
test_04_create_template_with_checksum_md5	Error	5.14	test_templates.py
test_01_volume_usage	Failure	316.04	test_usage.py
test_01_vpc_remote_access_vpn	Error	60.60	test_vpc_vpn.py

[yadvr]

LGTM, +1 on initial phase to include all sql upgrade paths in fatjar. Including the default schema/sql files and using the db creator class could be next step for future.