CLOUDSTACK-10012: Towards CloudStack fat jar by yadvr · Pull Request #2226 · apache/cloudstack

yadvr · 2017-08-07T11:26:00Z

From #2207
I've moved the branch to ASF remote to allow cross-collaboration between ACS committers. Feel free to discuss and push your changes in this branch.

@blueorangutan package

yadvr · 2017-08-07T14:46:26Z

Travis green, so at least the mvn side of the world works: https://travis-ci.org/apache/cloudstack/builds/261836210 :)

yadvr · 2017-08-07T14:48:42Z

@blueorangutan package

blueorangutan · 2017-08-07T14:49:10Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

marcaurele · 2017-08-07T15:21:35Z

For postmortem, there are comments on this change in the linked PR #2207

blueorangutan · 2017-08-07T15:42:59Z

Packaging result: ✖centos6 ✖centos7 ✔debian. JID-928

yadvr · 2017-08-07T17:53:43Z

@blueorangutan package

blueorangutan · 2017-08-07T17:54:10Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2017-08-07T18:47:12Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-930

marcaurele · 2017-08-18T08:17:47Z

I came across another change that is required for the fatjar to work. We have to change the DbUpgrade process to not load File but return an array of InputStream since we have to load those files from inside the JAR. I'll push a change on the branch today

marcaurele · 2017-08-18T11:36:20Z

@rhtyd The DB files are kept out of the jar. Any opinion on that one if we should keep them outside (due to some possible customization for providers/plugins) or should we include them in the META-INF directory of the jar?

yadvr · 2017-08-18T12:20:08Z

@marcaurele I think we need to consider and discuss three things;

Whether to bundle the UI within the jar, or provider a webapp path to say /usr/share/cloudstack-management/webapp, in this directory we copy the UI asset files. This would allow admins to put their customizations. A solution could be to do both, so the fat jar has the UI files and however ServerDaemon would set the webapp path to the /usr path by default (if available) and fallback to assets in the jar when not found.
The DB files are kept out of the jar in case someone want to fix any upgrade failure issue. It's kind of subjective. My opinion is to bundle them in the jar, which reduces file management and opens scope for changes to the sql paths.
In addition, a lot of people may want to consume additional jars (plugins etc) which is why bounty castle and other jars may be put at /usr/share/cloudstack-management/lib. When mgmt server starts, we scan and include all the jars at this path in the classpath, with them first in the order in case someone want to overide any bundled jar etc.

What do you think?

marcaurele · 2017-08-21T12:15:55Z

@rhtyd

I will add a startup parameter -Dwebappdir=... to let the user change the path location instead of trying to guess things. Then if the path is incorrect at least it will return an error instead of loading the default UI. IMO it's better.
I'm in favor to bundle them inside the JAR. It's best for people customizing the installation to have an extra module where they do what they need, instead of editing files from the official releases.
+1 for the JARs ordering, that make sense for customization of the install with extra jars.

I'll push some of the changes today.

marcaurele · 2017-08-21T12:24:45Z

+setJavaHome
+
+JARS=$(ls /usr/share/cloudstack-management/lib/*.jar | tr '\n' ':' | sed s'/.$//')
+CLASSPATH="/etc/cloudstack/management:/usr/share/cloudstack-common:/usr/share/cloudstack-management/setup:/usr/share/cloudstack-management:/usr/share/cloudstack-management/cloud-management-server.jar:$JARS"


:$JARS should be put in front here to first load extra JARs

marcaurele · 2017-08-21T12:25:51Z

@rhtyd I cannot push to this branch

yadvr · 2017-08-21T13:41:20Z

@marcaurele perhaps you don't have commit rights yet, can you send a PR to the asf branch CLOUDSTACK-10012 and ping me I'll merge your changes right away

marcaurele · 2017-09-26T07:53:35Z

Related to 42d9297, I forgot to remove the jetty. in the class javadoc of ServerDaemon- if you could do it, thanks (PR seems overkill)

yadvr · 2017-09-26T09:03:22Z

Thanks @marcaurele, sure will do that. These days I'm down with some £dayjob project work but I've planned to resume working on this PR after next ~2 weeks.

yadvr · 2017-10-16T11:26:55Z

@blueorangutan package

blueorangutan · 2017-10-16T11:27:30Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

yadvr · 2017-10-16T11:30:15Z

@marcaurele I've resumed working on this now. I've squashed changes by authors, after painfully going through 27+ commits while rebasing against the latest master, hope you don't mind. I did not see any classes being overwritten for bountycastle and hence removed it from the exclusion list, do you think that could cause any issues? I'm exploring some ways to further improve the ServerDaemon launcher by using a customer properties or yml file that can be used as a single point to configure ports, listening interfaces, https, keystore etc. I've also upgraded several dependencies, notably Spring, bountycastle, and jetty.

blueorangutan · 2017-10-16T13:15:29Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2017-10-16T13:42:14Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1164

yadvr · 2017-11-01T12:18:33Z

@blueorangutan test

blueorangutan · 2017-11-01T12:19:26Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

wido · 2017-11-01T14:30:36Z

I just checked, building DEB files worked flawlessly. cloudstack.jar is 73M in size. Good!

yadvr · 2017-11-02T10:25:53Z

Thanks @wido, is that a LGTM on this PR?

blueorangutan · 2017-11-02T19:14:55Z

Trillian test result (tid-1606)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41902 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2226-t1606-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_host_annotations.py
Intermitten failure detected: /marvin/tests/smoke/test_iso.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_ssvm.py
Intermitten failure detected: /marvin/tests/smoke/test_templates.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Test completed. 54 look OK, 9 have error(s)

Test	Result	Time (s)	Test File
test_07_resize_fail	`Failure`	15.34	test_volumes.py
test_10_attachAndDetach_iso	`Failure`	1513.31	test_vm_life_cycle.py
test_09_destroy_ssvm	`Failure`	0.04	test_ssvm.py
test_07_reboot_ssvm	`Failure`	0.04	test_ssvm.py
test_05_stop_ssvm	`Failure`	0.04	test_ssvm.py
test_03_ssvm_internals	`Failure`	0.03	test_ssvm.py
test_01_list_sec_storage_vm	`Failure`	0.04	test_ssvm.py
test_04_rvpc_privategw_static_routes	`Failure`	223.27	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	117.63	test_privategw_acl.py
test_02_vpc_privategw_static_routes	`Failure`	117.59	test_privategw_acl.py
test_01_vpc_privategw_acl	`Failure`	46.57	test_privategw_acl.py
test_04_extract_Iso	`Failure`	5.08	test_iso.py
ContextSuite context=TestVpcSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestRVPCSite2SiteVpn>:setup	`Error`	0.00	test_vpc_vpn.py
ContextSuite context=TestVPCRedundancy>:setup	`Error`	0.00	test_vpc_redundant.py
test_06_download_detached_volume	`Error`	20.28	test_volumes.py
test_05_create_template_with_no_checksum	`Error`	65.61	test_templates.py
test_04_extract_template	`Error`	5.09	test_templates.py
test_04_create_template_with_checksum_md5	`Error`	65.51	test_templates.py
test_03_delete_template	`Error`	5.09	test_templates.py
test_03_create_template_with_checksum_sha256	`Error`	65.51	test_templates.py
test_02_create_template_with_checksum_sha1	`Error`	65.48	test_templates.py
test_01_create_template	`Error`	65.66	test_templates.py
test_05_add_annotation_for_invalid_entityType	`Error`	0.07	test_host_annotations.py
test_change_service_offering_for_vm_with_snapshots	Skipped	0.00	test_vm_snapshots.py
test_09_copy_delete_template	Skipped	0.02	test_templates.py
test_06_copy_template	Skipped	0.00	test_templates.py
test_static_role_account_acls	Skipped	0.02	test_staticroles.py
test_11_ss_nfs_version_on_ssvm	Skipped	0.02	test_ssvm.py
test_01_scale_vm	Skipped	0.00	test_scale_vm.py
test_01_primary_storage_iscsi	Skipped	0.09	test_primary_storage.py
test_vm_nic_adapter_vmxnet3	Skipped	0.00	test_nic_adapter_type.py
test_03_nic_multiple_vmware	Skipped	1.06	test_nic.py
test_nested_virtualization_vmware	Skipped	0.00	test_nested_virtualization.py
test_06_copy_iso	Skipped	0.00	test_iso.py
test_list_ha_for_host_valid	Skipped	0.04	test_hostha_simulator.py
test_list_ha_for_host_invalid	Skipped	0.04	test_hostha_simulator.py
test_list_ha_for_host	Skipped	0.03	test_hostha_simulator.py
test_hostha_enable_feature_without_setting_provider	Skipped	0.02	test_hostha_simulator.py
test_hostha_enable_feature_valid	Skipped	0.02	test_hostha_simulator.py
test_hostha_disable_feature_valid	Skipped	0.02	test_hostha_simulator.py
test_hostha_configure_invalid_provider	Skipped	0.02	test_hostha_simulator.py
test_hostha_configure_default_driver	Skipped	0.03	test_hostha_simulator.py
test_ha_verify_fsm_recovering	Skipped	0.03	test_hostha_simulator.py
test_ha_verify_fsm_fenced	Skipped	0.02	test_hostha_simulator.py
test_ha_verify_fsm_degraded	Skipped	0.02	test_hostha_simulator.py
test_ha_verify_fsm_available	Skipped	0.03	test_hostha_simulator.py
test_ha_multiple_mgmt_server_ownership	Skipped	0.03	test_hostha_simulator.py
test_ha_list_providers	Skipped	0.03	test_hostha_simulator.py
test_ha_enable_feature_invalid	Skipped	0.02	test_hostha_simulator.py
test_ha_disable_feature_invalid	Skipped	0.02	test_hostha_simulator.py
test_ha_configure_enabledisable_across_clusterzones	Skipped	0.02	test_hostha_simulator.py
test_configure_ha_provider_valid	Skipped	0.03	test_hostha_simulator.py
test_configure_ha_provider_invalid	Skipped	0.03	test_hostha_simulator.py
test_deploy_vgpu_enabled_vm	Skipped	0.04	test_deploy_vgpu_enabled_vm.py
test_3d_gpu_support	Skipped	0.04	test_deploy_vgpu_enabled_vm.py

- Changes packaging to shaded jar - Introduced embedded jetty - Updates jetty dependencies version 9.2 (java 7+) - Initial changes to debian config files - Initial removal of tomcat dependency - Introduce ServerDaemon class with customizations

borisstoyanov · 2017-11-03T12:04:46Z

@rhtyd after upgrading from 4.5 I wasn't able to start management server, got the following:

[root@trl-1020-v-cs45-bstoyanov-mgmt1 ~]# service cloudstack-management restart
ls: cannot access /usr/share/cloudstack-management/lib/*.jar: No such file or directory
Stopping CloudStack Management Server cloudstack-management[ OK ]
cat: /var/run/cloudstack-management.pid: No such file or directory]
[FAILED]

yadvr · 2017-11-03T12:04:54Z

Rebased to latest master. This is ready for merge now.
@blueorangutan package

blueorangutan · 2017-11-03T12:05:25Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

yadvr · 2017-11-03T12:05:51Z

@borisstoyanov okay I'll look into it

blueorangutan · 2017-11-03T12:32:08Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1201

wido · 2017-11-03T12:52:06Z

Yes, LGTM from me.

I tested a clean install on Ubuntu 16.04 with MySQL 5.7 ( https://blog.widodh.nl/2017/04/apache-cloudstack-and-mysql-5-7/ )

May I however ask that in server.properties this is set:

# The binding interface used by the management server
bind.interface=::

This will make sure that the server also listens on IPv6

wido · 2017-11-03T12:54:35Z

+# under the License.
+
+# The binding interface used by the management server
+bind.interface=0.0.0.0


Can we change 0.0.0.0 to :: so that it listens on IPv6 by default?
I tested this on Ubuntu 16.04 and works fine:
root@cloudstack-man-test:~# curl -I http://127.0.0.1:8080/client/ HTTP/1.1 200 OK Content-Type: text/html ETag: W/"114557-1499056602000 Content-Length: 114557 Server: Jetty(9.2.z-SNAPSHOT) root@cloudstack-man-test:~# curl -I http://[::1]:8080/client/ HTTP/1.1 200 OK Content-Type: text/html ETag: W/"114557-1499056602000 Content-Length: 114557 Server: Jetty(9.2.z-SNAPSHOT) root@cloudstack-man-test:~#

Hi @wido I think the standard I've seen in mysql and elsewhere is to use this. It's a matter of config issue, and ipv6 may not be enabled on all systems.
For me it fails when JAVA_OPT has -Djava.net.preferIPv4Stack=true set, and bind.interface has :: set. Exception seen is:
WARN [o.e.j.u.c.AbstractLifeCycle] (main:null) (logid:) FAILED ServerConnector@4e9ba398{HTTP/1.1}{:::8080}: java.net.SocketException: Protocol family unavailable java.net.SocketException: Protocol family unavailable at sun.nio.ch.Net.bind0(Native Method)
Command ran to test this case (build cloudstack and cd to client/target, fix conf/server.properties): java -Dlog4j.configuration=conf/log4j-cloud.xml -cp cloud-client-ui-4.11.0.0-SNAPSHOT.jar:lib/bcpkix-jdk15on-1.58.jar:lib/bcprov-jdk15on-1.58.jar:.:/usr/share/java/mysql-connector-java.jar:conf:common:utilities/scripts/db/ -Djava.net.preferIPv4Stack=true org.apache.cloudstack.ServerDaemon

Yes, that's true. However, all modern Linux systems have IPv6 enabled. It's a manual action to disable IPv6 on a system.
You don't need a public IPv6 address though to listen on ::, you already have your localhost ::1 to listen on.
Don't forget that the current Tomcat6/7 already listen on IPv6 by default. So it would be a change from 4.10 to 4.11 if we wouldn't listen on IPv6 anymore.

@wido I agree with you, however, for the default I needed to put some configuration that would work out of the box on most/all systems. I know several installations where they have put -Djava.net.preferIPv4Stack=true in their global JAVA opts (or in cloudstack specific env/init/systemd files). That said, we can have a compromise where I can remove the default value set to empty, i.e. bind.interface=, that way the system picks up the default interface that could work automatically.
With this approach, on systems without ipv6, this listens on:
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN
And with ipv6 enabled, it listens on:
tcp6 0 0 127.0.0.1:8080 :::* LISTEN

@wido check now, I've made the above change.

yadvr · 2017-11-03T13:09:56Z

Thanks for testing and reviewing @wido.

blueorangutan · 2017-11-03T13:35:37Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1203

yadvr · 2017-11-03T14:01:38Z

I've confirmed manual upgrade (in-place) testing, from ACS 4.9.3.0 -> 4.11.0.0-SNAPSHOT on CentOS6, CentOS7 and Ubuntu 14.04. Worked as expected, other than seeding systemvmtemplates, the upgrade worked without issues, the old tomcat files are not removed. I'll update upgrade/release notes on the same after this PR gets merged.

- Migrate to embedded Jetty server. - Improve ServerDaemon implementation. - Introduce a new server.properties file for easier configuration. - Have a single /etc/default/cloudstack-management to configure env. - Reduce shaded jar file, removing unnecessary dependencies. - Upgrade to Spring 5.x, upgrade several jar dependencies. - Does not shade and include mysql-connector, used from classpath instead. - Upgrade and use bountcastle as a separate un-shaded jar dependency. - Remove tomcat related configuration and files. - Have both embedded UI assets in uber jar and separate webapp directory. - Refactor systemd and init scripts, cleanup packaging. - Made cloudstack-setup-databases faster, using `urandom`. - Remove unmaintained distro packagings. - Moves creation and usage of server keystore in CA manager, this deprecates the need to create/store cloud.jks in conf folder and the db.cloud.keyStorePassphrase in db.properties file. This also remove the need of the --keystore-passphrase in the cloudstack-setup-encryption script. - GZip contents dynamically in embedded Jetty Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

borisstoyanov

Upgrade was successful on:
CentOS7 4.9 -> master
CentOS6 4.5 -> master
LGTM

yadvr · 2017-11-03T18:27:01Z

LGTM, merging based on reviews, tests and upgrade-tests. Thank you everyone for helping with reviews and feedback.

@wido

Per @wido's comment on PR #2226, this adds default server configuration to make embedded Jetty listen on all interfaces ipv4 and ipv6. This also fixes default deployment and mgmt server start issues on Trillian and other CI systems. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

borisstoyanov

I've setup and configured rabbitmq broker and it worked out of the box, following the official documentation.

yadvr mentioned this pull request Aug 7, 2017
Embedded jetty inside shaded jar #2207
Closed
4 tasks

asfgit force-pushed the CLOUDSTACK-10012 branch from c6b707f to df16dfeCompare August 8, 2017 02:09

kiwiflyer added the type:enhancement label Aug 8, 2017

yadvr added component:packaging status:work-in-progress labels Aug 10, 2017

wido added the component:management-server label Aug 14, 2017

wido added this to the 4.11 milestone Aug 14, 2017

marcaurele reviewed Aug 21, 2017
View reviewed changes

yadvr force-pushed the CLOUDSTACK-10012 branch from 42d9297 to 5291c51Compare October 16, 2017 11:26

asfgit force-pushed the CLOUDSTACK-10012 branch from 9c89b70 to c3b4cadCompare November 3, 2017 12:04

asfgit force-pushed the CLOUDSTACK-10012 branch from c3b4cad to 8323ccdCompare November 3, 2017 12:08

asfgit force-pushed the CLOUDSTACK-10012 branch from 8323ccd to c002597Compare November 3, 2017 12:45

wido reviewed Nov 3, 2017
View reviewed changes

asfgit force-pushed the CLOUDSTACK-10012 branch from c002597 to 749b059Compare November 3, 2017 13:03

asfgit force-pushed the CLOUDSTACK-10012 branch from 749b059 to d6a3dbfCompare November 3, 2017 13:11

asfgit force-pushed the CLOUDSTACK-10012 branch from d6a3dbf to a04da57Compare November 3, 2017 14:11

borisstoyanov approved these changes Nov 3, 2017
View reviewed changes

yadvr merged commit eda3b35 into masterNov 3, 2017

yadvr mentioned this pull request Nov 4, 2017
[4.11/master] Smoketest Health Check #2289
Closed

borisstoyanov reviewed Nov 7, 2017
View reviewed changes

rafaelweingartner deleted the CLOUDSTACK-10012 branch April 3, 2018 21:40

Uh oh!

Conversation

yadvr commented Aug 7, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yadvr commented Aug 7, 2017

Uh oh!

yadvr commented Aug 7, 2017

Uh oh!

blueorangutan commented Aug 7, 2017

Uh oh!

marcaurele commented Aug 7, 2017

Uh oh!

blueorangutan commented Aug 7, 2017

Uh oh!

yadvr commented Aug 7, 2017

Uh oh!

blueorangutan commented Aug 7, 2017

Uh oh!

blueorangutan commented Aug 7, 2017

Uh oh!

marcaurele commented Aug 18, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcaurele commented Aug 18, 2017

Uh oh!

yadvr commented Aug 18, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

marcaurele commented Aug 21, 2017

Uh oh!

marcaureleAug 21, 2017

Choose a reason for hiding this comment

Uh oh!

marcaurele commented Aug 21, 2017

Uh oh!

yadvr commented Aug 21, 2017

Uh oh!

marcaurele commented Sep 26, 2017

Uh oh!

yadvr commented Sep 26, 2017

Uh oh!

yadvr commented Oct 16, 2017

Uh oh!

blueorangutan commented Oct 16, 2017

Uh oh!

yadvr commented Oct 16, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueorangutan commented Oct 16, 2017

Uh oh!

blueorangutan commented Oct 16, 2017

Uh oh!

yadvr commented Nov 1, 2017

Uh oh!

blueorangutan commented Nov 1, 2017

Uh oh!

wido commented Nov 1, 2017

Uh oh!

yadvr commented Nov 2, 2017

Uh oh!

blueorangutan commented Nov 2, 2017

Uh oh!

borisstoyanov commented Nov 3, 2017

Uh oh!

yadvr commented Nov 3, 2017

Uh oh!

blueorangutan commented Nov 3, 2017

Uh oh!

yadvr commented Nov 3, 2017

Uh oh!

blueorangutan commented Nov 3, 2017

Uh oh!

wido commented Nov 3, 2017

Uh oh!

widoNov 3, 2017

Choose a reason for hiding this comment

Uh oh!

yadvrNov 3, 2017

Choose a reason for hiding this comment

yadvr commented Aug 7, 2017•
edited
Loading

marcaurele commented Aug 18, 2017•
edited
Loading

yadvr commented Aug 18, 2017•
edited
Loading

yadvr commented Oct 16, 2017•
edited
Loading

yadvr commented Nov 3, 2017•
edited
Loading

yadvr commented Nov 3, 2017•
edited
Loading