Systemd packaging for Ubuntu 16.04 by wido · Pull Request #1541 · apache/cloudstack

wido · 2016-05-12T13:42:08Z

This PR is for introducing packaging for Ubuntu 16.04

Ubuntu 14.04 still uses sysvinit and Java 7. 16.04 uses systemd and Java 8.

The 'build-deb.sh' script can be used to generate packages. The comments in the file show how to use Docker to build for the different versions of Ubuntu.

I've just build:

cloudstack-agent_4.9.0-SNAPSHOT~xenial_all.deb
cloudstack-agent_4.9.0-SNAPSHOT~trusty_all.deb

wido · 2016-05-12T13:43:30Z

Asking @rhtyd for a review

yadvr · 2016-05-13T03:44:40Z

@wido thanks, much needed. I had a quick look, I could not find a cloudstack-management.sysconfig.

yadvr · 2016-05-13T05:49:33Z

I was able to build the packages on Ubuntu 16.04:
cloudstack-agent_4.9.0-SNAPSHOT~~xenial_all.deb cloudstack-common_4.9.0-SNAPSHOT~~xenial_all.deb cloudstack-management_4.9.0-SNAPSHOTxenial_all.deb
cloudstack-cli_4.9.0-SNAPSHOTxenial_all.deb cloudstack-docs_4.9.0-SNAPSHOT~~xenial_all.deb cloudstack-usage_4.9.0-SNAPSHOT~~xenial_all.deb

I was able to deploy-database using cloudstack-setup-databases but I was unable to setup management server. It failed with following: (I tried fixing the tomcat user and configs in the initd script but that did not help either)

-- Unit cloudstack-management.service has begun starting up.
May 13 11:15:09 antrix.scaleninja.com cloudstack-management[19721]: * cloudstack-management is not installed
May 13 11:15:09 antrix.scaleninja.com systemd[1]: cloudstack-management.service: Control process exited, code=exited status=1
May 13 11:15:09 antrix.scaleninja.com systemd[1]: Failed to start LSB: Start Tomcat (CloudStack)..
-- Subject: Unit cloudstack-management.service has failed
-- Defined-By: systemd

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit cloudstack-management.service has failed.

-- The result is failed.
May 13 11:15:09 antrix.scaleninja.com systemd[1]: cloudstack-management.service: Unit entered failed state.
May 13 11:15:09 antrix.scaleninja.com systemd[1]: cloudstack-management.service: Failed with result 'exit-code'.

wido · 2016-05-13T08:07:04Z

@rhtyd Forgot to mention, I didn't fix the MGMT server yet. Tomcat is a bit scary to me.

I only worked on the Agent mainly. I think that is the highest prio for most users right now.

yadvr · 2016-05-13T08:16:53Z

@wido alright, but while we're at it let's see if we can fix the mgmt server as well.

I tested agent and was able to perform setup:

root@antrix:/home/bhaisaab/Lab/apache/deb# cloudstack-setup-agent -m localhost -z 1 -p 1 -c 1 -t kvm -g uuid-some -a --pubNic=enp0s31f6 --guestNic=enp0s31f6 --prvNic=enp0s31f6
Starting to configure your system:
Configure Apparmor ... [OK]
Configure Network ... [OK]
Configure Libvirt ... [OK]
Configure Firewall ... [OK]
Configure Nfs ... [OK]
Configure cloudAgent ... [OK]
CloudStack Agent setup is done!
root@antrix:/var/log/cloudstack/agent# cat agent.log 2016-05-13 13:44:54,336 INFO [cloud.agent.AgentShell] (main:null) (logid:) Agent started
2016-05-13 13:44:54,337 INFO [cloud.agent.AgentShell] (main:null) (logid:) Implementation Version is 4.9.0-SNAPSHOT
2016-05-13 13:44:54,339 INFO [cloud.agent.AgentShell] (main:null) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
2016-05-13 13:44:54,341 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to using properties file for storage
2016-05-13 13:44:54,343 INFO [cloud.agent.AgentShell] (main:null) (logid:) Defaulting to the constant time backoff algorithm
root@antrix:/var/log/cloudstack/agent# systemctl status cloudstack-agent
● cloudstack-agent.service - CloudStack Agent
Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2016-05-13 13:46:43 IST; 1s ago

yadvr · 2016-05-13T08:18:27Z

Tested installation and started cloudstack-usage that seems to be working as well:

root@antrix:/var/log/cloudstack/agent# systemctl status cloudstack-usage
● cloudstack-usage.service - CloudStack Usage Server
Loaded: loaded (/lib/systemd/system/cloudstack-usage.service; disabled; vendor preset: enabled)
Active: active (running) since Fri 2016-05-13 13:47:14 IST; 969ms ago
$ cat usage.log
...
DEBUG [cloud.usage.UsageManagerImpl] (main:null) (logid:) Checking to see if usage.vmops.pid exists.
INFO [cloud.usage.UsageManagerImpl] (main:null) (logid:) Implementation Version is 4.9.0-SNAPSHOT
INFO [cloud.usage.UsageServer] (main:null) (logid:) UsageServer ready...

yadvr · 2016-05-13T08:25:28Z

 /etc/profile.d/cloudstack-agent-profile.sh
 /etc/logrotate.d/cloudstack-agent
-/etc/init.d/cloudstack-agent
 /usr/bin/cloudstack-setup-agent


The built deb package still included and install cloudstack-agent initd script at /etc/init.d/cloudstack-agent and same thing happened with cloudstack-usage script. For running them with systemd, are they required? Removing them from *.install file still included them in the package.
dpkg -c
... drwxr-xr-x root/root 0 2016-05-13 10:00 ./etc/cloudstack/ drwxr-xr-x root/root 0 2016-05-13 10:00 ./etc/cloudstack/agent/ -rwxr-xr-x root/root 6529 2016-05-13 10:00 ./etc/cloudstack/agent/agent.properties -rwxr-xr-x root/root 906 2016-05-13 10:00 ./etc/cloudstack/agent/environment.properties -rwxr-xr-x root/root 3543 2016-05-13 10:00 ./etc/cloudstack/agent/log4j-cloud.xml drwxr-xr-x root/root 0 2016-05-13 10:00 ./etc/init.d/ -rwxr-xr-x root/root 4993 2016-05-13 10:00 ./etc/init.d/cloudstack-agent

Yes, that is true. I 'stole' this from other packages which still do the same.
In debian/rules I would have to make the decision to install these or not, but I wasn't really sure how to do so.

You can keep the init.d file for everybody. If the the system is running systemd, a service with the same basename will override the init.d file.

@wido can we also move the debian directory to packaging folder? The deb files are build at .. which is the parent folder of the cloudstack repo/source code, and causes issues when doing concurrent builds using Jenkins.

Maybe, never tried that before. Might be possible. But shall we do that in a different PR? Merge this one and then fix that.

wido · 2016-05-13T10:55:15Z

ok @rhtyd ! Let's look at the Agent and Usage first. When they work as they should I'll look at the MGMT server.

The packages work fine on my Ubuntu 14.04 and 16.04 systems.

One thing, the systemd service is not enabled by default. Should we do that?

yadvr · 2016-05-13T11:11:33Z

@wido I'm LGTM on agent and usage package, it's a good first steps. Though, I also want to see mgmt server as well. For enabling the services, we can have some sort of systemctl enable in the post installation scripts; or in cloudstack-setup-{agent,management} etc.

wido · 2016-06-01T12:36:55Z

I just added a README.md to the packaging directory which adds some documentation on how to package CloudStack for DEB and RPM.

I would like to get this done rather soon so that for 4.9 or 4.10 we have Ubuntu 16.04 packages with systemd.

bvbharatk · 2016-06-20T00:29:02Z

ACS CI BVT Run

Sumarry:
Build Number 199
Hypervisor xenserver
NetworkType Advanced
Passed=68
Failed=5
Skipped=3

Link to logs Folder (search by build_no):https://www.dropbox.com/sh/yj3wnzbceo9uef2/AAB6u-Iap-xztdm6jHX9SjPja?dl=0

Failed tests:

test_vpc_vpn.py
- ContextSuite context=TestRVPCSite2SiteVpn>:setup Failing since 22 runs
- ContextSuite context=TestVpcRemoteAccessVpn>:setup Failing since 22 runs
- ContextSuite context=TestVpcSite2SiteVpn>:setup Failing since 21 runs
test_volumes.py
- test_06_download_detached_volume Failed
test_vm_life_cycle.py
- test_10_attachAndDetach_iso Failed

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_static_role_account_acls
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_routers.py
test_reset_vm_on_reboot.py
test_snapshots.py
test_deploy_vms_with_varied_deploymentplanners.py
test_login.py
test_list_ids_parameter.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_disk_offerings.py

blueorangutan · 2016-08-05T20:24:08Z

Packaging result: ✔centos6 ✔centos7 ✖debian repo: http://packages.shapeblue.com/cloudstack/pr/1541

vincentbernat · 2016-08-08T07:13:55Z

 Maintainer: Wido den Hollander <wido@widodh.nl>
 Build-Depends: debhelper (>= 9), openjdk-8-jdk | openjdk-7-jdk, genisoimage,
- python-mysql.connector, maven (>= 3) | maven3, python (>= 2.7)
+ python-mysql.connector, maven (>= 3) | maven3, python (>= 2.7), lsb-release


For systemd integration, you need to Build-Depends on dh-systemd. Unfortunately, this will make the package unbuildable on Precise. See below.

@vincentbernat that should be okay, we can use Xenial to build pkgs for both Xenial and Precise. At some point we'll need to stop building packages for Precise.

yadvr · 2016-08-08T07:18:33Z

@blueorangutan package

blueorangutan · 2016-08-08T07:19:02Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2016-08-08T07:43:03Z

Packaging result: ✔centos6 ✔centos7 ✔debian repo: http://packages.shapeblue.com/cloudstack/pr/1541
Job ID #67

wido · 2016-08-08T10:41:22Z

Thank you @vincentbernat, I applied your fixes. Can you take a look?

vincentbernat · 2016-08-08T10:49:12Z

The only change I have noticed is the change in postinst. This could bypass the local administrator policy. It would not restart the agent on upgrade either. I would suggest to :

Build-Depends on dh-systemd
Add --with systemd, python2 to dh invocation
Remove the postinst snippet

If you don't want to add a dependency to dh-systemd, the override in debian/rules is useless and the postinst snippet should be more complex. Check for example /var/lib/dpkg/info/openssh-server.postinst for the sections dh_systemd_enable. Note that there are other snippets in postrm.

wido · 2016-08-08T10:50:19Z

@vincentbernat Thanks! Good pointers. I will look into those.

This allows users to easily override variables passed to Java when starting up. It also creates a foundation for sharing the systemd service profile between CentOS and Ubuntu since it only requires the environment file to be changed.

The libvirt and Qemu versions (1.2.2 and 2.0) in Ubuntu 12.04 are to old to support for CloudStack going forward. Ubuntu 14.04 and 16.04 provide the support required to provide new features and more stability.

wido · 2016-08-10T07:15:08Z

I pushed a new version of the commits, could @rhtyd and @vincentbernat take a look?

Yes, Ubuntu 12.04 is dropped, but I did that earlier. See my commits. It will go EOL in April next year, but libvirt and Qemu are already far to old to use any way.

vincentbernat · 2016-08-10T07:43:31Z

 	dh_installinit -pcloudstack-management -pcloudstack-agent -pcloudstack-usage --onlyscripts --no-start

+override_dh_systemd_enable:
+	dh_systemd_enable -pcloudstack-agent -pcloudstack-usage --with systemd


I didn't mean the --with systemd to be here, just on the dh invocation in the % target (like it now is).

Ah, understood! Fixed and pushed

vincentbernat · 2016-08-10T07:47:11Z

Except the extra --with systemd, LGTM

yadvr · 2016-08-10T10:57:38Z

@wido sure I'll spend sometime testing them as well. Since this is packaging only changes, should we get this on 4.9 for the lts /cc @jburwell @karuturi ?

yadvr · 2016-08-10T10:58:31Z

@blueorangutan package

blueorangutan · 2016-08-10T10:59:02Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2016-08-10T11:20:21Z

Packaging result: ✔centos6 ✔centos7 ✖debian repo: http://packages.shapeblue.com/cloudstack/pr/1541
Job ID-78

yadvr · 2016-08-10T11:30:36Z

Build system did not have dh-systemd pkg installed, rekicked build job.

yadvr · 2016-08-10T12:00:11Z

@wido can you apply the PR on 4.9/master and fix build? I got following failures. It would be great if this PR can opened for 4.9. Thanks.

Error from Jenkins build:
dh_install
dh_install: cloudstack-agent missing files: /etc/init.d/cloudstack-agent
dh_install: cloudstack-usage missing files: /etc/init.d/cloudstack-usage
dh_install: missing files, aborting
debian/rules:15: recipe for target 'binary' failed
make: *** [binary] Error 255
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
Build step 'Execute shell' marked build as failure
Finished: FAILURE

Ubuntu 16.04 differs from Ubuntu 14.04 in a few ways: - systemd instead of sysvinit / upstart - Java 8 support The packaging now detects on which distribution it is being build and based on that it installs different files in the packages, but it also changes the Dependencies. Packages for Ubuntu 16.04 will require Java 8 as a JRE

wido · 2016-08-10T15:53:11Z

Done @rhtyd , it should build now

yadvr · 2016-08-10T19:28:34Z

@blueorangutan package

blueorangutan · 2016-08-10T19:29:03Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2016-08-10T19:49:31Z

Packaging result: ✔centos6 ✔centos7 ✔debian repo: http://packages.shapeblue.com/cloudstack/pr/1541
Job ID-79

yadvr · 2016-08-18T10:04:02Z

@wido can you reopen or edit the PR for base branch 4.9? Let's get this merged, as we'll need to support Ubuntu 16.04 for our LTS release.

wido · 2016-08-18T17:40:28Z

@rhtyd That's a total merge hell with all the version number conflicts in the pom.xml files.

This PR has been open since May this year, far before 4.9 was even cut.

Can't we merge it into master and cherry-pick back to 4.9?

yadvr · 2016-08-19T08:35:39Z

@wido if this helps you I can port your PR to 4.9 and push on asf remote so we can work on it together? Cherry-picking to 4.9 would against cause conflicts, and nice we've a fwd-merging based workflow, it would mess up the git-history/graph the next time someone fwd-merged 4.9 to master after this is cherry-picked to 4.9.

yadvr · 2016-08-19T09:11:28Z

@wido I've used your PR to create this new branch on asf remote (so we both and anyone else can collaborate) and this PR: #1647 -- let's close this PR and continue on the new one against 4.9 branch?

@wido

[lts] CLOUDSTACK-9462: Systemd support for Ubuntu 16.04Created this based on @wido 's origin PR #1541 . Requesting for review and testing -- @jburwell @karuturi @wido @vincentbernat @wido I think this change only brings systemd support to agent and usage packages, or does cloudstack-management pkg has systemd support too? @blueorangutan package - systemd: Add a /etc/sysconfig/cloudstack-* file This allows users to easily override variables passed to Java when starting up. It also creates a foundation for sharing the systemd service profile between CentOS and Ubuntu since it only requires the environment file to be changed. - deb: Add Ubuntu 16.04 support Ubuntu 16.04 differs from Ubuntu 14.04 in a few ways: - systemd instead of sysvinit / upstart - Java 8 support The packaging now detects on which distribution it is being build and based on that it installs different files in the packages, but it also changes the Dependencies. * pr/1647: CLOUDSTACK-9462: Refactor systemd scripts CLOUDSTACK-9462: Systemd support for Ubuntu 16.04 Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr reviewed May 13, 2016
View reviewed changes

wido force-pushed the systemd-ubuntu1604 branch from be2cd06 to b4125d3Compare June 1, 2016 12:35

wido force-pushed the systemd-ubuntu1604 branch from b4125d3 to 7b4c9caCompare July 20, 2016 09:55

vincentbernat reviewed Aug 8, 2016
View reviewed changes

wido force-pushed the systemd-ubuntu1604 branch from 7b4c9ca to a05c67aCompare August 8, 2016 10:31

wido added 2 commits August 10, 2016 09:10

systemd: Add a /etc/sysconfig/cloudstack-* file
dd14b74
This allows users to easily override variables passed to Java when starting up. It also creates a foundation for sharing the systemd service profile between CentOS and Ubuntu since it only requires the environment file to be changed.

deb: Drop Ubuntu 12.04 support
352f411
The libvirt and Qemu versions (1.2.2 and 2.0) in Ubuntu 12.04 are to old to support for CloudStack going forward. Ubuntu 14.04 and 16.04 provide the support required to provide new features and more stability.

wido force-pushed the systemd-ubuntu1604 branch from a05c67a to a535ff9Compare August 10, 2016 07:13

vincentbernat reviewed Aug 10, 2016
View reviewed changes

wido force-pushed the systemd-ubuntu1604 branch from a535ff9 to be7dae8Compare August 10, 2016 07:50

wido force-pushed the systemd-ubuntu1604 branch from be7dae8 to 9334ce2Compare August 10, 2016 15:53

yadvr mentioned this pull request Aug 19, 2016
[lts] CLOUDSTACK-9462: Systemd support for Ubuntu 16.04 #1647
Merged

wido closed this Sep 7, 2016

Uh oh!

Conversation

wido commented May 12, 2016

Uh oh!

wido commented May 12, 2016

Uh oh!

yadvr commented May 13, 2016

Uh oh!

yadvr commented May 13, 2016

-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit cloudstack-management.service has failed.

Uh oh!

wido commented May 13, 2016

Uh oh!

yadvr commented May 13, 2016

Uh oh!

yadvr commented May 13, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wido commented May 13, 2016

Uh oh!

yadvr commented May 13, 2016

Uh oh!

wido commented Jun 1, 2016

Uh oh!

bvbharatk commented Jun 20, 2016

ACS CI BVT Run

Uh oh!

blueorangutan commented Aug 5, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yadvr commented Aug 8, 2016

Uh oh!

blueorangutan commented Aug 8, 2016

Uh oh!

blueorangutan commented Aug 8, 2016

Uh oh!

wido commented Aug 8, 2016

Uh oh!

vincentbernat commented Aug 8, 2016• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wido commented Aug 8, 2016

Uh oh!

wido commented Aug 10, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vincentbernat commented Aug 10, 2016

Uh oh!

yadvr commented Aug 10, 2016

Uh oh!

yadvr commented Aug 10, 2016

Uh oh!

blueorangutan commented Aug 10, 2016

Uh oh!

blueorangutan commented Aug 10, 2016

Uh oh!

yadvr commented Aug 10, 2016

Uh oh!

yadvr commented Aug 10, 2016

Uh oh!

wido commented Aug 10, 2016

Uh oh!

yadvr commented Aug 10, 2016

Uh oh!

vincentbernat commented Aug 8, 2016•
edited
Loading