KVM: Prevent regenerating keystore on provisionCertificate API by nvazquez · Pull Request #3075 · apache/cloudstack

nvazquez · 2018-12-02T21:59:21Z

Description

It was found out that direct download feature stops working after 'provisionCertificate' API is executed, as both features import certificates into the same /etc/cloudstack/agent/cloud.jks keystore.
The 'provisionCertificate' API invokes a script which regenerates the keystore before adding the new certificate, removing the existing certificates from the keystore. This fix prevents regenerating the keystore before adding the new certificates.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

yadvr · 2018-12-03T04:00:50Z

Can we reuse keystore table for this maybe?

borisstoyanov · 2019-02-20T14:30:11Z

@blueorangutan package

blueorangutan · 2019-02-20T14:31:05Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-02-20T14:51:34Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2610

borisstoyanov · 2019-02-21T07:24:57Z

@blueorangutan test

borisstoyanov · 2019-02-25T11:45:30Z

@blueorangutan test

blueorangutan · 2019-02-25T11:46:27Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-02-25T22:33:46Z

Trillian test result (tid-3409)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37252 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3075-t3409-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Smoke tests completed. 64 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_extract_Iso	`Failure`	1.05	test_iso.py
test_05_stop_ssvm	`Error`	950.85	test_ssvm.py
test_06_stop_cpvm	`Error`	957.93	test_ssvm.py
test_07_reboot_ssvm	`Error`	911.62	test_ssvm.py
test_08_reboot_cpvm	`Error`	912.01	test_ssvm.py
test_04_extract_template	`Failure`	1.06	test_templates.py
test_06_download_detached_volume	`Failure`	10.27	test_volumes.py

borisstoyanov

@nvazquez there seems to be something wrong with the SSVM tests here, can you look when you have time.

nvazquez · 2019-02-26T12:07:28Z

Sure @borisstoyanov let me take a look later today

nvazquez · 2019-02-26T18:50:30Z

@blueorangutan package

blueorangutan · 2019-02-26T18:51:33Z

@nvazquez a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-02-26T19:11:49Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2614

nvazquez · 2019-02-26T20:09:11Z

@borisstoyanov I have rebased the 4.11 branch and tested those files locally without errors on failing tests (results below). It seems to be errors with checksums tests.

test_ssvm.py:

Test List secondary storage VMs ... === TestName: test_01_list_sec_storage_vm | Status : SUCCESS ===
ok
Test List console proxy VMs ... === TestName: test_02_list_cpvm_vm | Status : SUCCESS ===
ok
Test SSVM Internals ... === TestName: test_03_ssvm_internals | Status : SUCCESS ===
ok
Test CPVM Internals ... === TestName: test_04_cpvm_internals | Status : SUCCESS ===
ok
Test stop SSVM ... === TestName: test_05_stop_ssvm | Status : SUCCESS ===
ok
Test stop CPVM ... === TestName: test_06_stop_cpvm | Status : SUCCESS ===
ok
Test reboot SSVM ... === TestName: test_07_reboot_ssvm | Status : SUCCESS ===
ok
Test reboot CPVM ... === TestName: test_08_reboot_cpvm | Status : SUCCESS ===
ok
Test destroy SSVM ... SKIP: Marvin configuration has no host credentials to check router services
Test destroy CPVM ... === TestName: test_10_destroy_cpvm | Status : SUCCESS ===
ok
Test NFS Version on Secondary Storage mounted properly on SSVM ... SKIP: No NFS version provided in test data
----------------------------------------------------------------------
Ran 11 tests in 407.304s
OK (SKIP=2)

test_iso.py, test_templates.py, test_volumes.py:

test_01_1_create_iso_with_checksum_sha1_negative (integration.smoke.test_iso.TestCreateISOWithChecksum) ... === TestName: test_01_1_create_iso_with_checksum_sha1_negative | Status : SUCCESS ===
ok
test_01_create_iso_with_checksum_sha1 (integration.smoke.test_iso.TestCreateISOWithChecksum) ... === TestName: test_01_create_iso_with_checksum_sha1 | Status : SUCCESS ===
ok
test_02_1_create_iso_with_checksum_sha256_negative (integration.smoke.test_iso.TestCreateISOWithChecksum) ... === TestName: test_02_1_create_iso_with_checksum_sha256_negative | Status : SUCCESS ===
ok
test_02_create_iso_with_checksum_sha256 (integration.smoke.test_iso.TestCreateISOWithChecksum) ... === TestName: test_02_create_iso_with_checksum_sha256 | Status : EXCEPTION ===
ERROR
=== TestName: test_02_create_iso_with_checksum_sha256 | Status : EXCEPTION ===
ERROR
test_03_1_create_iso_with_checksum_md5_negative (integration.smoke.test_iso.TestCreateISOWithChecksum) ... === TestName: test_03_1_create_iso_with_checksum_md5_negative | Status : EXCEPTION ===
ERROR
test_03_create_iso_with_checksum_md5 (integration.smoke.test_iso.TestCreateISOWithChecksum) ... === TestName: test_03_create_iso_with_checksum_md5 | Status : SUCCESS ===
ok
test_04_create_iso_with_no_checksum (integration.smoke.test_iso.TestCreateISOWithChecksum) ... === TestName: test_04_create_iso_with_no_checksum | Status : SUCCESS ===
ok
Test create public & private ISO ... === TestName: test_01_create_iso | Status : SUCCESS ===
ok
Test Edit ISO ... === TestName: test_02_edit_iso | Status : SUCCESS ===
ok
Test delete ISO ... === TestName: test_03_delete_iso | Status : SUCCESS ===
ok
Test for extract ISO ... === TestName: test_04_extract_Iso | Status : SUCCESS ===
ok
Update & Test for ISO permissions ... === TestName: test_05_iso_permissions | Status : SUCCESS ===
ok
Test for copy ISO from one zone to another ... SKIP: Not enough zones available to perform copy template
Test delete ISO ... === TestName: test_07_list_default_iso | Status : SUCCESS ===
ok
test_09_copy_delete_template (integration.smoke.test_templates.TestCopyDeleteTemplate) ... SKIP: Skipping test due to there are less than two zones.
Test create public & private template ... === TestName: test_01_create_template | Status : SUCCESS ===
ok
Test when createTemplate is used to create templates having the same name all of them get ... === TestName: test_CreateTemplateWithDuplicateName | Status : SUCCESS ===
ok
test_02_1_create_template_with_checksum_sha1_negative (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_02_1_create_template_with_checksum_sha1_negative | Status : SUCCESS ===
ok
test_02_create_template_with_checksum_sha1 (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_02_create_template_with_checksum_sha1 | Status : EXCEPTION ===
ERROR
test_03_1_create_template_with_checksum_sha256_negative (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_03_1_create_template_with_checksum_sha256_negative | Status : SUCCESS ===
ok
test_03_create_template_with_checksum_sha256 (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_03_create_template_with_checksum_sha256 | Status : SUCCESS ===
ok
test_04_1_create_template_with_checksum_md5_negative (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_04_1_create_template_with_checksum_md5_negative | Status : SUCCESS ===
ok
test_04_create_template_with_checksum_md5 (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_04_create_template_with_checksum_md5 | Status : SUCCESS ===
ok
test_05_create_template_with_no_checksum (integration.smoke.test_templates.TestCreateTemplateWithChecksum) ... === TestName: test_05_create_template_with_no_checksum | Status : SUCCESS ===
ok
Register a template using Direct Download flag ... === TestName: test_01_register_template_direct_download_flag | Status : SUCCESS ===
ok
Deploy a VM from a Direct Download registered template ... === TestName: test_02_deploy_vm_from_direct_download_template | Status : SUCCESS ===
ok
Deploy a VM from a Direct Download registered template with wrong checksum ... === TestName: test_03_deploy_vm_wrong_checksum | Status : SUCCESS ===
ok
Test Edit template ... === TestName: test_02_edit_template | Status : SUCCESS ===
ok
Test delete template ... === TestName: test_03_delete_template | Status : SUCCESS ===
ok
Test for extract template ... === TestName: test_04_extract_template | Status : SUCCESS ===
ok
Update & Test for template permissions ... === TestName: test_05_template_permissions | Status : SUCCESS ===
ok
Test for copy template from one zone to another ... SKIP: Not enough zones available to perform copy template
Test only public templates are visible to normal user ... === TestName: test_07_list_public_templates | Status : SUCCESS ===
ok
Test System templates are not visible to normal user ... === TestName: test_08_list_system_templates | Status : SUCCESS ===
ok
Test Volume creation for all Disk Offerings (incl. custom) ... === TestName: test_01_create_volume | Status : SUCCESS ===
ok
Attach a created Volume to a Running VM ... === TestName: test_02_attach_volume | Status : SUCCESS ===
ok
Download a Volume attached to a VM ... === TestName: test_03_download_attached_volume | Status : SUCCESS ===
ok
Delete a Volume attached to a VM ... === TestName: test_04_delete_attached_volume | Status : SUCCESS ===
ok
Detach a Volume attached to a VM ... === TestName: test_05_detach_volume | Status : SUCCESS ===
ok
Download a Volume unattached to an VM ... === TestName: test_06_download_detached_volume | Status : SUCCESS ===
ok
Test resize (negative) non-existent volume ... === TestName: test_07_resize_fail | Status : SUCCESS ===
ok
Test resize a volume ... === TestName: test_08_resize_volume | Status : SUCCESS ===
ok
Delete a Volume unattached to an VM ... === TestName: test_09_delete_detached_volume | Status : SUCCESS ===
ok
test_10_list_volumes (integration.smoke.test_volumes.TestVolumes) ... === TestName: test_10_list_volumes | Status : SUCCESS ===
ok

blueorangutan · 2019-02-26T20:09:33Z

@nvazquez unsupported parameters provided. Supported mgmt server os are: centos6, centos7, ubuntu. Supported hypervisors are: kvm-centos6, kvm-centos7, kvm-ubuntu, xenserver-71, xenserver-65sp1, xenserver-62sp1, vmware-65, vmware-60u2, vmware-55u3, vmware-51u1, vmware-50u1

nvazquez · 2019-02-26T20:12:11Z

@blueorangutan test

blueorangutan · 2019-02-26T20:12:33Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2019-02-27T07:05:38Z

Trillian test result (tid-3410)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37431 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3075-t3410-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 64 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_extract_Iso	`Failure`	1.08	test_iso.py
test_05_stop_ssvm	`Error`	941.19	test_ssvm.py
test_06_stop_cpvm	`Error`	960.49	test_ssvm.py
test_07_reboot_ssvm	`Error`	912.87	test_ssvm.py
test_08_reboot_cpvm	`Error`	912.91	test_ssvm.py
test_05_create_template_with_no_checksum	`Error`	65.50	test_templates.py
test_04_extract_template	`Failure`	1.09	test_templates.py
test_06_download_detached_volume	`Failure`	23.77	test_volumes.py

yadvr · 2019-05-30T06:18:44Z

@blueorangutan package

blueorangutan · 2019-06-04T19:42:22Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2843

nvazquez · 2019-06-04T19:46:50Z

@blueorangutan test

blueorangutan · 2019-06-04T19:47:51Z

@nvazquez a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

nvazquez · 2019-06-04T19:49:20Z

@rhtyd this is ready for review

yadvr · 2019-06-05T03:20:15Z

 fi

-# Use a new keystore file
-NEW_KS_FILE="$KS_FILE.new"


The script was create to be idempotent for jks creation and cert import. I'll have to test concurrent cases and see if this script would pass/fail. I'll test and review today and keep you posted @nvazquez thanks.

I'll also need to test for certificate renewal.

blueorangutan · 2019-06-05T07:50:50Z

Trillian test result (tid-3645)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 41854 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3075-t3645-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 68 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_05_stop_ssvm	`Error`	961.34	test_ssvm.py
test_06_stop_cpvm	`Error`	955.20	test_ssvm.py
test_07_reboot_ssvm	`Error`	911.84	test_ssvm.py
test_08_reboot_cpvm	`Error`	910.82	test_ssvm.py

…ficates

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2019-06-05T10:55:42Z

@nvazquez LGTM, I reviewed and fixed few issues around certificate renewal
@blueorangutan package

blueorangutan · 2019-06-05T10:55:53Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-06-05T11:16:16Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2847

yadvr · 2019-06-05T11:19:30Z

@blueorangutan test

blueorangutan · 2019-06-05T11:19:54Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

DaanHoogland

code changes look good, testing this is done and i'll have to trust that but it is scary ;)

blueorangutan · 2019-06-05T19:28:46Z

Trillian test result (tid-3650)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 27669 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3075-t3650-kvm-centos7.zip
Smoke tests completed. 69 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

nvazquez · 2019-06-05T20:29:21Z

Thanks for the last fix @rhtyd
Merging after test results and reviews

nvazquez requested review from DaanHoogland, rafaelweingartner and yadvr December 2, 2018 21:59

nvazquez added this to the 4.11.3.0 milestone Dec 2, 2018

nvazquez mentioned this pull request Dec 2, 2018
KVM: Improvements on upload direct download certificates #2995
Merged
5 tasks

rafaelweingartner reviewed Dec 12, 2018
View reviewed changes

Comment thread...cloudstack/api/command/admin/direct/download/UploadTemplateDirectDownloadCertificateCmd.java Outdated
Comment threadengine/schema/src/com/cloud/host/dao/HostDaoImpl.java Outdated
Comment threadserver/src/org/apache/cloudstack/direct/download/DirectDownloadManagerImpl.java Outdated

nvazquez changed the title ~~Prevent regenerating keystore and add background task to upload certificates~~WIP: Prevent regenerating keystore and add background task to upload certificatesFeb 25, 2019

borisstoyanov requested changes Feb 26, 2019
View reviewed changes

nvazquez force-pushed the directdownloadkeystoreissue branch from 322ca1a to 35fa34dCompare February 26, 2019 18:29

yadvr added the status:work-in-progress label Mar 25, 2019

apache deleted a comment from blueorangutan May 30, 2019

nvazquez removed the component:templates label Jun 4, 2019

nvazquez requested review from borisstoyanov and rafaelweingartner June 4, 2019 19:50

nvazquez mentioned this pull request Jun 5, 2019
KVM: Enhancements for direct download feature #3374
Merged
5 tasks

yadvr reviewed Jun 5, 2019
View reviewed changes

apache deleted a comment from blueorangutan Jun 5, 2019

yadvr reviewed Jun 5, 2019
View reviewed changes

Comment threadscripts/util/keystore-setup Outdated

nvazquez added 2 commits June 5, 2019 14:45

Prevent regenerating keystore and add background task to upload certi…
b7e99a3
…ficates

Remove background task to upload certificates
07a550c

yadvr force-pushed the directdownloadkeystoreissue branch from 4c9bca6 to 07a550cCompare June 5, 2019 09:15

yadvr added 2 commits June 5, 2019 16:03

keystore-setup: delete old key in jks
e1d8555
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

also don't allow ca-cert alias
7831806
Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr approved these changes Jun 5, 2019
View reviewed changes

yadvr requested a review from PaulAngus June 5, 2019 10:59

DaanHoogland approved these changes Jun 5, 2019
View reviewed changes

nvazquez merged commit d108f32 into apache:4.11Jun 5, 2019

nvazquez deleted the directdownloadkeystoreissue branch April 6, 2020 14:43

Uh oh!

Conversation

nvazquez commented Dec 2, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

yadvr commented Dec 3, 2018

Uh oh!

Uh oh!

Uh oh!

Uh oh!

borisstoyanov commented Feb 20, 2019

Uh oh!

blueorangutan commented Feb 20, 2019

Uh oh!

blueorangutan commented Feb 20, 2019

Uh oh!

borisstoyanov commented Feb 21, 2019

Uh oh!

borisstoyanov commented Feb 25, 2019

Uh oh!

blueorangutan commented Feb 25, 2019

Uh oh!

blueorangutan commented Feb 25, 2019

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

nvazquez commented Feb 26, 2019

Uh oh!

nvazquez commented Feb 26, 2019

Uh oh!

blueorangutan commented Feb 26, 2019

Uh oh!

blueorangutan commented Feb 26, 2019

Uh oh!

nvazquez commented Feb 26, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blueorangutan commented Feb 26, 2019

Uh oh!

nvazquez commented Feb 26, 2019

Uh oh!

blueorangutan commented Feb 26, 2019

Uh oh!

blueorangutan commented Feb 27, 2019

Uh oh!

yadvr commented May 30, 2019

Uh oh!

blueorangutan commented Jun 4, 2019

Uh oh!

nvazquez commented Jun 4, 2019

Uh oh!

blueorangutan commented Jun 4, 2019

Uh oh!

nvazquez commented Jun 4, 2019

Uh oh!

yadvrJun 5, 2019

Choose a reason for hiding this comment

Uh oh!

yadvrJun 5, 2019

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jun 5, 2019

Uh oh!

Uh oh!

yadvr commented Jun 5, 2019

Uh oh!

blueorangutan commented Jun 5, 2019

Uh oh!

blueorangutan commented Jun 5, 2019

Uh oh!

yadvr commented Jun 5, 2019

Uh oh!

blueorangutan commented Jun 5, 2019

Uh oh!

DaanHoogland left a comment

nvazquez commented Dec 2, 2018•
edited
Loading

nvazquez commented Feb 26, 2019•
edited
Loading