Skip to content

template: Ensuring template is cross zone if type changed to system - #4522

Merged
DaanHoogland merged 1 commit into
apache:masterfrom
shapeblue:fix-template-type
Dec 14, 2020
Merged

template: Ensuring template is cross zone if type changed to system#4522
DaanHoogland merged 1 commit into
apache:masterfrom
shapeblue:fix-template-type

Conversation

@davidjumani

@davidjumanidavidjumani commented Dec 8, 2020

Copy link
Copy Markdown
Contributor

Description

This PR enhances #3945 by ensuring that all to-be System templates are cross zone

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

Comment threadserver/src/main/java/com/cloud/template/TemplateManagerImpl.java Outdated
@yadvryadvr added this to the 4.15.0.0 milestone Dec 8, 2020
@yadvr

yadvr commented Dec 8, 2020

Copy link
Copy Markdown
Member

cc @ravening@weizhouapache pl review
@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@ravening

Copy link
Copy Markdown
Member

cc @ravening@weizhouapache pl review
@blueorangutan package

@rhtyd will review this when its ready

@blueorangutan

Copy link
Copy Markdown

@ravening a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@davidjumani

Copy link
Copy Markdown
ContributorAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2468

@davidjumani

Copy link
Copy Markdown
ContributorAuthor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

s_logger.info("Copying template to all zones since it will be a system template");
List<DataCenterVO> zones = _dcDao.listEnabledZones();
if (zones != null) {
List<Long> zoneIds = zones.stream().map(zone -> zone.getId()).collect(Collectors.toList());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we skip the zone on which the template is already in?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this seems to be handled by

DataStoredstSecStore = getImageStore(destZoneId, templateId);
if (dstSecStore != null) {
s_logger.debug("There is template " + templateId + " in secondary storage " + dstSecStore.getName() +
" in zone " + destZoneId + " , don't need to copy");
continue;
}

Comment threadserver/src/main/java/com/cloud/template/TemplateManagerImpl.java Outdated
Comment threadserver/src/main/java/com/cloud/template/TemplateManagerImpl.java Outdated
Comment threadserver/src/main/java/com/cloud/template/TemplateManagerImpl.java Outdated
@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-3321)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32648 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4522-t3321-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Smoke tests completed. 86 look OK, 0 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File

@davidjumanidavidjumani changed the title template: Copying templates to all zones if type changed to systemtemplate: Ensuring template is cross zone if type changed to systemDec 9, 2020
@davidjumani

Copy link
Copy Markdown
ContributorAuthor

Updated the PR to just add a cross zone check when trying to change a template to a System / Builtin template

@davidjumani
davidjumaniforce-pushed the fix-template-type branch 3 times, most recently from 022a476 to ab2f68cCompareDecember 9, 2020 06:31
@davidjumani

Copy link
Copy Markdown
ContributorAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2470

@davidjumani

Copy link
Copy Markdown
ContributorAuthor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-3323)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 70571 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4522-t3323-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_storage_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 23 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File
test_03_deploy_and_upgrade_kubernetes_clusterFailure249.40test_kubernetes_clusters.py
ContextSuite context=TestResetVmOnReboot>:setupError0.00test_reset_vm_on_reboot.py
ContextSuite context=TestRouterDHCPHosts>:setupError0.00test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setupError0.00test_router_dhcphosts.py
ContextSuite context=TestRAMCPUResourceAccounting>:setupError0.00test_resource_accounting.py
ContextSuite context=TestRouterDns>:setupError0.00test_router_dns.py
ContextSuite context=TestRouterDnsService>:setupError0.00test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setupError0.00test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setupError0.00test_routers_iptables_default_policy.py
ContextSuite context=TestIsolatedNetworks>:setupError0.00test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setupError0.00test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setupError0.00test_routers.py
test_01_sys_vm_startFailure0.14test_secondary_storage.py
ContextSuite context=TestCpuCapServiceOfferings>:setupError0.00test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setupError0.19test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setupError0.00test_snapshots.py
test_01_list_sec_storage_vmFailure0.06test_ssvm.py
test_02_list_cpvm_vmFailure0.05test_ssvm.py
test_03_ssvm_internalsFailure0.05test_ssvm.py
test_04_cpvm_internalsFailure0.05test_ssvm.py
test_05_stop_ssvmFailure0.05test_ssvm.py
test_06_stop_cpvmFailure0.05test_ssvm.py
test_07_reboot_ssvmFailure0.05test_ssvm.py
test_08_reboot_cpvmFailure0.05test_ssvm.py
test_09_destroy_ssvmFailure0.05test_ssvm.py
test_10_destroy_cpvmFailure0.05test_ssvm.py
ContextSuite context=TestVMWareStoragePolicies>:setupError0.00test_storage_policy.py
test_02_create_template_with_checksum_sha1Error65.55test_templates.py
test_03_create_template_with_checksum_sha256Error65.55test_templates.py
test_04_create_template_with_checksum_md5Error65.55test_templates.py
test_05_create_template_with_no_checksumError65.54test_templates.py
test_02_deploy_vm_from_direct_download_templateError1.30test_templates.py
test_03_deploy_vm_wrong_checksumError1.34test_templates.py
ContextSuite context=TestTemplates>:setupError17.41test_templates.py
ContextSuite context=TestISOUsage>:setupError0.00test_usage.py
ContextSuite context=TestLBRuleUsage>:setupError0.00test_usage.py
ContextSuite context=TestNatRuleUsage>:setupError0.00test_usage.py
ContextSuite context=TestPublicIPUsage>:setupError0.00test_usage.py
ContextSuite context=TestSnapshotUsage>:setupError0.00test_usage.py
ContextSuite context=TestVmUsage>:setupError0.00test_usage.py
ContextSuite context=TestVolumeUsage>:setupError0.00test_usage.py
ContextSuite context=TestVpnUsage>:setupError0.00test_usage.py
ContextSuite context=Test01DeployVM>:setupError0.00test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setupError0.00test_vm_life_cycle.py
test_14_secure_to_secure_vm_migrationError11.41test_vm_life_cycle.py
test_15_secured_to_nonsecured_vm_migrationError74.06test_vm_life_cycle.py
test_16_nonsecured_to_secured_vm_migrationError1.29test_vm_life_cycle.py
ContextSuite context=TestVmSnapshot>:setupError1.87test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setupError0.00test_volumes.py
ContextSuite context=TestVolumes>:setupError0.00test_volumes.py
ContextSuite context=TestVPCRedundancy>:setupError0.00test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setupError0.00test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setupError0.00test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setupError0.00test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setupError0.00test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setupError0.00test_vpc_vpn.py
test_disable_oobm_ha_state_ineligibleError1514.91test_hostha_kvm.py

@yadvryadvr modified the milestones: 4.15.0.0, 4.15.1.0Dec 11, 2020
@davidjumani
davidjumani marked this pull request as ready for review December 11, 2020 10:26
@yadvr

Copy link
Copy Markdown
Member

@davidjumani can you kick pkg/test again?

@davidjumani

Copy link
Copy Markdown
ContributorAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2479

@davidjumani

Copy link
Copy Markdown
ContributorAuthor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@davidjumani a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-3336)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34583 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4522-t3336-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 85 look OK, 1 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File
test_04_rvpc_privategw_static_routesFailure863.31test_privategw_acl.py

Comment threadserver/src/main/java/com/cloud/template/TemplateManagerImpl.java Outdated
Comment threadserver/src/main/java/com/cloud/template/TemplateManagerImpl.java Outdated
@yadvr

Copy link
Copy Markdown
Member

@davidjumani can you revert the two lines I've comments on, move the conditional up if that makes sense? Let's keep the changes minimal if possible, easier to review and test. Thanks.

@davidjumani

Copy link
Copy Markdown
ContributorAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@davidjumani a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2490

@yadvr

Copy link
Copy Markdown
Member

@weizhouapache@DaanHoogland should we consider this for RC2, with current changes it addresses concerns from #3945
@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@weizhouapache

Copy link
Copy Markdown
Member

@weizhouapache@DaanHoogland should we consider this for RC2, with current changes it addresses concerns from #3945
@blueorangutan test

@rhtyd I am ok to merge this into 4.15.0.0

@DaanHooglandDaanHoogland modified the milestones: 4.15.1.0, 4.15.0.0Dec 14, 2020
@DaanHoogland
DaanHoogland merged commit 2f78b88 into apache:masterDec 14, 2020
@DaanHoogland
DaanHoogland deleted the fix-template-type branch December 14, 2020 08:59
@blueorangutan

Copy link
Copy Markdown

Trillian test result (tid-3347)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 72524 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4522-t3347-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_migration.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_secondary_storage.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Intermittent failure detected: /marvin/tests/smoke/test_storage_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_templates.py
Intermittent failure detected: /marvin/tests/smoke/test_usage.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermittent failure detected: /marvin/tests/smoke/test_vm_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_volumes.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 23 have error(s)
Only failed tests results shown below:

TestResultTime (s)Test File
test_04_rvpc_privategw_static_routesFailure239.75test_privategw_acl.py
ContextSuite context=TestResetVmOnReboot>:setupError0.00test_reset_vm_on_reboot.py
ContextSuite context=TestRouterDHCPHosts>:setupError0.00test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setupError0.00test_router_dhcphosts.py
ContextSuite context=TestRAMCPUResourceAccounting>:setupError0.00test_resource_accounting.py
ContextSuite context=TestRouterDns>:setupError0.00test_router_dns.py
ContextSuite context=TestRouterDnsService>:setupError0.00test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setupError0.00test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setupError0.00test_routers_iptables_default_policy.py
ContextSuite context=TestIsolatedNetworks>:setupError0.00test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setupError0.00test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setupError0.00test_routers.py
test_01_sys_vm_startFailure0.12test_secondary_storage.py
ContextSuite context=TestCpuCapServiceOfferings>:setupError0.00test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setupError0.18test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setupError0.00test_snapshots.py
test_01_list_sec_storage_vmFailure0.05test_ssvm.py
test_02_list_cpvm_vmFailure0.04test_ssvm.py
test_03_ssvm_internalsFailure0.04test_ssvm.py
test_04_cpvm_internalsFailure0.04test_ssvm.py
test_05_stop_ssvmFailure0.04test_ssvm.py
test_06_stop_cpvmFailure0.04test_ssvm.py
test_07_reboot_ssvmFailure0.04test_ssvm.py
test_08_reboot_cpvmFailure0.05test_ssvm.py
test_09_destroy_ssvmFailure0.04test_ssvm.py
test_10_destroy_cpvmFailure0.04test_ssvm.py
ContextSuite context=TestVMWareStoragePolicies>:setupError0.00test_storage_policy.py
test_02_create_template_with_checksum_sha1Error65.53test_templates.py
test_03_create_template_with_checksum_sha256Error65.56test_templates.py
test_04_create_template_with_checksum_md5Error65.52test_templates.py
test_05_create_template_with_no_checksumError65.54test_templates.py
test_02_deploy_vm_from_direct_download_templateError1.28test_templates.py
test_03_deploy_vm_wrong_checksumError1.35test_templates.py
ContextSuite context=TestTemplates>:setupError17.43test_templates.py
ContextSuite context=TestISOUsage>:setupError0.00test_usage.py
ContextSuite context=TestLBRuleUsage>:setupError0.00test_usage.py
ContextSuite context=TestNatRuleUsage>:setupError0.00test_usage.py
ContextSuite context=TestPublicIPUsage>:setupError0.00test_usage.py
ContextSuite context=TestSnapshotUsage>:setupError0.00test_usage.py
ContextSuite context=TestVmUsage>:setupError0.00test_usage.py
ContextSuite context=TestVolumeUsage>:setupError0.00test_usage.py
ContextSuite context=TestVpnUsage>:setupError0.00test_usage.py
ContextSuite context=Test01DeployVM>:setupError0.00test_vm_life_cycle.py
ContextSuite context=Test02VMLifeCycle>:setupError0.00test_vm_life_cycle.py
test_14_secure_to_secure_vm_migrationError11.43test_vm_life_cycle.py
test_15_secured_to_nonsecured_vm_migrationError74.10test_vm_life_cycle.py
test_16_nonsecured_to_secured_vm_migrationError1.27test_vm_life_cycle.py
ContextSuite context=TestVmSnapshot>:setupError1.96test_vm_snapshots.py
ContextSuite context=TestCreateVolume>:setupError0.00test_volumes.py
ContextSuite context=TestVolumes>:setupError0.00test_volumes.py
ContextSuite context=TestVPCRedundancy>:setupError0.00test_vpc_redundant.py
ContextSuite context=TestVPCNics>:setupError0.00test_vpc_router_nics.py
ContextSuite context=TestRVPCSite2SiteVpn>:setupError0.00test_vpc_vpn.py
ContextSuite context=TestVPCSite2SiteVPNMultipleOptions>:setupError0.00test_vpc_vpn.py
ContextSuite context=TestVpcRemoteAccessVpn>:setupError0.00test_vpc_vpn.py
ContextSuite context=TestVpcSite2SiteVpn>:setupError0.00test_vpc_vpn.py
test_disable_oobm_ha_state_ineligibleError1514.47test_hostha_kvm.py

qrry added a commit to qrry/cloudstack that referenced this pull request Dec 23, 2020
* master:
server: add conditions for custom offerings (apache#4540)
vr: Ensuring dnsmasq.leases file is populated (apache#4529)
template: Ensuring template is cross zone if type changed to system (apache#4522)
storage: Fix hypervisor type cast to string (apache#4516)
db upgrade: fix sql exception: Access denied; you need (at least one of) the SUPER privilege(s) for this operation (apache#4533)
CLOUDSTACK-10423:Potential sensitive information disclosure (apache#4536)
jobs: The patch remove the password from resultObject and make it be humanreadable (apache#4538)
listphysicalnetworks: Honouring keyword parameter (apache#4511)
Fix NPE when Volume exists on secondary store but doesn't have a download URL (apache#4530)
apidoc issue (apache#4532)
db: Fix description of volume.stats.interval which is in milliseconds not seconds (apache#4526)
kvm: set cpu topology only if cpucore per socket is positive value (apache#4527)
xenserver: check and eject patch vbd for systemvms (apache#4525)
Fix warning when setup cloudstack-common (apache#4523)
kvm: FIX cpucorespersocket is not working on KVM (apache#4497)
change debug to warn for unknown exceptions (apache#4521)
Fix failure in validating IP address in case of multiple Management Servers (apache#4507)
Update log output for FirstFitPlanner (apache#4515)
ui: deprecate old UI and move to legacy to be served at /client/legacy (apache#4518)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants

@davidjumani@yadvr@blueorangutan@ravening@weizhouapache@DaanHoogland@PaulAngus