Skip to content

[4.20] VR: fix issue if userdata is binary data - #8739

Merged
DaanHoogland merged 6 commits into
apache:mainfrom
weizhouapache:4.20-vr-fix-binary-userdata
Jul 25, 2024
Merged

[4.20] VR: fix issue if userdata is binary data#8739
DaanHoogland merged 6 commits into
apache:mainfrom
weizhouapache:4.20-vr-fix-binary-userdata

Conversation

@weizhouapache

Copy link
Copy Markdown
Member

Description

This PR fixes the issue that VM cannot start with binary userdata. This is a regression of #8497

Steps to reproduce the issue and verify the fix

  1. create a userdata with content below
    cloud-init supports gzip compress userdata. for example
$ echo "Apache CloudStack" |gzip |base64 -w0
H4sIAAAAAAAAA3MsSEzOSFVwzskvTQkuSUzO5gIAxXcOkxIAAAA=
  1. deploy a vm with the userdata, it failed
2024-03-04T11:13:27,400 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:[]) (logid:) Seq 1-8288593639198687341: Processing: { Ans: , MgmtId: 167781234, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.routing.GroupAnswer":{"results":
["null - success: Creating file in VR, with ip: 169.254.52.151, file: vm_password.json.5f0a36da-6fa7-48e4-8aae-
dcc8923fc42d","null - success: "],"result":"true","wait":"0","bypassHostMaintenance":"false"}},
{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - success: Creating file in VR, with ip: 169.254.52.151, file: vm_metadata.json.53c05bda-dc05-49df-b548-f517ae0fa546","null - failed: Traceback (most recent call last): File "/opt/cloud/bin/update_config.py", line 147, in <module> process_file() File "/opt/cloud/bin/update_config.py", line 57, in process_file finish_config() File "/opt/cloud/bin/update_config.py", line 42, in finish_config returncode = configure.main(sys.argv) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/cloud/bin/configure.py", line 1413, in main execDatabag(json_type, databag_map) File "/opt/cloud/bin/configure.py", line 1400, in execDatabag executor.process() File "/opt/cloud/bin/configure.py", line 625, in process self.__createfile(ip, folder, file, data) File "/opt/cloud/bin/configure.py", line 652, in __createfile fh.write(data.decode()) ^^^^^^^^^^^^^UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte"],"result":"false","wait":"0","bypassHostMaintenance":"false"}}] }
  1. apply this PR, vm is started successfully
  2. check the userdata in vm
    image

Types of changes

  • Breaking change (fix or feature that would cause existing functionality to change)
  • New feature (non-breaking change which adds functionality)
  • Bug fix (non-breaking change which fixes an issue)
  • Enhancement (improves an existing feature and functionality)
  • Cleanup (Code refactoring and cleanup, that may add test cases)
  • build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

  • Major
  • Minor

Bug Severity

  • BLOCKER
  • Critical
  • Major
  • Minor
  • Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

@weizhouapacheweizhouapache added this to the 20.0.0 milestone Mar 4, 2024
@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@codecov

codecovBot commented Mar 4, 2024

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 15.53%. Comparing base (cea4801) to head (ea6f47b).
Report is 27 commits behind head on main.

Additional details and impacted files
@@ Coverage Diff @@## main #8739 +/- ##
==========================================
Coverage 15.53% 15.53% Complexity 11967 11967 ==========================================
Files 5492 5492 Lines 480934 480934 Branches 60876 60056 -820 ==========================================
Hits 74711 74711 Misses 397962 397962 Partials 8261 8261 
FlagCoverage Δ
uitests4.17% <ø> (ø)
unittests16.30% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@DaanHooglandDaanHoogland left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 8847

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan test matrix

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Trillian-Jenkins matrix job (centos7 mgmt + xenserver71, rocky8 mgmt + vmware67u3, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-9387)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43267 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8739-t9387-kvm-centos7.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-9385)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44270 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8739-t9385-xenserver-71.zip
Smoke tests completed. 129 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-9386)
Environment: vmware-67u3 (x2), Advanced Networking with Mgmt server r8
Total time taken: 49495 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8739-t9386-vmware-67u3.zip
Smoke tests completed. 128 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_02_balanced_drs_algorithmError425.77test_cluster_drs.py

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9386

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@weizhouapache a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9392

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@blueorangutan test alma9 kvm-alma9

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

This fixes the issue that CAPC cannot create kubernetes cluster on 4.20/main

cc @rohityadavcloud

@github-actions

Copy link
Copy Markdown

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan package

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

@blueorangutan

Copy link
Copy Markdown

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10325

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@DaanHoogland
will you test this pr ? the PR description might help

@DaanHoogland

Copy link
Copy Markdown
Contributor

@DaanHoogland will you test this pr ? the PR description might help

will find time... (as if that would be a promise ;)

@DaanHooglandDaanHoogland self-assigned this Jul 12, 2024
@weizhouapache

Copy link
Copy Markdown
MemberAuthor

@DaanHoogland will you test this pr ? the PR description might help

will find time... (as if that would be a promise ;)

ah ha, I consider it as a promise. thanks buddy @DaanHoogland

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-10807)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 65194 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8739-t10807-kvm-centos7.zip
Smoke tests completed. 113 look OK, 24 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_role_account_acls_multiple_mgmt_serversError2.17test_dynamicroles.py
test_query_async_job_resultError105.94test_async_job.py
test_revoke_certificateError0.01test_certauthority_root.py
test_configure_ha_provider_invalidError0.01test_hostha_simulator.py
test_configure_ha_provider_validError0.01test_hostha_simulator.py
test_ha_configure_enabledisable_across_clusterzonesError0.01test_hostha_simulator.py
test_ha_disable_feature_invalidError0.01test_hostha_simulator.py
test_ha_enable_feature_invalidError0.01test_hostha_simulator.py
test_ha_list_providersError0.01test_hostha_simulator.py
test_ha_multiple_mgmt_server_ownershipError0.01test_hostha_simulator.py
test_ha_verify_fsm_availableError0.01test_hostha_simulator.py
test_ha_verify_fsm_degradedError0.01test_hostha_simulator.py
test_ha_verify_fsm_fencedError0.01test_hostha_simulator.py
test_ha_verify_fsm_recoveringError0.01test_hostha_simulator.py
test_hostha_configure_default_driverError0.01test_hostha_simulator.py
test_hostha_configure_invalid_providerError0.01test_hostha_simulator.py
test_hostha_disable_feature_validError0.01test_hostha_simulator.py
test_hostha_enable_feature_validError0.01test_hostha_simulator.py
test_hostha_enable_feature_without_setting_providerError0.01test_hostha_simulator.py
test_list_ha_for_hostError0.01test_hostha_simulator.py
test_list_ha_for_host_invalidError0.01test_hostha_simulator.py
test_list_ha_for_host_validError0.01test_hostha_simulator.py
test_01_host_ping_on_alertError0.10test_host_ping.py
test_01_host_ping_on_alertError0.10test_host_ping.py
test_01_browser_migrate_templateError15.34test_image_store_object_migration.py
test_01_invalid_upgrade_kubernetes_clusterFailure237.87test_kubernetes_clusters.py
test_02_upgrade_kubernetes_clusterFailure247.94test_kubernetes_clusters.py
test_03_deploy_and_scale_kubernetes_clusterFailure223.49test_kubernetes_clusters.py
test_04_autoscale_kubernetes_clusterFailure239.74test_kubernetes_clusters.py
test_05_basic_lifecycle_kubernetes_clusterFailure222.24test_kubernetes_clusters.py
test_06_delete_kubernetes_clusterFailure230.60test_kubernetes_clusters.py
test_08_upgrade_kubernetes_ha_clusterFailure317.22test_kubernetes_clusters.py
test_10_vpc_tier_kubernetes_clusterFailure258.87test_kubernetes_clusters.py
test_11_test_unmanaged_cluster_lifecycleError93.42test_kubernetes_clusters.py
test_01_add_delete_kubernetes_supported_versionError0.11test_kubernetes_supported_versions.py
login_test_saml_userError3.06test_login.py
test_01_deployVMInSharedNetworkError77.40test_network.py
test_03_destroySharedNetworkFailure1.07test_network.py
ContextSuite context=TestSharedNetwork>:teardownError2.16test_network.py
test_oobm_issue_power_cycleError3.30test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_offError3.33test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_onError3.31test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_resetError2.26test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_softError3.28test_outofbandmanagement_nestedplugin.py
test_oobm_issue_power_statusError2.23test_outofbandmanagement_nestedplugin.py
test_oobm_background_powerstate_syncFailure21.58test_outofbandmanagement.py
test_oobm_background_powerstate_syncError21.58test_outofbandmanagement.py
test_oobm_configure_default_driverError0.05test_outofbandmanagement.py
test_oobm_configure_invalid_driverError0.05test_outofbandmanagement.py
test_oobm_disable_feature_invalidError0.04test_outofbandmanagement.py
test_oobm_disable_feature_validError1.14test_outofbandmanagement.py
test_oobm_enable_feature_invalidError0.04test_outofbandmanagement.py
test_oobm_enable_feature_validError1.11test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzonesError11.82test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzonesError11.82test_outofbandmanagement.py
test_oobm_issue_power_cycleError4.33test_outofbandmanagement.py
test_oobm_issue_power_cycleError4.33test_outofbandmanagement.py
test_oobm_issue_power_offError4.33test_outofbandmanagement.py
test_oobm_issue_power_offError4.33test_outofbandmanagement.py
test_oobm_issue_power_onError4.31test_outofbandmanagement.py
test_oobm_issue_power_onError4.31test_outofbandmanagement.py
test_oobm_issue_power_resetError4.32test_outofbandmanagement.py
test_oobm_issue_power_resetError4.32test_outofbandmanagement.py
test_oobm_issue_power_softError4.36test_outofbandmanagement.py
test_oobm_issue_power_softError4.36test_outofbandmanagement.py
test_oobm_issue_power_statusError3.30test_outofbandmanagement.py
test_oobm_issue_power_statusError3.30test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownershipError1.14test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownershipError1.14test_outofbandmanagement.py
test_oobm_zchange_passwordError1.17test_outofbandmanagement.py
test_oobm_zchange_passwordError1.17test_outofbandmanagement.py
test_02_edit_primary_storage_tagsError0.01test_primary_storage.py
test_01_primary_storage_scope_changeError0.07test_primary_storage_scope.py
test_01_vpc_privategw_aclError0.03test_privategw_acl_ovs_gre.py
test_03_vpc_privategw_restart_vpc_cleanupError0.02test_privategw_acl_ovs_gre.py
test_05_vpc_privategw_check_interfaceError0.02test_privategw_acl_ovs_gre.py
test_01_vpc_privategw_aclError55.70test_privategw_acl.py
test_02_vpc_privategw_static_routesError208.13test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanupError202.28test_privategw_acl.py
test_04_rvpc_privategw_static_routesError321.51test_privategw_acl.py
test_01_purge_expunged_api_vm_start_dateError47.88test_purge_expunged_vms.py
test_02_purge_expunged_api_vm_end_dateError44.28test_purge_expunged_vms.py
test_03_purge_expunged_api_vm_start_end_dateError43.05test_purge_expunged_vms.py
test_04_purge_expunged_api_vm_no_dateError41.03test_purge_expunged_vms.py
test_05_purge_expunged_vm_service_offeringError269.14test_purge_expunged_vms.py
test_06_purge_expunged_vm_background_taskError333.31test_purge_expunged_vms.py
test_01_snapshot_root_diskError7.37test_snapshots.py
test_CreateTemplateWithDuplicateNameError19.75test_templates.py
test_01_register_template_direct_download_flagError0.15test_templates.py
test_01_positive_tests_usageError13.49test_usage_events.py
test_01_ISO_usageError1.08test_usage.py
test_01_lb_usageError5.26test_usage.py
test_01_nat_usageError7.33test_usage.py
test_01_public_ip_usageError1.08test_usage.py
test_01_snapshot_usageError25.75test_usage.py
test_01_template_usageError17.09test_usage.py
test_01_vm_usageError136.44test_usage.py
test_01_volume_usageError125.08test_usage.py
test_01_vpn_usageError9.52test_usage.py
test_12_start_vm_multiple_volumes_allocatedError12.61test_vm_life_cycle.py
test_01_vmschedule_createError0.10test_vm_schedule.py
test_disable_oobm_ha_state_ineligibleError0.05test_hostha_kvm.py
test_hostha_configure_default_driverError0.04test_hostha_kvm.py
test_hostha_enable_ha_when_host_disabledError0.04test_hostha_kvm.py
test_hostha_enable_ha_when_host_disconectedError0.04test_hostha_kvm.py
test_hostha_enable_ha_when_host_in_maintenanceError0.04test_hostha_kvm.py
test_hostha_kvm_host_degradedError0.04test_hostha_kvm.py
test_hostha_kvm_host_fencingError0.04test_hostha_kvm.py
test_hostha_kvm_host_recoveringError0.05test_hostha_kvm.py
test_remove_ha_provider_not_possibleError0.04test_hostha_kvm.py

@DaanHoogland

Copy link
Copy Markdown
Contributor

@blueorangutan test alma9 kvm-alma9

@blueorangutan

Copy link
Copy Markdown

@DaanHoogland a [SL] Trillian-Jenkins test job (alma9 mgmt + kvm-alma9) has been kicked to run smoke tests

@blueorangutan

Copy link
Copy Markdown

[SF] Trillian test result (tid-10823)
Environment: kvm-alma9 (x2), Advanced Networking with Mgmt server a9
Total time taken: 51389 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8739-t10823-kvm-alma9.zip
Smoke tests completed. 136 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

TestResultTime (s)Test File
test_06_purge_expunged_vm_background_taskFailure332.40test_purge_expunged_vms.py

@DaanHoogland

Copy link
Copy Markdown
Contributor

just noticed I didn't report back here; tests ok:
image
used blApache to make sure nothing fishy was going on there ;)

@weizhouapache

Copy link
Copy Markdown
MemberAuthor

just noticed I didn't report back here; tests ok:
image
used blApache to make sure nothing fishy was going on there ;)

Great, thanks @DaanHoogland !

@weizhouapache
weizhouapache requested a review from yadvrJuly 23, 2024 22:27

@vishesh92vishesh92 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clgtm. didn't test.

@DaanHoogland
DaanHoogland merged commit 1662af1 into apache:mainJul 25, 2024
@DaanHoogland
DaanHoogland deleted the 4.20-vr-fix-binary-userdata branch July 25, 2024 08:05
dhslove pushed a commit to ablecloud-team/ablestack-cloud that referenced this pull request Jul 31, 2024
@DaanHooglandDaanHoogland removed their assignment Dec 24, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@weizhouapache@blueorangutan@DaanHoogland@vishesh92@sureshanaparti