CLOUDSTACK-9801: IPSec VPN does not work after vRouter reboot or recreate

[Slair1]

After a vRouter is recreated (e.g. reboot via CloudStack UI) and Remote Access VPN enabled, VPN won't work anymore. Here is the abbreviated output of "ipsec auto -status" while we were having the issue:

root@r-10-VM:~# ipsec auto --status
000 using kernel interface: netkey
000 interface lo/lo 127.0.0.1
000 interface lo/lo 127.0.0.1
000 interface eth0/eth0 169.254.1.45
000 interface eth0/eth0 169.254.1.45
000 %myid = (none)

After this commit, the following occurs and VPNs work:

root@r-10-VM:~# ipsec auto --status
000 using kernel interface: netkey
000 interface lo/lo 127.0.0.1
000 interface lo/lo 127.0.0.1
000 interface eth0/eth0 169.254.1.45
000 interface eth0/eth0 169.254.1.45
000 interface eth1/eth1 xxx.xxx.xxx.172
000 interface eth1/eth1 xxx.xxx.xxx.172
000 interface eth2/eth2 192.168.1.1
000 interface eth2/eth2 192.168.1.1
000 %myid = (none)

eth1 interface IP is masked, but now ipsec sees all the interfaces and VPN works.

Looks like this bug was introduced by Pull Request #1423

It added code to start ipsec (cloudstack/systemvm/patches/debian/config/opt/cloud/bin/configure.py)

if vpnconfig['create']:
logging.debug("Enabling remote access vpn on "+ public_ip)
CsHelper.start_if_stopped("ipsec")

[swill]

I can't see what branch this is opened against on my phone. What version of ACS is this opened against and which version do you have a problem in. The reason I ask is because #1741 was added in master to upgrade from openswan to strongswan. . Thx...

[asfbot]

Sean Lair on dev@cloudstack.apache.org replies:
It is open against the 4.9 branch.

We are running 4.9.2.0, looks like it affects all 4.9.x.x

We haven't tested against 4.10 (strongswan) yet. But it could be a problem=
and will be worth testing. If strongswan starts before cloudstack adds th=
e nics to the VM it could have same issue.
=20
on of ACS is this opened against and which version do you have a problem in=
. The reason I ask is because #1741 was added in master to upgrade from ope=
nswan to strongswan. . Thx...=20
ur
e
se

[asfbot]

Will Stevens on dev@cloudstack.apache.org replies:
I ha e done pretty extensive testing of the strongswan implementation, but
would be happy for you to verify if the problem exists in 4.10.

[kiwiflyer]

@Slair1@swill Gents, where do we stand on this PR? I know there is a whole bunch of stuff related to this in 4.10, but this was opened against 4.9.

[cloudmonger]

ACS CI BVT Run

Sumarry:
Build Number 760
Hypervisor xenserver
NetworkType Advanced
Passed=108
Failed=2
Skipped=12

Link to logs Folder (search by build_no):https://www.dropbox.com/sh/r2si930m8xxzavs/AAAzNrnoF1fC3auFrvsKo_8-a?dl=0

Failed tests:

• test_list_ids_parameter.py

• ContextSuite context=TestListIdsParams>:setup Failing since 21 runs

• test_volumes.py

• test_06_download_detached_volume Failed

Skipped tests:
test_vm_nic_adapter_vmxnet3
test_01_verify_libvirt
test_02_verify_libvirt_after_restart
test_03_verify_libvirt_attach_disk
test_04_verify_guest_lspci
test_05_change_vm_ostype_restart
test_06_verify_guest_lspci_again
test_static_role_account_acls
test_11_ss_nfs_version_on_ssvm
test_nested_virtualization_vmware
test_3d_gpu_support
test_deploy_vgpu_enabled_vm

Passed test suits:
test_deploy_vm_with_userdata.py
test_affinity_groups_projects.py
test_portable_publicip.py
test_vm_snapshots.py
test_over_provisioning.py
test_global_settings.py
test_scale_vm.py
test_service_offerings.py
test_routers_iptables_default_policy.py
test_loadbalance.py
test_routers.py
test_reset_vm_on_reboot.py
test_deploy_vms_with_varied_deploymentplanners.py
test_network.py
test_router_dns.py
test_non_contigiousvlan.py
test_login.py
test_deploy_vm_iso.py
test_public_ip_range.py
test_multipleips_per_nic.py
test_metrics_api.py
test_regions.py
test_affinity_groups.py
test_network_acl.py
test_pvlan.py
test_nic.py
test_deploy_vm_root_resize.py
test_resource_detail.py
test_secondary_storage.py
test_vm_life_cycle.py
test_routers_network_ops.py
test_disk_offerings.py

[Slair1]

@kiwiflyer I'm not sure if this issue still exists in 4.10, but it is an issue in 4.9. We've been running this fix since February and it has fixed it. FYI, it is the same code/logic used on line 458. Let me know if you need anything on this

[yadvr]

@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-813

[yadvr]

@blueorangutan test

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-1215)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 37222 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr1966-t1215-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Test completed. 46 look ok, 3 have error(s)

Test	Result	Time (s)	Test File
test_04_rvpc_privategw_static_routes	Failure	441.94	test_privategw_acl.py
test_01_internallb_roundrobin_1VPC_3VM_HTTP_port80	Failure	288.56	test_internal_lb.py
test_02_list_snapshots_with_removed_data_store	Error	0.04	test_snapshots.py
test_01_vpc_site2site_vpn	Success	150.21	test_vpc_vpn.py
test_01_vpc_remote_access_vpn	Success	66.24	test_vpc_vpn.py
test_01_redundant_vpc_site2site_vpn	Success	276.32	test_vpc_vpn.py
test_02_VPC_default_routes	Success	260.44	test_vpc_router_nics.py
test_01_VPC_nics_after_destroy	Success	633.17	test_vpc_router_nics.py
test_05_rvpc_multi_tiers	Success	500.04	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	Success	1371.53	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	Success	584.17	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	Success	790.52	test_vpc_redundant.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	Success	1384.74	test_vpc_redundant.py
test_09_delete_detached_volume	Success	151.61	test_volumes.py
test_08_resize_volume	Success	156.50	test_volumes.py
test_07_resize_fail	Success	161.54	test_volumes.py
test_06_download_detached_volume	Success	156.41	test_volumes.py
test_05_detach_volume	Success	145.77	test_volumes.py
test_04_delete_attached_volume	Success	146.29	test_volumes.py
test_03_download_attached_volume	Success	151.32	test_volumes.py
test_02_attach_volume	Success	79.23	test_volumes.py
test_01_create_volume	Success	768.04	test_volumes.py
test_deploy_vm_multiple	Success	292.95	test_vm_life_cycle.py
test_deploy_vm	Success	0.03	test_vm_life_cycle.py
test_advZoneVirtualRouter	Success	0.02	test_vm_life_cycle.py
test_10_attachAndDetach_iso	Success	27.09	test_vm_life_cycle.py
test_09_expunge_vm	Success	125.19	test_vm_life_cycle.py
test_08_migrate_vm	Success	141.61	test_vm_life_cycle.py
test_07_restore_vm	Success	0.12	test_vm_life_cycle.py
test_06_destroy_vm	Success	130.88	test_vm_life_cycle.py
test_03_reboot_vm	Success	125.86	test_vm_life_cycle.py
test_02_start_vm	Success	10.20	test_vm_life_cycle.py
test_01_stop_vm_forced	Success	5.14	test_vm_life_cycle.py
test_01_stop_vm	Success	30.27	test_vm_life_cycle.py
test_CreateTemplateWithDuplicateName	Success	136.04	test_templates.py
test_08_list_system_templates	Success	0.04	test_templates.py
test_07_list_public_templates	Success	0.04	test_templates.py
test_05_template_permissions	Success	0.06	test_templates.py
test_04_extract_template	Success	5.22	test_templates.py
test_03_delete_template	Success	5.10	test_templates.py
test_02_edit_template	Success	90.13	test_templates.py
test_01_create_template	Success	80.69	test_templates.py
test_10_destroy_cpvm	Success	161.44	test_ssvm.py
test_09_destroy_ssvm	Success	224.19	test_ssvm.py
test_08_reboot_cpvm	Success	101.56	test_ssvm.py
test_07_reboot_ssvm	Success	133.19	test_ssvm.py
test_06_stop_cpvm	Success	131.71	test_ssvm.py
test_05_stop_ssvm	Success	133.52	test_ssvm.py
test_04_cpvm_internals	Success	1.01	test_ssvm.py
test_03_ssvm_internals	Success	2.98	test_ssvm.py
test_02_list_cpvm_vm	Success	0.14	test_ssvm.py
test_01_list_sec_storage_vm	Success	0.17	test_ssvm.py
test_01_snapshot_root_disk	Success	11.17	test_snapshots.py
test_04_change_offering_small	Success	205.23	test_service_offerings.py
test_03_delete_service_offering	Success	0.04	test_service_offerings.py
test_02_edit_service_offering	Success	0.08	test_service_offerings.py
test_01_create_service_offering	Success	0.12	test_service_offerings.py
test_02_sys_template_ready	Success	0.13	test_secondary_storage.py
test_01_sys_vm_start	Success	0.19	test_secondary_storage.py
test_09_reboot_router	Success	30.29	test_routers.py
test_08_start_router	Success	25.43	test_routers.py
test_07_stop_router	Success	10.16	test_routers.py
test_06_router_advanced	Success	0.06	test_routers.py
test_05_router_basic	Success	0.04	test_routers.py
test_04_restart_network_wo_cleanup	Success	5.57	test_routers.py
test_03_restart_network_cleanup	Success	55.54	test_routers.py
test_02_router_internal_adv	Success	1.26	test_routers.py
test_01_router_internal_basic	Success	0.67	test_routers.py
test_router_dns_guestipquery	Success	73.88	test_router_dns.py
test_router_dns_externalipquery	Success	0.08	test_router_dns.py
test_router_dhcphosts	Success	279.49	test_router_dhcphosts.py
test_router_dhcp_opts	Success	21.90	test_router_dhcphosts.py
test_01_updatevolumedetail	Success	0.12	test_resource_detail.py
test_01_reset_vm_on_reboot	Success	130.99	test_reset_vm_on_reboot.py
test_createRegion	Success	0.04	test_regions.py
test_create_pvlan_network	Success	5.22	test_pvlan.py
test_dedicatePublicIpRange	Success	0.47	test_public_ip_range.py
test_03_vpc_privategw_restart_vpc_cleanup	Success	530.89	test_privategw_acl.py
test_02_vpc_privategw_static_routes	Success	466.67	test_privategw_acl.py
test_01_vpc_privategw_acl	Success	127.50	test_privategw_acl.py
test_01_primary_storage_nfs	Success	35.89	test_primary_storage.py
test_createPortablePublicIPRange	Success	15.21	test_portable_publicip.py
test_createPortablePublicIPAcquire	Success	15.56	test_portable_publicip.py
test_isolate_network_password_server	Success	59.74	test_password_server.py
test_UpdateStorageOverProvisioningFactor	Success	0.13	test_over_provisioning.py
test_oobm_zchange_password	Success	30.80	test_outofbandmanagement.py
test_oobm_multiple_mgmt_server_ownership	Success	16.37	test_outofbandmanagement.py
test_oobm_issue_power_status	Success	10.28	test_outofbandmanagement.py
test_oobm_issue_power_soft	Success	10.35	test_outofbandmanagement.py
test_oobm_issue_power_reset	Success	15.38	test_outofbandmanagement.py
test_oobm_issue_power_on	Success	15.37	test_outofbandmanagement.py
test_oobm_issue_power_off	Success	15.36	test_outofbandmanagement.py
test_oobm_issue_power_cycle	Success	10.35	test_outofbandmanagement.py
test_oobm_enabledisable_across_clusterzones	Success	87.78	test_outofbandmanagement.py
test_oobm_enable_feature_valid	Success	5.18	test_outofbandmanagement.py
test_oobm_enable_feature_invalid	Success	0.11	test_outofbandmanagement.py
test_oobm_disable_feature_valid	Success	5.19	test_outofbandmanagement.py
test_oobm_disable_feature_invalid	Success	0.11	test_outofbandmanagement.py
test_oobm_configure_invalid_driver	Success	0.09	test_outofbandmanagement.py
test_oobm_configure_default_driver	Success	0.09	test_outofbandmanagement.py
test_oobm_background_powerstate_sync	Success	18.67	test_outofbandmanagement.py
test_extendPhysicalNetworkVlan	Success	15.35	test_non_contigiousvlan.py
test_01_nic	Success	389.18	test_nic.py
test_releaseIP	Success	207.91	test_network.py
test_reboot_router	Success	414.15	test_network.py
test_public_ip_user_account	Success	10.29	test_network.py
test_public_ip_admin_account	Success	40.32	test_network.py
test_network_rules_acquired_public_ip_3_Load_Balancer_Rule	Success	67.05	test_network.py
test_network_rules_acquired_public_ip_2_nat_rule	Success	61.90	test_network.py
test_network_rules_acquired_public_ip_1_static_nat_rule	Success	121.64	test_network.py
test_delete_account	Success	368.37	test_network.py
test_02_port_fwd_on_non_src_nat	Success	55.74	test_network.py
test_01_port_fwd_on_src_nat	Success	111.97	test_network.py
test_nic_secondaryip_add_remove	Success	227.76	test_multipleips_per_nic.py
test_list_zones_metrics	Success	0.26	test_metrics_api.py
test_list_volumes_metrics	Success	5.47	test_metrics_api.py
test_list_vms_metrics	Success	257.21	test_metrics_api.py
test_list_pstorage_metrics	Success	0.35	test_metrics_api.py
test_list_infrastructure_metrics	Success	0.43	test_metrics_api.py
test_list_hosts_metrics	Success	0.41	test_metrics_api.py
test_list_clusters_metrics	Success	0.36	test_metrics_api.py
login_test_saml_user	Success	19.32	test_login.py
test_assign_and_removal_lb	Success	134.53	test_loadbalance.py
test_02_create_lb_rule_non_nat	Success	187.79	test_loadbalance.py
test_01_create_lb_rule_src_nat	Success	208.22	test_loadbalance.py
test_03_list_snapshots	Success	0.08	test_list_ids_parameter.py
test_02_list_templates	Success	0.04	test_list_ids_parameter.py
test_01_list_volumes	Success	0.03	test_list_ids_parameter.py
test_07_list_default_iso	Success	0.07	test_iso.py
test_05_iso_permissions	Success	0.07	test_iso.py
test_04_extract_Iso	Success	5.17	test_iso.py
test_03_delete_iso	Success	95.15	test_iso.py
test_02_edit_iso	Success	0.09	test_iso.py
test_01_create_iso	Success	21.02	test_iso.py
test_04_rvpc_internallb_haproxy_stats_on_all_interfaces	Success	344.20	test_internal_lb.py
test_03_vpc_internallb_haproxy_stats_on_all_interfaces	Success	183.06	test_internal_lb.py
test_02_internallb_roundrobin_1RVPC_3VM_HTTP_port80	Success	648.05	test_internal_lb.py
test_dedicateGuestVlanRange	Success	10.29	test_guest_vlan_range.py
test_UpdateConfigParamWithScope	Success	0.14	test_global_settings.py
test_rolepermission_lifecycle_update	Success	6.23	test_dynamicroles.py
test_rolepermission_lifecycle_list	Success	6.19	test_dynamicroles.py
test_rolepermission_lifecycle_delete	Success	5.92	test_dynamicroles.py
test_rolepermission_lifecycle_create	Success	5.90	test_dynamicroles.py
test_rolepermission_lifecycle_concurrent_updates	Success	6.04	test_dynamicroles.py
test_role_lifecycle_update_role_inuse	Success	5.92	test_dynamicroles.py
test_role_lifecycle_update	Success	11.00	test_dynamicroles.py
test_role_lifecycle_list	Success	5.92	test_dynamicroles.py
test_role_lifecycle_delete	Success	11.00	test_dynamicroles.py
test_role_lifecycle_create	Success	5.91	test_dynamicroles.py
test_role_inuse_deletion	Success	5.87	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	Success	8.23	test_dynamicroles.py
test_role_account_acls	Success	8.64	test_dynamicroles.py
test_default_role_deletion	Success	6.08	test_dynamicroles.py
test_04_create_fat_type_disk_offering	Success	0.08	test_disk_offerings.py
test_03_delete_disk_offering	Success	0.05	test_disk_offerings.py
test_02_edit_disk_offering	Success	0.05	test_disk_offerings.py
test_02_create_sparse_type_disk_offering	Success	0.07	test_disk_offerings.py
test_01_create_disk_offering	Success	0.11	test_disk_offerings.py
test_deployvm_userdispersing	Success	20.63	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userconcentrated	Success	20.61	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_firstfit	Success	116.35	test_deploy_vms_with_varied_deploymentplanners.py
test_deployvm_userdata_post	Success	45.59	test_deploy_vm_with_userdata.py
test_deployvm_userdata	Success	226.83	test_deploy_vm_with_userdata.py
test_02_deploy_vm_root_resize	Success	5.97	test_deploy_vm_root_resize.py
test_01_deploy_vm_root_resize	Success	6.04	test_deploy_vm_root_resize.py
test_00_deploy_vm_root_resize	Success	252.80	test_deploy_vm_root_resize.py
test_deploy_vm_from_iso	Success	202.70	test_deploy_vm_iso.py
test_DeployVmAntiAffinityGroup	Success	106.34	test_affinity_groups.py
test_03_delete_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_02_revert_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_01_create_vm_snapshots	Skipped	0.00	test_vm_snapshots.py
test_06_copy_template	Skipped	0.00	test_templates.py
test_static_role_account_acls	Skipped	0.03	test_staticroles.py
test_01_scale_vm	Skipped	0.00	test_scale_vm.py
test_01_primary_storage_iscsi	Skipped	0.04	test_primary_storage.py
test_06_copy_iso	Skipped	0.00	test_iso.py
test_deploy_vgpu_enabled_vm	Skipped	0.01	test_deploy_vgpu_enabled_vm.py

[yadvr]

Regression tests have no new error (except known intermittent failures)
@Slair1@swill@kiwiflyer any comment, can this break if we fwd-merge on 4.10+? Has anyone tested this, advise on merging this?

[yadvr]

I'm not in position to decide merging this one, needs an additional review from one of the VPN gurus @swill@JayapalUradi and others

[yadvr]

Ping, any discussions/reviews?

[Slair1]

It's the same code/logic used on line 458, so should be safe

[yadvr]

LGTM, merging based on your feedback @Slair1 and my assessment of the code, and test results.