security_group: Use execute() function instead of non-existing bash() by wido · Pull Request #2732 · apache/cloudstack

wido · 2018-07-05T12:12:37Z

In 6233a77 as a part of PR #2432 the
bash() function was replaced by the execute() function.

Somehow this last calling of the bash() function was not caught by testing
and is still in there.

This causes Exceptions to be thrown by the Security Group script.

Signed-off-by: Wido den Hollander wido@widodh.nl

borisstoyanov · 2018-07-05T13:21:22Z

@blueorangutan package

blueorangutan · 2018-07-05T13:21:46Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-07-05T13:51:25Z

Packaging result: ✔centos6 ✖centos7 ✔debian. JID-2164

GabrielBrascher

Code LGTM, tested with master and it fixed the exceptions related with the bash. Thanks.

yadvr · 2018-07-06T07:19:42Z

@wido @GabrielBrascher can you show some test results wrt SG+KVM?

borisstoyanov · 2018-07-06T07:54:18Z

@blueorangutan package

blueorangutan · 2018-07-06T07:54:47Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-07-06T08:26:06Z

Packaging result: ✖centos6 ✔centos7 ✖debian. JID-2165

yadvr · 2018-07-06T09:01:22Z

@blueorangutan package

blueorangutan · 2018-07-06T09:01:46Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-07-06T09:28:23Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2166

borisstoyanov · 2018-07-06T10:02:28Z

@blueorangutan test

blueorangutan · 2018-07-06T10:02:47Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-07-06T19:25:15Z

Trillian test result (tid-2837)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32135 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2732-t2837-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_certauthority_root.py
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 62 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_provision_certificate	`Error`	8.17	test_certauthority_root.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	1238.09	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	373.81	test_privategw_acl.py
test_01_secure_vm_migration	`Error`	5.13	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	1.07	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1.07	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	3.11	test_vm_life_cycle.py
test_11_migrate_volume_and_change_offering	`Error`	128.29	test_volumes.py

borisstoyanov

@wido @GabrielBrascher I think most of the errors within the smoketest results are addressed by now, we should be able to get clean results if you sync with latest master. Can you also share some test results with Security groups on?

In 6233a77 as a part of PR apache#2432 the bash() function was replaced by the execute() function. Somehow this last calling of the bash() function was not caught by testing and is still in there. This causes Exceptions to be thrown by the Security Group script. Signed-off-by: Wido den Hollander <wido@widodh.nl>

GabrielBrascher · 2018-07-18T18:53:00Z

The following log messages are from /var/log/cloudstack/agent/security_group.log.
On 4.11.1:

2018-07-10 21:27:00,514 - Executing command: network_rules_vmSecondaryIp
2018-07-10 21:27:00,514 - vmName = i-2-3-VM
2018-07-10 21:27:00,514 - action = -A
2018-07-10 21:27:00,516 - vm ip 192.168.100.77
2018-07-10 21:27:00,516 - ipset -A i-2-3-VM 192.168.100.77
2018-07-10 21:27:00,520 - ip = 192.168.100.77
2018-07-10 21:27:00,520 - ebtables -t nat -I i-2-3-VM-in-ips -p ARP --arp-ip-src 192.168.100.77 -j RETURN
2018-07-10 21:27:00,524 - ebtables -t nat -I i-2-3-VM-out-ips -p ARP --arp-ip-dst 192.168.100.77 -j RETURN
2018-07-10 21:27:40,175 - Executing command: get_rule_logs_for_vms
2018-07-10 21:27:47,250 - Executing command: destroy_network_rules_for_vm
2018-07-10 21:27:47,250 - iptables-save | awk '/BF(.*)physdev-is-bridged(.*)i-2-3-def/ { sub(/-A/, "-D", $1) ; print }'
2018-07-10 21:27:47,257 - iptables -D BF-cloudbr1-IN -m physdev --physdev-in vnet8 --physdev-is-bridged -j i-2-3-def
2018-07-10 21:27:47,261 - iptables -D BF-cloudbr1-OUT -m physdev --physdev-out vnet8 --physdev-is-bridged -j i-2-3-def
2018-07-10 21:27:47,264 - ip6tables-save | awk '/BF(.*)physdev-is-bridged(.*)i-2-3-def/ { sub(/-A/, "-D", $1) ; print }'
2018-07-10 21:27:47,269 - ip6tables -D BF-cloudbr1-IN -m physdev --physdev-in vnet8 --physdev-is-bridged -j i-2-3-def
2018-07-10 21:27:47,274 - ip6tables -D BF-cloudbr1-OUT -m physdev --physdev-out vnet8 --physdev-is-bridged -j i-2-3-def
2018-07-10 21:27:47,277 - ebtables -t nat -L PREROUTING | grep i-2-3-VM
2018-07-10 21:27:47,283 - ebtables -t nat -L POSTROUTING | grep i-2-3-VM
2018-07-10 21:27:47,288 - ebtables -t nat -D PREROUTING -i vnet8 -j i-2-3-VM-in
2018-07-10 21:27:47,293 - ebtables -t nat -D POSTROUTING -o vnet8 -j i-2-3-VM-out
2018-07-10 21:27:47,298 - ebtables -t nat -F i-2-3-VM-in
2018-07-10 21:27:47,304 - ebtables -t nat -X i-2-3-VM-in
2018-07-10 21:27:47,309 - ebtables -t nat -F i-2-3-VM-out
2018-07-10 21:27:47,315 - ebtables -t nat -X i-2-3-VM-out
2018-07-10 21:27:47,320 - ebtables -t nat -F i-2-3-VM-in-ips
2018-07-10 21:27:47,326 - ebtables -t nat -X i-2-3-VM-in-ips
2018-07-10 21:27:47,331 - ebtables -t nat -F i-2-3-VM-out-ips
2018-07-10 21:27:47,337 - ebtables -t nat -X i-2-3-VM-out-ips

After updating to 4.12:
Note that there are lines returning non-zero exit status 1 and other lines returning exit status 255, which does not necessary means an error.

2018-07-10 22:08:04,889 - Failed to execute: ebtables -t nat -L PREROUTING | grep s-5-VM
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 62, in execute
return check_output(cmd, shell=True)
File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'ebtables -t nat -L PREROUTING | grep s-5-VM' returned non-zero exit status 1
2018-07-10 22:08:04,891 - ebtables -t nat -L POSTROUTING | grep s-5-VM
2018-07-10 22:08:04,895 - Failed to execute: ebtables -t nat -L POSTROUTING | grep s-5-VM
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 62, in execute
return check_output(cmd, shell=True)
File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'ebtables -t nat -L POSTROUTING | grep s-5-VM' returned non-zero exit status 1
2018-07-10 22:08:04,895 - ebtables -t nat -F s-5-VM-in
2018-07-10 22:08:04,898 - Failed to execute: ebtables -t nat -F s-5-VM-in
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 62, in execute
return check_output(cmd, shell=True)
File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'ebtables -t nat -F s-5-VM-in' returned non-zero exit status 255
2018-07-10 22:08:04,898 - ebtables -t nat -X s-5-VM-in
2018-07-10 22:08:04,902 - Failed to execute: ebtables -t nat -X s-5-VM-in
Traceback (most recent call last):
File "/usr/share/cloudstack-common/scripts/vm/network/security_group.py", line 62, in execute
return check_output(cmd, shell=True)
File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command 'ebtables -t nat -X s-5-VM-in' returned non-zero exit status 255

After adding changes from this PR:

2018-07-18 19:46:43,797 - vmName = i-2-38-VM
2018-07-18 19:46:43,797 - action = -D
2018-07-18 19:46:43,798 - vm ip 2001:db8::2
2018-07-18 19:46:43,798 - ipset -D i-2-38-VM 2001:db8::2
2018-07-18 19:46:43,804 - ip = 2001:db8::2
2018-07-18 19:46:43,804 - ebtables -t nat -D i-2-38-VM-in-ips -p ARP --arp-ip-src 2001:db8::2 -j RETURN
2018-07-18 19:46:43,809 - ebtables -t nat -D i-2-38-VM-out-ips -p ARP --arp-ip-dst 2001:db8::2 -j RETURN
2018-07-18 19:53:08,700 - Executing command: cleanup_rules
2018-07-18 19:53:08,703 - Vms on the host : ['s-34-VM', 'v-35-VM', 'r-37-VM', 'i-2-38-VM']
2018-07-18 19:53:08,703 - iptables-save | grep -P '^:(?!.*-(def|eg))' | awk '{sub(/^:/, "", $1) ; print $1}' | sort | uniq
2018-07-18 19:53:08,710 - iptables chains in the host :['BF-cloudbr0', 'BF-cloudbr0-IN', 'BF-cloudbr0-OUT', 'BF-cloudbr1', 'BF-cloudbr1-IN', 'BF-cloudbr1-OUT', 'FORWARD', 'i-2-38-VM', 'INPUT', 'OUTPUT', 'POSTROUTING', 'PREROUTING', 'r-37-VM', 's-34-VM', 'v-35-VM', '']
2018-07-18 19:53:08,711 - grep -E '^ebtable_' /proc/modules | cut -f1 -d' ' | sed s/ebtable_//
2018-07-18 19:53:08,716 - ebtables -t nat -L | awk '/chain:/ { gsub(/(^.*chain: |-(in|out|ips).*)/, ""); print $1}' | sort | uniq
2018-07-18 19:53:08,724 - ebtables -t filter -L | awk '/chain:/ { gsub(/(^.*chain: |-(in|out|ips).*)/, ""); print $1}' | sort | uniq
2018-07-18 19:53:08,732 - ebtables chains in the host: ['FORWARD,', 'INPUT,', 'OUTPUT,', '']
2018-07-18 19:53:08,732 - Cleaned up rules for 0 chains
2018-07-18 20:37:18,701 - Executing command: cleanup_rules
2018-07-18 20:37:18,704 - Vms on the host : ['s-34-VM', 'v-35-VM', 'r-37-VM', 'i-2-38-VM']
2018-07-18 20:37:18,704 - iptables-save | grep -P '^:(?!.*-(def|eg))' | awk '{sub(/^:/, "", $1) ; print $1}' | sort | uniq
2018-07-18 20:37:18,710 - iptables chains in the host :['BF-cloudbr0', 'BF-cloudbr0-IN', 'BF-cloudbr0-OUT', 'BF-cloudbr1', 'BF-cloudbr1-IN', 'BF-cloudbr1-OUT', 'FORWARD', 'i-2-38-VM', 'INPUT', 'OUTPUT', 'POSTROUTING', 'PREROUTING', 'r-37-VM', 's-34-VM', 'v-35-VM', '']
2018-07-18 20:37:18,710 - grep -E '^ebtable_' /proc/modules | cut -f1 -d' ' | sed s/ebtable_//
2018-07-18 20:37:18,714 - ebtables -t nat -L | awk '/chain:/ { gsub(/(^.*chain: |-(in|out|ips).*)/, ""); print $1}' | sort | uniq
2018-07-18 20:37:18,722 - ebtables -t filter -L | awk '/chain:/ { gsub(/(^.*chain: |-(in|out|ips).*)/, ""); print $1}' | sort | uniq
2018-07-18 20:37:18,727 - ebtables chains in the host: ['FORWARD,', 'INPUT,', 'OUTPUT,', '']
2018-07-18 20:37:18,727 - Cleaned up rules for 0 chains
2018-07-18 20:43:01,624 - Executing command: network_rules_vmSecondaryIp
2018-07-18 20:43:01,624 - vmName = i-2-38-VM
2018-07-18 20:43:01,624 - action = -A
2018-07-18 20:43:01,626 - vm ip 192.168.100.71
2018-07-18 20:43:01,626 - ipset -A i-2-38-VM 192.168.100.71
2018-07-18 20:43:01,629 - ip = 192.168.100.71
2018-07-18 20:43:01,629 - ebtables -t nat -I i-2-38-VM-in-ips -p ARP --arp-ip-src 192.168.100.71 -j RETURN
2018-07-18 20:43:01,632 - ebtables -t nat -I i-2-38-VM-out-ips -p ARP --arp-ip-dst 192.168.100.71 -j RETURN

…apache#2732) In 6233a77 as a part of PR apache#2432 the bash() function was replaced by the execute() function. Somehow this last calling of the bash() function was not caught by testing and is still in there. This causes Exceptions to be thrown by the Security Group script. Signed-off-by: Wido den Hollander <wido@widodh.nl>

wido added component:kvm component:security-group labels Jul 5, 2018

wido added this to the 4.11.2.0 milestone Jul 5, 2018

wido requested a review from GabrielBrascher July 5, 2018 12:12

wido modified the milestones: 4.11.2.0, 4.12.0.0 Jul 5, 2018

GabrielBrascher approved these changes Jul 5, 2018
View reviewed changes

yadvr approved these changes Jul 6, 2018
View reviewed changes

borisstoyanov approved these changes Jul 9, 2018
View reviewed changes

wido force-pushed the secgroup-bash-function branch from d99d68c to 9c2fa03Compare July 18, 2018 09:32

DaanHoogland merged commit 1ccb32f into apache:masterJul 20, 2018

Uh oh!

Conversation

wido commented Jul 5, 2018

Uh oh!

borisstoyanov commented Jul 5, 2018

Uh oh!

blueorangutan commented Jul 5, 2018

Uh oh!

blueorangutan commented Jul 5, 2018

Uh oh!

GabrielBrascher left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Jul 6, 2018

Uh oh!

borisstoyanov commented Jul 6, 2018

Uh oh!

blueorangutan commented Jul 6, 2018

Uh oh!

blueorangutan commented Jul 6, 2018

Uh oh!

yadvr commented Jul 6, 2018

Uh oh!

blueorangutan commented Jul 6, 2018

Uh oh!

blueorangutan commented Jul 6, 2018

Uh oh!

borisstoyanov commented Jul 6, 2018

Uh oh!

blueorangutan commented Jul 6, 2018

Uh oh!

blueorangutan commented Jul 6, 2018

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

GabrielBrascher commented Jul 18, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

GabrielBrascher commented Jul 18, 2018•
edited
Loading