[CLOUDSTACK-10226] CloudStack is not importing Local storage properly

[rafaelweingartner]

CloudStack is importing as Local storage any XenServer SR that is of type LVM or EXT. This causes a problem when one wants to use both Direct attach storage and local storage. Moreover, CloudStack was not importing all of the local storage that a host has available when local storage is enabled. It was only importing the First SR it sees.

To fix the first problem we started ignoring SRs that have the flag shared=true when discovering local storages. SRs configured to be shared are used as direct attached storage, and therefore should not be imported again as local ones.

To fix the second problem, we started loading all Local storage and importing them accordingly to ACS.

Tip for reviewers: the code changes is mainly in CitrixResourceBase, method initializeLocalSR

[rafaelweingartner]

@blueorangutan package

[blueorangutan]

@rafaelweingartner a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1792

[DaanHoogland]

@blueorangutan help

[blueorangutan]

@DaanHoogland I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [additional tests]
Mgmt OS options: ['centos6', 'centos7', 'ubuntu']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-ubuntu', 'xenserver-71', 'xenserver-65sp1', 'xenserver-62sp1', 'vmware-65', 'vmware-60u2', 'vmware-55u3', 'vmware-51u1', 'vmware-50u1']
Additional tests: list of space separated tests with paths relative to the test/integration directory, for example: component/test_acl_listvm.py component/test_volumes.py
Note: when additional tests are passed, you need to specify mgmt server os and hypervisor or use the matrix command.

Blessed contributors for kicking Trillian test jobs: ['rhtyd', 'nvazquez', 'PaulAngus', 'borisstoyanov', 'DaanHoogland']

[DaanHoogland]

@blueorangutan test centos7 xenserver-71

[blueorangutan]

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[rafaelweingartner]

I checked the logs. The problems did not seem to be related to the PR.
However, it is always good to have the feedback of an expert ;)

[blueorangutan]

Trillian test result (tid-2436)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 22786 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2401-t2436-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_certauthority_root.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 66 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_rvpc_network_garbage_collector_nics	Failure	452.86	test_vpc_redundant.py

[rafaelweingartner]

@DaanHoogland I checked this last log result. Can I consider that the tests is successful then?

[DaanHoogland]

@rafaelweingartner , i'm cautious here. This one seems consistent. Did you explain the cause of the failure?
@mike-tutkowski can you review this (and add yourself to the apache org)?

[rafaelweingartner]

@DaanHoogland I checked the error, and I still do not understand how it could be related to the code of this PR. Here I am only changing how local storages are detected and loaded to the system. Whereas, the error is related to redundant VRs in VPCs.

Anyways, I am trying to run the tests locally with simulator, but I am getting this error:

ERROR: do_vpc_test (integration.smoke.test_vpc_redundant.TestVPCRedundancy)
----------------------------------------------------------------------
TypeError: do_vpc_test() takes exactly 2 arguments (1 given)
-------------------- >> begin captured stdout << ---------------------
=== TestName: do_vpc_test | Status : EXCEPTION ===

I am not understanding the cause of that error. This is the command I am using to execute the test:
nosetests --with-marvin --marvin-config=/root/marvinCloudStackConfig.cfg --hypervisor=simulator /root/cloudstack/test/integration/smoke/test_vpc_redundant.py

Am I missing something else that is required to run tests?

[rafaelweingartner]

@mike-tutkowski could you share your wisdom with us here?

[mike-tutkowski]

Sorry @DaanHoogland - I didn't seem to get notified when you tagged me. I just noticed this PR when @rafaelweingartner tagged me today. Let me review this PR and think about it.

[mike-tutkowski]

I do not see how the VPC issue would be related to the code of this PR either. The PR code LGTM.

[rafaelweingartner]

@borisstoyanov and @DaanHoogland shall we execute the tests again?

[DaanHoogland]

I don't think it makes sense but no hurt
@blueorangutan centos7 xenserver-71

[rafaelweingartner]

@DaanHoogland I do not think @blueorangutan heard your request for testing here...

[borisstoyanov]

@blueorangutan test centos7 xenserver-71

[blueorangutan]

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + xenserver-71) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-2478)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 22757 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2401-t2478-xenserver-71.zip
Intermitten failure detected: /marvin/tests/smoke/test_certauthority_root.py
Intermitten failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermitten failure detected: /marvin/tests/smoke/test_ssvm.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 64 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	Error	9.23	test_scale_vm.py
ContextSuite context=TestCpuCapServiceOfferings>:teardown	Error	0.00	test_service_offerings.py
test_04_rvpc_network_garbage_collector_nics	Failure	518.16	test_vpc_redundant.py

[rafaelweingartner]

@borisstoyanov now I believe that the test result is ok. These failures seem to be intermittent, not just here, but in other PRs as well.

[borisstoyanov]

yes @rafaelweingartner, LGTM

[DaanHoogland]

looks great as usual @rafaelweingartner

[yadvr]

@rafaelweingartner I scanned PRs merged on master but not 4.11 and came across this PR, do you think this would be useful for 4.11? If so, can you help create a backport PR for 4.11?

[rafaelweingartner]

I do not think so. It seems that no other people had this necessity so far. This PR enables people to use local storage and pre-setup storage at the same time. Moreover, I organized things, and now it is possible to have more than a single local storage per host.

[yadvr]

@rafaelweingartner fair enough, but this seems a useful fix for LTS nonetheless. I'll leave the decision of backporting to you on this (and other PRs).