VM ingestion by shwstppr · Pull Request #3606 · apache/cloudstack

shwstppr · 2019-09-24T15:32:21Z

Description

Documentation PR: apache/cloudstack-documentation#84
FS Doc: https://cwiki.apache.org/confluence/display/CLOUDSTACK/VM+Ingestion

VM ingestion feature allows CloudStack to discover, on-board, import existing VMs in an infra. The feature currently works only for VMware, with a hypervisor agnostic framework which may be extended for KVM and XenServer in future.
Two new APIs have been added, listUnmanagedInstances and importUnmanagedInstance.

listUnmanagedInstances

API will list all unmanaged virtual machine for a give cluster. Optionally, name for an existing unmanaged virtual machine can be given to retrieve VM details.
API request params -
clusterid(UUID of cluster)
name(instance name)
Response -
clusterid
hostid
name
osdisplayname
memory
powerstate
cpuCoresPerSocket
cpunumber
cpuspeed
disk
- id
- capacity
- controller
- controllerunit
- imagepath
- position
nic
- id
- macaddress
- networkname
- vlanid
- pcislot

importUnmanagedInstance

API will import an exisitng unmanaged virtual machine into CloudStack for a given cluster and virtual machine name. Service offering for the VM, disk offerings for volumes and networks for NICs of the VM can be mapped. Some optional parameters like projectid, domainid, hostname, details, etc can also be given. Appropritate networks, service offering and disk offerings need to be present before import and cannot be created on the fly during the API call.
API request params -
clusterid(UUID of cluster)
name(instance name)
displayname
hostname
domainid
projectid
templateid
serviceofferingid
nicnetworklist(Map for NIC ID and corresponding Network UUID)
datadiskofferinglist(Map for disk ID and corresponding disk offering UUID)
details(Map for VM details)
migrateallowed(VM and its volumes are allowed to migrate to different host/pool
Response -
Same response as that of deployVirtualMachine API

A python based script (scripts/vm/hypervisor/vmware/discover_networks.py) has been created to aid migrations. It leverages VMware’s pyvmomi library (https://github.com/vmware/pyvmomi) and allows listing all networks for a vCenter host or cluster.

This script can take following arguments,

-h, --help show this help message and exit
-s HOST, --host HOST vSphere service to connect to
-o PORT, --port PORT Port to connect on
-u USER, --user USER User name to use
-p PASSWORD, --password PASSWORD Password to use
-c CLUSTER, --cluster CLUSTER Cluster for which discover networks
-S, --disable_ssl_verification Disable ssl host certificate verification
-d, --debug Debug log messages

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

API sample calls:

List

> list unmanagedinstances clusterid=079a19c2-2710-4ed6-9110-027dd5e0f262 {
"count": 2,
"unmanagedinstance": [
{
"clusterid": "079a19c2-2710-4ed6-9110-027dd5e0f262",
"cpucorespersocket": 1,
"cpunumber": 1,
"cpuspeed": 0,
"disk": [
{
"capacity": 2097152,
"controller": "ide",
"controllerunit": 0,
"id": "51-3001",
"imagepath": "[8aca30d6f7803420973f01cc91c1c011] test-unmanaged/test-unmanaged_64.vmdk",
"label": "Hard disk 1",
"position": 1
},
{
"capacity": 5242880,
"controller": "scsi",
"controllerunit": 0,
"id": "51-2000",
"imagepath": "[8aca30d6f7803420973f01cc91c1c011] test-unmanaged/test-unmanaged_63.vmdk",
"label": "Hard disk 2",
"position": 0
}
],
"hostid": "4ab4147a-7e40-4abe-a0e2-2fac1aa6f0bf",
"memory": 512,
"name": "test-unmanaged",
"nic": [
{
"id": "Network adapter 1",
"macaddress": "02:00:39:7c:00:24",
"networkname": "cloud.guest.1212.200.1-vSwitch1",
"vlanid": 1212
}
],
"osdisplayname": "CentOS 4/5 or later (64-bit)",
"powerstate": "POWERED_ON"
}
]
}

Import

> import unmanagedinstance name=test-unmanaged templateid=200f3fe9-c845-11e9-b69f-1e003c0107c4 clusterid=079a19c2-2710-4ed6-9110-027dd5e0f262 serviceofferingid=c4c5b9e8-8a4a-422d-bccc-2f1bcc166c59 datadiskofferinglist[0].disk=51-2000 datadiskofferinglist[0].diskOffering=e9eb069f-2946-4e26-8d81-07aebba868d1 displayname=NewVm nicNetworkList[0].nic="Network adapter 1" nicNetworkList[0].network=4c44312b-f009-4d33-b942-989169f3d2fc
{
"virtualmachine": {
"account": "admin",
"affinitygroup": [],
"cpunumber": 1,
"cpuspeed": 500,
"created": "2019-09-05T11:32:04+0000",
"details": {},
"diskofferingid": "e9eb069f-2946-4e26-8d81-07aebba868d1",
"diskofferingname": "Custom",
"displayname": "NewVm",
"displayvm": true,
"domain": "ROOT",
"domainid": "20088b4d-c845-11e9-b69f-1e003c0107c4",
"guestosid": "2015cd08-c845-11e9-b69f-1e003c0107c4",
"haenable": false,
"hostid": "4ab4147a-7e40-4abe-a0e2-2fac1aa6f0bf",
"hostname": "10.2.2.144",
"hypervisor": "VMware",
"id": "4fda861e-e693-4227-a8aa-1b4ce90d5041",
"instancename": "test-unmanaged",
"isdynamicallyscalable": false,
"memory": 512,
"name": "test-unmanaged",
"nic": [
{
"broadcasturi": "vlan://1212",
"extradhcpoption": [],
"id": "61952f25-7efa-4bc2-aa92-ecd38c122e65",
"isdefault": true,
"isolationuri": "vlan://1212",
"macaddress": "02:00:39:7c:00:24",
"networkid": "4c44312b-f009-4d33-b942-989169f3d2fc",
"networkname": "lol",
"secondaryip": [],
"traffictype": "Guest",
"type": "L2"
}
],
"ostypeid": "2015cd08-c845-11e9-b69f-1e003c0107c4",
"passwordenabled": false,
"rootdeviceid": 0,
"rootdevicetype": "ROOT",
"securitygroup": [],
"serviceofferingid": "c4c5b9e8-8a4a-422d-bccc-2f1bcc166c59",
"serviceofferingname": "Small Instance",
"state": "Stopped",
"tags": [],
"templatedisplaytext": "CentOS 5.3(64-bit) no GUI (vSphere)",
"templateid": "200f3fe9-c845-11e9-b69f-1e003c0107c4",
"templatename": "CentOS 5.3(64-bit) no GUI (vSphere)",
"userid": "4dfc315c-c845-11e9-b69f-1e003c0107c4",
"username": "admin",
"zoneid": "db18d224-279d-4c08-aeef-0cb7030d0edb",
"zonename": "pr3540-t341-vmware-65u2"
}
}

Usage of list_vcenter_networks script

./discover_networks.py -s 192.168.1.170 -u administrator@vsphere.local -p password -S
{
"count": 4,
"networks": [
{
"cluster": "p1-c1",
"host": "10.2.3.184",
"portgroup": "cloud.guest.409.200.1-vSwitch1",
"switch": "vSwitch1",
"virtualmachines": [
{
"device": {
"label": "Network adapter 1",
"macaddress": "02:00:37:b5:00:06"
},
"name": "i-2-8-VM"
},
{
"device": {
"label": "Network adapter 1",
"macaddress": "02:00:24:72:00:02"
},
"name": "r-4-VM"
}
],
"vlanid": "409"
},
{
"cluster": "p1-c1",
"host": "10.2.3.184",
"portgroup": "cloud.public.7.200.1-vSwitch1",
"switch": "vSwitch1",
"virtualmachines": [
{
"device": {
"label": "Network adapter 3",
"macaddress": "1e:00:6f:00:00:03"
},
"name": "r-4-VM"
}
],
"vlanid": "7"
},
{
"cluster": "p1-c1",
"host": "10.2.3.184",
"portgroup": "cloud.private.untagged.0.1-vSwitch0",
"switch": "vSwitch0",
"virtualmachines": [
{
"device": {
"label": "Network adapter 1",
"macaddress": "02:00:13:e1:00:01"
},
"name": "s-1-VM"
},
{
"device": {
"label": "Network adapter 2",
"macaddress": "1e:00:07:00:00:19"
},
"name": "s-1-VM"
},
{
"device": {
"label": "Network adapter 1",
"macaddress": "02:00:22:40:00:02"
},
"name": "v-2-VM"
},
{
"device": {
"label": "Network adapter 2",
"macaddress": "1e:00:a6:00:00:1a"
},
"name": "v-2-VM"
},
{
"device": {
"label": "Network adapter 2",
"macaddress": "02:00:72:27:00:04"
},
"name": "r-4-VM"
}
],
"vlanid": "0"
},
{
"cluster": "p1-c1",
"host": "10.2.3.184",
"portgroup": "cloud.public.7.0.1-vSwitch1",
"switch": "vSwitch1",
"virtualmachines": [
{
"device": {
"label": "Network adapter 3",
"macaddress": "1e:00:41:00:00:01"
},
"name": "s-1-VM"
},
{
"device": {
"label": "Network adapter 3",
"macaddress": "1e:00:a6:00:00:02"
},
"name": "v-2-VM"
},
{
"device": {
"label": "Network adapter 4",
"macaddress": "02:00:50:56:00:05"
},
"name": "r-4-VM"
},
{
"device": {
"label": "Network adapter 5",
"macaddress": "02:00:20:3d:00:06"
},
"name": "r-4-VM"
}
],
"vlanid": "7"
}
]
}

How Has This Been Tested?

A VMware added to running CloudStack with multiple hosts.
A new vitrual machine deployed from VMware vCenter with multiple disks, networks in the cluster.
At this point CloudStack does not show or control newly deployed virtual machine.
listUnmanagedInstances API is executed for the cluster. It shows newly deployed VM (deployed from vCenter)
Import API, importUnmanagedInstance, is executed, with appropriate service offering, network- VM NIC details and custom disk offerings for this unmanaged VM
On successful completion of import API, UI is verified showing imported VM. listVirtualMachines API is called to verify showing new VM and cloud database is checked for containing new records in vm_instance, volumes, nics tables.
Lifecycle tasks were performed on the imported VM - stop, start, reboot, migrate, attach new disk.

blueorangutan · 2019-09-24T15:55:07Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-288

yadvr · 2019-09-26T04:45:22Z

@shwstppr is there a doc PR or spec for this feature?
@blueorangutan package

blueorangutan · 2019-09-26T04:46:06Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-09-26T05:08:19Z

Packaging result: ✖centos6 ✖centos7 ✔debian. JID-294

yadvr · 2019-09-26T05:10:29Z

@blueorangutan package

blueorangutan · 2019-09-26T05:11:07Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-09-26T05:30:57Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-295

borisstoyanov · 2019-09-26T07:55:18Z

@blueorangutan package

blueorangutan · 2019-09-26T07:56:06Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-09-26T08:17:20Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-296

borisstoyanov · 2019-09-26T08:48:25Z

@blueorangutan test matrix

blueorangutan · 2019-09-26T08:49:07Z

@borisstoyanov a Trillian-Jenkins matrix job (centos7 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2019-09-26T19:31:29Z

Trillian test result (tid-386)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 36468 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3606-t386-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	28.15	test_scale_vm.py

shwstppr · 2019-09-27T10:47:15Z

@blueorangutan package

blueorangutan · 2019-09-27T10:48:12Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-09-27T11:08:04Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-300

…existing VMs in an infra. The feature currently works only for VMware, with a hypervisor agnostic framework which may be extended for KVM and XenServer in future. Two new APIs have been added, listUnmanagedInstances and importUnmanagedInstance. listUnmanagedInstances API will list all unmanaged virtual machine for a give cluster. Optionally, name for an existing unmanaged virtual machine can be given to retrieve VM details. API request params - clusterid(UUID of cluster) name(instance name) Response - clusterid hostid name osdisplayname memory powerstate cpuCoresPerSocket cpunumber cpuspeed disk - id - capacity - controller - controllerunit - imagepath - position nic - id - macaddress - networkname - vlanid - pcislot - ipaddress(Only with VMware Tools installed on VM for VMware) importUnmanagedInstance API will import an exisitng unmanaged virtual machine into CloudStack for a given cluster and virtual machine name. Service offering for the VM, disk offerings for volumes and networks for NICs of the VM can be mapped. Some optional parameters like projectid, domainid, hostname, details, etc can also be given. Appropritate networks, service offering and disk offerings need to be present before import and cannot be created on the fly during the API call. API request params - clusterid(UUID of cluster) name(instance name) displayname hostname domainid projectid templateid serviceofferingid diskofferingid(UUID of disk offering for root disk) nicnetworklist(Map for NIC ID and corresponding Network UUID) nicipaddresslist(Map for NIC ID and corresponding IP address) datadiskofferinglist(Map for disk ID and corresponding disk offering UUID) details(Map for VM details) migrateAllowed(VM and its volumes are allowed to migrate to different host/pool when offerings passed are incompatible with current host/pool) Response - Same response as that of deployVirtualMachine API A python 3 based script (scripts/vm/hypervisor/vmware/discover_networks.py) has been created to aid migrations. It leverages VMware’s pyvmomi library (https://github.com/vmware/pyvmomi) and allows listing all networks for a vCenter host or cluster. This script can take following arguments, -h, --help show this help message and exit -s HOST, --host HOST vSpehre service to connect to -o PORT, --port PORT Port to connect on -u USER, --user USER User name to use -p PASSWORD, --password PASSWORD Password to use -c CLUSTER, --cluster CLUSTER Cluster for which discover networks -S, --disable_ssl_verification Disable ssl host certificate verification -d, --debug Debug log messages Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

Check for NPE during nic network validation Filter orphan, removed volumes while list unmanaged VMs from VMware Publish usage and update resource count only after import complete Use disk size when disk offering for disk is of higher size durng import Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

borisstoyanov · 2020-01-20T08:34:20Z

@blueorangutan test

blueorangutan · 2020-01-20T08:34:41Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

shwstppr · 2020-01-20T13:48:06Z

@blueorangutan package

blueorangutan · 2020-01-20T13:48:42Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-01-20T14:20:07Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-631

blueorangutan · 2020-01-20T17:12:24Z

Trillian test result (tid-794)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29187 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3606-t794-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_vpn.py
Smoke tests completed. 78 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

borisstoyanov · 2020-01-21T09:15:53Z

@blueorangutan test

blueorangutan · 2020-01-21T09:16:45Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-01-21T18:39:26Z

Trillian test result (tid-802)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31800 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3606-t802-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Smoke tests completed. 78 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

yadvr · 2020-01-28T05:19:10Z

@DaanHoogland can you do a final review? After that squash-merge.

yadvr · 2020-01-31T10:09:11Z

@blueorangutan package

blueorangutan · 2020-01-31T10:09:30Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

DaanHoogland · 2020-01-31T10:45:19Z

@rhtyd I will do a final review and merge once travis and the bba agree

blueorangutan · 2020-01-31T11:18:42Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-728

DaanHoogland · 2020-01-31T12:46:27Z

@blueorangutan help

blueorangutan · 2020-01-31T12:46:29Z

@DaanHoogland I understand these words: "help", "hello", "thanks", "package", "test"
Test command usage: test [mgmt os] [hypervisor] [additional tests]
Mgmt OS options: ['centos6', 'centos7', 'ubuntu']
Hypervisor options: ['kvm-centos6', 'kvm-centos7', 'kvm-ubuntu', 'xenserver-71', 'xenserver-65sp1', 'xenserver-62sp1', 'vmware-65u2', 'vmware-60u2', 'vmware-55u3', 'vmware-51u1', 'vmware-50u1']
Additional tests: list of space separated tests with paths relative to the test/integration directory, for example: component/test_acl_listvm.py component/test_volumes.py
Note: when additional tests are passed, you need to specify mgmt server os and hypervisor or use the matrix command.

Blessed contributors for kicking Trillian test jobs: ['rhtyd', 'nvazquez', 'PaulAngus', 'borisstoyanov', 'DaanHoogland', 'shwstppr', 'andrijapanicsb', 'Spaceman1984', 'Pearl1594', 'davidjumani']

DaanHoogland · 2020-02-01T07:55:36Z

@blueorangutan test centos7 vmware-65u2

blueorangutan · 2020-02-01T07:56:33Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + vmware-65u2) has been kicked to run smoke tests

blueorangutan · 2020-02-01T17:56:34Z

Trillian test result (tid-874)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 34649 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3606-t874-vmware-65u2.zip
Intermittent failure detected: /marvin/tests/smoke/test_ssvm.py
Smoke tests completed. 79 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

shwstppr mentioned this pull request Sep 24, 2019
[WIP DO NOT MERGE] VM ingestion #3540
Closed
5 tasks

shwstppr force-pushed the feature-vm-ingestion branch 2 times, most recently from 5936345 to 0f8ae81Compare September 25, 2019 09:28

shwstppr force-pushed the feature-vm-ingestion branch from 0f8ae81 to 08fe2ebCompare September 26, 2019 05:43

svenvogel added type:new-feature component:vmware labels Sep 26, 2019

shwstppr force-pushed the feature-vm-ingestion branch from 08fe2eb to 7dab9e2Compare September 27, 2019 10:33

shwstppr force-pushed the feature-vm-ingestion branch 4 times, most recently from ccc1d29 to 0e32971Compare September 30, 2019 12:32

shwstppr added 2 commits October 1, 2019 14:26

shwstppr force-pushed the feature-vm-ingestion branch from 0e32971 to f9e3936Compare October 1, 2019 22:59

added null check for network broadcast uri
0ec876f
Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

yadvr approved these changes Jan 28, 2020
View reviewed changes

yadvr closed this Jan 29, 2020

yadvr reopened this Jan 29, 2020

DaanHoogland changed the title ~~[WIP DO NOT MERGE] VM ingestion~~VM ingestionJan 29, 2020

yadvr closed this Jan 31, 2020

yadvr reopened this Jan 31, 2020

apache deleted a comment from blueorangutan Feb 3, 2020

DaanHoogland merged commit 0f5b0e6 into apache:masterFeb 3, 2020

DaanHoogland deleted the feature-vm-ingestion branch February 3, 2020 14:43

Uh oh!

Conversation

shwstppr commented Sep 24, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

listUnmanagedInstances

importUnmanagedInstance

Types of changes

API sample calls:

How Has This Been Tested?

Uh oh!

blueorangutan commented Sep 24, 2019

Uh oh!

yadvr commented Sep 26, 2019

Uh oh!

blueorangutan commented Sep 26, 2019

Uh oh!

blueorangutan commented Sep 26, 2019

Uh oh!

yadvr commented Sep 26, 2019

Uh oh!

blueorangutan commented Sep 26, 2019

Uh oh!

blueorangutan commented Sep 26, 2019

Uh oh!

borisstoyanov commented Sep 26, 2019

Uh oh!

blueorangutan commented Sep 26, 2019

Uh oh!

blueorangutan commented Sep 26, 2019

Uh oh!

borisstoyanov commented Sep 26, 2019

Uh oh!

blueorangutan commented Sep 26, 2019

Uh oh!

blueorangutan commented Sep 26, 2019

Uh oh!

shwstppr commented Sep 27, 2019

Uh oh!

blueorangutan commented Sep 27, 2019

Uh oh!

blueorangutan commented Sep 27, 2019

Uh oh!

borisstoyanov commented Jan 20, 2020

Uh oh!

blueorangutan commented Jan 20, 2020

Uh oh!

shwstppr commented Jan 20, 2020

Uh oh!

blueorangutan commented Jan 20, 2020

Uh oh!

blueorangutan commented Jan 20, 2020

Uh oh!

blueorangutan commented Jan 20, 2020

Uh oh!

borisstoyanov commented Jan 21, 2020

Uh oh!

blueorangutan commented Jan 21, 2020

Uh oh!

blueorangutan commented Jan 21, 2020

Uh oh!

yadvr commented Jan 28, 2020

Uh oh!

yadvr commented Jan 31, 2020

Uh oh!

blueorangutan commented Jan 31, 2020

Uh oh!

DaanHoogland commented Jan 31, 2020

Uh oh!

blueorangutan commented Jan 31, 2020

Uh oh!

DaanHoogland commented Jan 31, 2020

Uh oh!

blueorangutan commented Jan 31, 2020

Uh oh!

DaanHoogland commented Feb 1, 2020

Uh oh!

blueorangutan commented Feb 1, 2020

Uh oh!

blueorangutan commented Feb 1, 2020

shwstppr commented Sep 24, 2019•
edited
Loading