[WIP DO NOT MERGE] VM ingestion by shwstppr · Pull Request #3540 · apache/cloudstack

shwstppr · 2019-08-05T05:16:37Z

Description

VM ingestion feature allows CloudStack to discover, on-board, import existing VMs in an infra. The feature currently works only for VMware, with a hypervisor agnostic framework which may be extended for KVM and XenServer in future.
Two new APIs have been added, listUnmanagedInstances and importUnmanagedInstance.

listUnmanagedInstances

API will list all unmanaged virtual machine for a give cluster. Optionally, name for an existing unmanaged virtual machine can be given to retrieve VM details.
API request params -
clusterid(UUID of cluster)
name(instance name)
Response -
clusterid
hostid
name
osdisplayname
memory
powerstate
cpuCoresPerSocket
cpunumber
cpuspeed
disk
- id
- capacity
- controller
- controllerunit
- imagepath
- position
nic
- id
- macaddress
- networkname
- vlanid
- pcislot

importUnmanagedInstance

API will import an exisitng unmanaged virtual machine into CloudStack for a given cluster and virtual machine name. Service offering for the VM, disk offerings for volumes and networks for NICs of the VM can be mapped. Some optional parameters like projectid, domainid, hostname, details, etc can also be given. Appropritate networks, service offering and disk offerings need to be present before import and cannot be created on the fly during the API call.
API request params -
clusterid(UUID of cluster)
name(instance name)
displayname
hostname
domainid
projectid
templateid
serviceofferingid
diskofferingid(UUID of disk offering for root disk)
nicnetworklist(Map for NIC ID and corresponding Network UUID)
datadiskofferinglist(Map for disk ID and corresponding disk offering UUID)
details(Map for VM details)
Response -
Same response as that of deployVirtualMachine API

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

API sample calls:

List

> list unmanagedinstances clusterid=079a19c2-2710-4ed6-9110-027dd5e0f262 {
"count": 2,
"unmanagedinstance": [
{
"clusterid": "079a19c2-2710-4ed6-9110-027dd5e0f262",
"cpucorespersocket": 1,
"cpunumber": 1,
"cpuspeed": 0,
"disk": [
{
"capacity": 2097152,
"controller": "ide",
"controllerunit": 0,
"id": "51-3001",
"imagepath": "[8aca30d6f7803420973f01cc91c1c011] test-unmanaged/test-unmanaged_64.vmdk",
"label": "Hard disk 1",
"position": 1
},
{
"capacity": 5242880,
"controller": "scsi",
"controllerunit": 0,
"id": "51-2000",
"imagepath": "[8aca30d6f7803420973f01cc91c1c011] test-unmanaged/test-unmanaged_63.vmdk",
"label": "Hard disk 2",
"position": 0
}
],
"hostid": "4ab4147a-7e40-4abe-a0e2-2fac1aa6f0bf",
"memory": 512,
"name": "test-unmanaged",
"nic": [
{
"id": "Network adapter 1",
"macaddress": "02:00:39:7c:00:24",
"networkname": "cloud.guest.1212.200.1-vSwitch1",
"vlanid": 1212
}
],
"osdisplayname": "CentOS 4/5 or later (64-bit)",
"powerstate": "POWERED_ON"
}
]
}

Import

> import unmanagedinstance name=test-unmanaged templateid=200f3fe9-c845-11e9-b69f-1e003c0107c4 clusterid=079a19c2-2710-4ed6-9110-027dd5e0f262 diskofferingid=e9eb069f-2946-4e26-8d81-07aebba868d1 serviceofferingid=c4c5b9e8-8a4a-422d-bccc-2f1bcc166c59 datadiskofferinglist[0].disk=51-2000 datadiskofferinglist[0].diskOffering=e9eb069f-2946-4e26-8d81-07aebba868d1 displayname=NewVm nicNetworkList[0].nic="Network adapter 1" nicNetworkList[0].network=4c44312b-f009-4d33-b942-989169f3d2fc
{
"virtualmachine": {
"account": "admin",
"affinitygroup": [],
"cpunumber": 1,
"cpuspeed": 500,
"created": "2019-09-05T11:32:04+0000",
"details": {},
"diskofferingid": "e9eb069f-2946-4e26-8d81-07aebba868d1",
"diskofferingname": "Custom",
"displayname": "NewVm",
"displayvm": true,
"domain": "ROOT",
"domainid": "20088b4d-c845-11e9-b69f-1e003c0107c4",
"guestosid": "2015cd08-c845-11e9-b69f-1e003c0107c4",
"haenable": false,
"hostid": "4ab4147a-7e40-4abe-a0e2-2fac1aa6f0bf",
"hostname": "10.2.2.144",
"hypervisor": "VMware",
"id": "4fda861e-e693-4227-a8aa-1b4ce90d5041",
"instancename": "test-unmanaged",
"isdynamicallyscalable": false,
"memory": 512,
"name": "test-unmanaged",
"nic": [
{
"broadcasturi": "vlan://1212",
"extradhcpoption": [],
"id": "61952f25-7efa-4bc2-aa92-ecd38c122e65",
"isdefault": true,
"isolationuri": "vlan://1212",
"macaddress": "02:00:39:7c:00:24",
"networkid": "4c44312b-f009-4d33-b942-989169f3d2fc",
"networkname": "lol",
"secondaryip": [],
"traffictype": "Guest",
"type": "L2"
}
],
"ostypeid": "2015cd08-c845-11e9-b69f-1e003c0107c4",
"passwordenabled": false,
"rootdeviceid": 0,
"rootdevicetype": "ROOT",
"securitygroup": [],
"serviceofferingid": "c4c5b9e8-8a4a-422d-bccc-2f1bcc166c59",
"serviceofferingname": "Small Instance",
"state": "Stopped",
"tags": [],
"templatedisplaytext": "CentOS 5.3(64-bit) no GUI (vSphere)",
"templateid": "200f3fe9-c845-11e9-b69f-1e003c0107c4",
"templatename": "CentOS 5.3(64-bit) no GUI (vSphere)",
"userid": "4dfc315c-c845-11e9-b69f-1e003c0107c4",
"username": "admin",
"zoneid": "db18d224-279d-4c08-aeef-0cb7030d0edb",
"zonename": "pr3540-t341-vmware-65u2"
}
}importUnmanagedInstance

How Has This Been Tested?

A VMware added to running CloudStack with multiple hosts.
A new vitrual machine deployed from VMware vCenter with multiple disks, networks in the cluster.
At this point CloudStack does not show or control newly deployed virtual machine.
listUnmanagedInstances API is executed for the cluster. It shows newly deployed VM (deployed from vCenter)
Import API, importUnmanagedInstance, is executed, with appropriate service offering, network- VM NIC details and custom disk offerings for this unmanaged VM
On successful completion of import API, UI is verified showing imported VM. listVirtualMachines API is called to verify showing new VM and cloud database is checked for containing new records in vm_instance, volumes, nics tables.
Lifecycle tasks were performed on the imported VM - stop, start, reboot, migrate, attach new disk.

shwstppr · 2019-08-05T05:17:23Z

@blueorangutan package

blueorangutan · 2019-08-05T05:18:08Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-08-05T05:32:21Z

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-213

blueorangutan · 2019-08-05T06:26:51Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-214

blueorangutan · 2019-08-05T12:40:05Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-216

blueorangutan · 2019-08-05T17:14:48Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-218

blueorangutan · 2019-08-05T20:44:20Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-221

blueorangutan · 2019-08-05T21:16:43Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-222

blueorangutan · 2019-08-06T07:07:53Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-224

blueorangutan · 2019-08-06T08:48:41Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-225

blueorangutan · 2019-08-06T09:39:42Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-226

blueorangutan · 2019-08-06T11:18:18Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-227

blueorangutan · 2019-08-06T12:11:49Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-228

blueorangutan · 2019-08-07T10:37:20Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-229

blueorangutan · 2019-08-09T10:21:29Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-237

blueorangutan · 2019-08-09T11:59:30Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-239

blueorangutan · 2019-08-09T13:00:53Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-240

blueorangutan · 2019-08-09T20:42:12Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-241

blueorangutan · 2019-08-09T22:02:52Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-242

blueorangutan · 2019-08-09T23:01:28Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-243

blueorangutan · 2019-08-13T06:43:36Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-249

blueorangutan · 2019-08-13T09:04:55Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-251

blueorangutan · 2019-08-26T20:54:36Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-265

Signed-off-by: Paul Angus <paul.angus@shapeblue.com>

shwstppr · 2019-09-06T06:12:10Z

@blueorangutan package

blueorangutan · 2019-09-06T06:12:35Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

borisstoyanov · 2019-09-11T18:45:59Z

@blueorangutan test matrix

blueorangutan · 2019-09-11T18:47:00Z

@borisstoyanov a Trillian-Jenkins matrix job (centos6 mgmt + xs71, centos7 mgmt + vmware65, centos7 mgmt + kvmcentos7) has been kicked to run smoke tests

blueorangutan · 2019-09-12T04:12:35Z

Trillian test result (tid-350)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 32125 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3540-t350-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_05_rvpc_multi_tiers	`Failure`	416.14	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	442.54	test_vpc_redundant.py

blueorangutan · 2019-09-12T05:06:03Z

Trillian test result (tid-351)
Environment: vmware-65u2 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35456 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3540-t351-vmware-65u2.zip
Smoke tests completed. 77 look OK, 0 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File

blueorangutan · 2019-09-12T05:14:54Z

Trillian test result (tid-353)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35082 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3540-t353-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	23.94	test_scale_vm.py

borisstoyanov

@shwstppr can you add unit tests please

shwstppr · 2019-09-13T18:06:33Z

@blueorangutan package

blueorangutan · 2019-09-13T18:07:10Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2019-09-13T18:28:31Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-276

blueorangutan · 2019-09-17T19:26:54Z

Trillian test result (tid-365)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35662 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr3540-t365-xenserver-71.zip
Intermittent failure detected: /marvin/tests/smoke/test_scale_vm.py
Smoke tests completed. 76 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_scale_vm	`Failure`	24.86	test_scale_vm.py

blueorangutan · 2019-09-18T07:25:03Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-279

shwstppr · 2019-09-20T05:37:18Z

@blueorangutan package

blueorangutan · 2019-09-20T05:37:39Z

@shwstppr a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

shwstppr · 2019-09-20T05:37:40Z

@shwstppr can you add unit tests please

added tests

blueorangutan · 2019-09-20T05:57:47Z

Packaging result: ✖centos6 ✔centos7 ✔debian. JID-284

blueorangutan · 2019-09-24T11:18:44Z

Packaging result: ✖centos6 ✖centos7 ✖debian. JID-287

…existing VMs in an infra. The feature currently works only for VMware, with a hypervisor agnostic framework which may be extended for KVM and XenServer in future. Two new APIs have been added, listUnmanagedInstances and importUnmanagedInstance. listUnmanagedInstances API will list all unmanaged virtual machine for a give cluster. Optionally, name for an existing unmanaged virtual machine can be given to retrieve VM details. API request params - clusterid(UUID of cluster) name(instance name) Response - clusterid hostid name osdisplayname memory powerstate cpuCoresPerSocket cpunumber cpuspeed disk - id - capacity - controller - controllerunit - imagepath - position nic - id - macaddress - networkname - vlanid - pcislot importUnmanagedInstance API will import an exisitng unmanaged virtual machine into CloudStack for a given cluster and virtual machine name. Service offering for the VM, disk offerings for volumes and networks for NICs of the VM can be mapped. Some optional parameters like projectid, domainid, hostname, details, etc can also be given. Appropritate networks, service offering and disk offerings need to be present before import and cannot be created on the fly during the API call. API request params - clusterid(UUID of cluster) name(instance name) displayname hostname domainid projectid templateid serviceofferingid diskofferingid(UUID of disk offering for root disk) nicnetworklist(Map for NIC ID and corresponding Network UUID) datadiskofferinglist(Map for disk ID and corresponding disk offering UUID) details(Map for VM details) Response - Same response as that of deployVirtualMachine API Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>

yadvr · 2019-09-24T11:44:14Z

@shwstppr a lot of conflicts here, can you rebase or fix conflict?

shwstppr · 2019-09-24T15:32:59Z

Moved here, #3606

yadvr added the status:work-in-progress label Aug 5, 2019

shwstppr force-pushed the vm-ingestion branch from 504a36e to 267f252Compare August 5, 2019 06:05

svenvogel added type:new-feature component:vmware labels Aug 6, 2019

Updating pom.xml version numbers for release 4.13.0.0
7c7efe7
Signed-off-by: Paul Angus <paul.angus@shapeblue.com>

borisstoyanov requested review from DaanHoogland and yadvr and removed request for DaanHoogland September 11, 2019 18:44

borisstoyanov added this to the 4.14.0.0 milestone Sep 11, 2019

borisstoyanov assigned borisstoyanov and shwstppr Sep 11, 2019

borisstoyanov requested a review from DaanHoogland September 11, 2019 18:45

borisstoyanov requested changes Sep 12, 2019
View reviewed changes

shwstppr force-pushed the vm-ingestion branch from ef5271b to f0d478fCompare September 24, 2019 11:13

shwstppr force-pushed the vm-ingestion branch from bb18dd0 to e0819daCompare September 24, 2019 11:27

shwstppr force-pushed the vm-ingestion branch from e0819da to 0549c44Compare September 24, 2019 11:43

shwstppr closed this Sep 24, 2019

Uh oh!

Conversation

shwstppr commented Aug 5, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

listUnmanagedInstances

importUnmanagedInstance

Types of changes

API sample calls:

How Has This Been Tested?

Uh oh!

shwstppr commented Aug 5, 2019

Uh oh!

blueorangutan commented Aug 5, 2019

Uh oh!

blueorangutan commented Aug 5, 2019

Uh oh!

blueorangutan commented Aug 5, 2019

Uh oh!

blueorangutan commented Aug 5, 2019

Uh oh!

blueorangutan commented Aug 5, 2019

Uh oh!

blueorangutan commented Aug 5, 2019

Uh oh!

blueorangutan commented Aug 5, 2019

Uh oh!

blueorangutan commented Aug 6, 2019

Uh oh!

blueorangutan commented Aug 6, 2019

Uh oh!

blueorangutan commented Aug 6, 2019

Uh oh!

blueorangutan commented Aug 6, 2019

Uh oh!

blueorangutan commented Aug 6, 2019

Uh oh!

blueorangutan commented Aug 7, 2019

Uh oh!

blueorangutan commented Aug 9, 2019

Uh oh!

blueorangutan commented Aug 9, 2019

Uh oh!

blueorangutan commented Aug 9, 2019

Uh oh!

blueorangutan commented Aug 9, 2019

Uh oh!

blueorangutan commented Aug 9, 2019

Uh oh!

blueorangutan commented Aug 9, 2019

Uh oh!

blueorangutan commented Aug 13, 2019

Uh oh!

blueorangutan commented Aug 13, 2019

Uh oh!

blueorangutan commented Aug 26, 2019

Uh oh!

shwstppr commented Sep 6, 2019

Uh oh!

blueorangutan commented Sep 6, 2019

Uh oh!

borisstoyanov commented Sep 11, 2019

Uh oh!

blueorangutan commented Sep 11, 2019

Uh oh!

blueorangutan commented Sep 12, 2019

Uh oh!

blueorangutan commented Sep 12, 2019

Uh oh!

blueorangutan commented Sep 12, 2019

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

shwstppr commented Sep 13, 2019

Uh oh!

blueorangutan commented Sep 13, 2019

Uh oh!

blueorangutan commented Sep 13, 2019

Uh oh!

shwstppr commented Aug 5, 2019•
edited
Loading