Handle listProjects API to list projects with user as members when listAll=true by Pearl1594 · Pull Request #4316 · apache/cloudstack

Pearl1594 · 2020-09-10T07:06:54Z

Description

When listAll is true, listProjects API doesn't list projects that have users as members
Added defensive checks to overcome NPEs

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

How Has This Been Tested?

This has been verified using the cmk and UI
There exists 7 projects:

MariaDB [cloud]> select name, account_id, user_id, created, removed from project_view where removed is NULL;
+--------------+------------+---------+---------------------+---------+
| name | account_id | user_id | created | removed |
+--------------+------------+---------+---------------------+---------+
| Project2 | 4 | NULL | 2020-09-09 10:08:52 | NULL |
| Project2 | 5 | 6 | 2020-09-09 10:08:52 | NULL |
| p3 | 2 | NULL | 2020-09-09 11:43:50 | NULL |
| p3 | 5 | 5 | 2020-09-09 11:43:50 | NULL |
| Project1 | 2 | NULL | 2020-09-09 12:36:03 | NULL |
| Project1 | 5 | 5 | 2020-09-09 12:36:03 | NULL |
| p1 | 2 | 2 | 2020-09-09 13:16:11 | NULL |
| p1 | 5 | 5 | 2020-09-09 13:16:11 | NULL |
| d1 | 12 | 8 | 2020-09-09 13:21:38 | NULL |
| p1 | 2 | 2 | 2020-09-09 13:16:11 | NULL |
| p1 | 5 | 5 | 2020-09-09 13:16:11 | NULL |
| p4 | 15 | 10 | 2020-09-10 05:54:44 | NULL |
| p41useradmin | 17 | 12 | 2020-09-10 06:23:43 | NULL |
| Project1 | 2 | NULL | 2020-09-09 12:36:03 | NULL |
| Project1 | 5 | 5 | 2020-09-09 12:36:03 | NULL |
+--------------+------------+---------+---------------------+---------+

Before fix:

(localcloud) SBCM5> > list projects filter=account,owner,name,id, listall=true {
"count": 2,
"project": [
{
"id": "cab2a46a-8250-4825-85ff-e18e7275eb18",
"name": "p3",
"owner": [
{
"account": "admin"
}
]
},
{
"id": "f0e64573-3483-4e4c-8164-c92e9efda80f",
"name": "Project2",
"owner": [
{
"account": "ACSUser"
}
]
}
]
}

With fix:

(localcloud) SBCM5> > list projects filter=account,owner,name,id, listall=true {
"count": 7,
"project": [
{
"id": "0f4a6c4f-a95e-4ea2-b840-e16eedf1085a",
"name": "p41useradmin",
"owner": [
{
"account": "u4",
"user": "u41",
"userid": "9d523792-a777-45c2-8b34-84052b8d4b34"
}
]
},
{
"id": "352b9a95-cc6e-412f-ad1a-7671eaebfdff",
"name": "p4",
"owner": [
{
"account": "u3",
"user": "u3",
"userid": "41d2a5e2-f45f-479b-bc2b-3c911d2b7b46"
}
]
},
{
"id": "02b5bc7b-a1e7-4e77-860e-99fd5fb30ec3",
"name": "d1",
"owner": [
{
"account": "dom1",
"user": "dom1",
"userid": "a534379c-804d-4a45-bf09-73e784adacb0"
}
]
},
{
"id": "7ab34cd7-2403-477a-a3cf-9d03c517345c",
"name": "p1",
"owner": [
{
"account": "admin",
"user": "admin",
"userid": "e125c80a-eebf-11ea-ba61-1e006a013aee"
},
{
"account": "user1",
"user": "user1",
"userid": "705136fe-2c09-4031-873f-835ee3dfad02"
}
]
},
{
"id": "f5884fbe-9ce1-4315-9a76-c1564d13dac0",
"name": "Project1",
"owner": [
{
"account": "admin"
},
{
"account": "user1",
"user": "user1",
"userid": "705136fe-2c09-4031-873f-835ee3dfad02"
}
]
},
{
"id": "cab2a46a-8250-4825-85ff-e18e7275eb18",
"name": "p3",
"owner": [
{
"account": "admin"
}
]
},
{
"id": "f0e64573-3483-4e4c-8164-c92e9efda80f",
"name": "Project2",
"owner": [
{
"account": "ACSUser"
}
]
}
]
}

…st_project_users_filter

Pearl1594 · 2020-09-10T07:07:09Z

@blueorangutan package

blueorangutan · 2020-09-10T07:07:41Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

yadvr · 2020-09-10T07:20:23Z

 private boolean checkOperationPermitted(Account caller, ControlledEntity entity) {
 User user = CallContext.current().getCallingUser();
 Project project = projectDao.findByProjectAccountId(entity.getAccountId());
+ if (project == null) {


Are there usage of this method where throwing an exception would break flow? (or, are we expecting all usages of this method to be project related?)

this method is called only if the entity is owned by a project - which would mean that ideally project should be present if the entity's account is of Project type

yadvr · 2020-09-10T07:20:42Z

 if (owner.getType() != Account.ACCOUNT_TYPE_PROJECT && networkOwner.getType() == Account.ACCOUNT_TYPE_PROJECT) {
 User user = CallContext.current().getCallingUser();
 Project project = projectDao.findByProjectAccountId(network.getAccountId());
+ if (project == null) {


Same as previous comment

here too, the check is explicitly for networks owned by a project

blueorangutan · 2020-09-10T07:50:53Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-1950

yadvr · 2020-09-10T07:59:37Z

@blueorangutan test

blueorangutan · 2020-09-10T08:00:36Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-09-11T00:43:56Z

Trillian test result (tid-2717)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 58171 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4316-t2717-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_accounts.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vms_with_varied_deploymentplanners.py
Intermittent failure detected: /marvin/tests/smoke/test_deploy_vm_with_userdata.py
Intermittent failure detected: /marvin/tests/smoke/test_diagnostics.py
Intermittent failure detected: /marvin/tests/smoke/test_domain_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_internal_lb.py
Intermittent failure detected: /marvin/tests/smoke/test_iso.py
Intermittent failure detected: /marvin/tests/smoke/test_list_ids_parameter.py
Intermittent failure detected: /marvin/tests/smoke/test_loadbalance.py
Intermittent failure detected: /marvin/tests/smoke/test_metrics_api.py
Intermittent failure detected: /marvin/tests/smoke/test_multipleips_per_nic.py
Intermittent failure detected: /marvin/tests/smoke/test_nested_virtualization.py
Intermittent failure detected: /marvin/tests/smoke/test_network_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_network.py
Intermittent failure detected: /marvin/tests/smoke/test_password_server.py
Intermittent failure detected: /marvin/tests/smoke/test_portforwardingrules.py
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_projects.py
Intermittent failure detected: /marvin/tests/smoke/test_reset_vm_on_reboot.py
Intermittent failure detected: /marvin/tests/smoke/test_resource_accounting.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dns.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dnsservice.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_iptables_default_policy.py
Intermittent failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermittent failure detected: /marvin/tests/smoke/test_routers.py
Intermittent failure detected: /marvin/tests/smoke/test_service_offerings.py
Intermittent failure detected: /marvin/tests/smoke/test_snapshots.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 54 look OK, 31 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_deploy_vm_from_iso	`Error`	0.00	test_deploy_vm_iso.py
ContextSuite context=TestInternalLb>:setup	`Error`	0.00	test_internal_lb.py
ContextSuite context=TestDeployVmWithVariedPlanners>:setup	`Error`	0.00	test_deploy_vms_with_varied_deploymentplanners.py
test_01_1_create_iso_with_checksum_sha1_negative	`Error`	66.71	test_iso.py
test_01_create_iso_with_checksum_sha1	`Error`	65.59	test_iso.py
test_01_create_iso_with_checksum_sha1	`Error`	66.70	test_iso.py
test_02_1_create_iso_with_checksum_sha256_negative	`Error`	66.86	test_iso.py
test_02_create_iso_with_checksum_sha256	`Error`	65.61	test_iso.py
test_02_create_iso_with_checksum_sha256	`Error`	66.74	test_iso.py
test_03_1_create_iso_with_checksum_md5_negative	`Error`	66.67	test_iso.py
test_03_create_iso_with_checksum_md5	`Error`	65.66	test_iso.py
test_03_create_iso_with_checksum_md5	`Error`	66.79	test_iso.py
test_04_create_iso_with_no_checksum	`Error`	65.67	test_iso.py
test_04_create_iso_with_no_checksum	`Error`	66.76	test_iso.py
test_01_create_iso	`Failure`	1517.64	test_iso.py
ContextSuite context=TestISO>:setup	`Error`	3036.02	test_iso.py
ContextSuite context=TestDeployVmWithUserData>:setup	`Error`	0.00	test_deploy_vm_with_userdata.py
ContextSuite context=TestRemoteDiagnostics>:setup	`Error`	0.00	test_diagnostics.py
ContextSuite context=TestLoadBalance>:setup	`Error`	0.00	test_loadbalance.py
ContextSuite context=TestDomainsServiceOfferings>:setup	`Error`	2.91	test_domain_service_offerings.py
ContextSuite context=TestListIdsParams>:setup	`Error`	0.00	test_list_ids_parameter.py
test_list_vms_metrics	`Error`	0.28	test_metrics_api.py
test_nic_secondaryip_add_remove	`Error`	0.05	test_multipleips_per_nic.py
ContextSuite context=TestNestedVirtualization>:setup	`Error`	0.00	test_nested_virtualization.py
ContextSuite context=TestNetworkACL>:setup	`Error`	0.00	test_network_acl.py
test_delete_account	`Error`	0.47	test_network.py
test_delete_network_while_vm_on_it	`Error`	0.09	test_network.py
test_deploy_vm_l2network	`Error`	0.10	test_network.py
test_l2network_restart	`Error`	0.15	test_network.py
ContextSuite context=TestPortForwarding>:setup	`Error`	1.54	test_network.py
ContextSuite context=TestPublicIP>:setup	`Error`	1.06	test_network.py
test_reboot_router	`Error`	0.06	test_network.py
test_releaseIP	`Error`	0.48	test_network.py
ContextSuite context=TestRouterRules>:setup	`Error`	0.58	test_network.py
ContextSuite context=TestIsolatedNetworksPasswdServer>:setup	`Error`	0.00	test_password_server.py
ContextSuite context=TestResetVmOnReboot>:setup	`Error`	0.00	test_reset_vm_on_reboot.py
ContextSuite context=TestPortForwardingRules>:setup	`Error`	0.00	test_portforwardingrules.py
ContextSuite context=TestPrivateGwACL>:setup	`Error`	0.00	test_privategw_acl.py
test_09_project_suspend	`Error`	0.10	test_projects.py
test_10_project_activation	`Error`	0.09	test_projects.py
ContextSuite context=TestRAMCPUResourceAccounting>:setup	`Error`	0.00	test_resource_accounting.py
ContextSuite context=TestRouterDHCPHosts>:setup	`Error`	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDHCPOpts>:setup	`Error`	0.00	test_router_dhcphosts.py
ContextSuite context=TestRouterDns>:setup	`Error`	0.00	test_router_dns.py
ContextSuite context=TestRouterDnsService>:setup	`Error`	0.00	test_router_dnsservice.py
ContextSuite context=TestRouterIpTablesPolicies>:setup	`Error`	0.00	test_routers_iptables_default_policy.py
ContextSuite context=TestVPCIpTablesPolicies>:setup	`Error`	0.00	test_routers_iptables_default_policy.py
test_01_isolate_network_FW_PF_default_routes_egress_true	`Error`	0.21	test_routers_network_ops.py
test_02_isolate_network_FW_PF_default_routes_egress_false	`Error`	0.28	test_routers_network_ops.py
ContextSuite context=TestRedundantIsolateNetworks>:setup	`Error`	0.87	test_routers_network_ops.py
ContextSuite context=TestRouterServices>:setup	`Error`	0.00	test_routers.py
ContextSuite context=TestCpuCapServiceOfferings>:setup	`Error`	0.00	test_service_offerings.py
ContextSuite context=TestServiceOfferings>:setup	`Error`	3.20	test_service_offerings.py
ContextSuite context=TestSnapshotRootDisk>:setup	`Error`	0.00	test_snapshots.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	340.68	test_vpc_redundant.py
test_hostha_kvm_host_fencing	`Error`	177.66	test_hostha_kvm.py

…account in response

Pearl1594 · 2020-09-11T03:50:19Z

@blueorangutan package

blueorangutan · 2020-09-11T03:51:45Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-09-11T04:22:18Z

Packaging result: ✔centos7 ✔centos8 ✖debian. JID-1953

Pearl1594 · 2020-09-11T04:29:02Z

@blueorangutan package

blueorangutan · 2020-09-11T04:29:39Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-09-11T04:56:54Z

Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1954

Pearl1594 · 2020-09-11T04:59:06Z

@blueorangutan package

blueorangutan · 2020-09-11T04:59:45Z

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-09-11T05:29:35Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-1955

Pearl1594 · 2020-09-11T05:31:12Z

@blueorangutan test

blueorangutan · 2020-09-11T05:32:40Z

@Pearl1594 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-09-12T02:53:31Z

Trillian test result (tid-2718)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 74705 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4316-t2718-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermittent failure detected: /marvin/tests/smoke/test_pvlan.py
Intermittent failure detected: /marvin/tests/smoke/test_router_dhcphosts.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 81 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_02_vpc_privategw_static_routes	`Failure`	253.21	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	338.73	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	501.95	test_privategw_acl.py
test_create_pvlan_network	`Error`	0.20	test_pvlan.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Failure`	541.83	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	3813.82	test_vpc_redundant.py
test_05_rvpc_multi_tiers	`Error`	3972.65	test_vpc_redundant.py
test_hostha_kvm_host_fencing	`Error`	191.82	test_hostha_kvm.py

yadvr · 2020-09-13T10:49:52Z

@blueorangutan package

blueorangutan · 2020-09-13T10:50:45Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-09-13T11:09:03Z

Packaging result: ✖centos7 ✖centos8 ✖debian. JID-1985

yadvr · 2020-09-15T07:54:35Z

@blueorangutan package

blueorangutan · 2020-09-15T07:55:34Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-09-15T08:35:04Z

Packaging result: ✖centos7 ✖centos8 ✖debian. JID-2009

yadvr · 2020-09-15T09:17:58Z

@blueorangutan package

blueorangutan · 2020-09-15T09:19:40Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2020-09-15T10:00:38Z

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2013

yadvr · 2020-09-16T07:40:12Z

@blueorangutan test

blueorangutan · 2020-09-16T07:41:20Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2020-09-16T22:23:18Z

Trillian test result (tid-2754)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 50934 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4316-t2754-kvm-centos7.zip
Intermittent failure detected: /marvin/tests/smoke/test_kubernetes_clusters.py
Intermittent failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermittent failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 82 look OK, 3 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_01_deploy_kubernetes_cluster	`Error`	0.08	test_kubernetes_clusters.py
test_02_deploy_kubernetes_ha_cluster	`Error`	0.04	test_kubernetes_clusters.py
test_04_deploy_and_upgrade_kubernetes_cluster	`Error`	0.04	test_kubernetes_clusters.py
test_05_deploy_and_upgrade_kubernetes_ha_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_06_deploy_and_invalid_upgrade_kubernetes_cluster	`Error`	0.03	test_kubernetes_clusters.py
test_07_deploy_and_scale_kubernetes_cluster	`Error`	0.04	test_kubernetes_clusters.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Failure`	433.15	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Failure`	436.55	test_vpc_redundant.py
test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers	`Error`	432.88	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Error`	3856.49	test_vpc_redundant.py
test_hostha_kvm_host_fencing	`Error`	98.32	test_hostha_kvm.py

shwstppr

LGTM

yadvr · 2020-09-17T04:50:09Z

Lgtm, test failures unrelated

Pearl Dsilva added 2 commits September 10, 2020 11:56

added defensive checks for avoiding NPE and list projects API fix
61f2dc2

Merge branch 'master' ofhttps://github.com/apache/cloudstackinto li…
b9443fd
…st_project_users_filter

yadvr reviewed Sep 10, 2020
View reviewed changes

yadvr approved these changes Sep 10, 2020
View reviewed changes

yadvr added this to the 4.15.0.0 milestone Sep 10, 2020

Pearl1594 changed the title ~~Handle listProjects API to list projects with user as Project Admin when listAll=true~~Handle listProjects API to list projects with user as members when listAll=trueSep 10, 2020

list projects with account name provided to not include users in the …
63c516a
…account in response

shwstppr approved these changes Sep 17, 2020
View reviewed changes

yadvr merged commit 82b6971 into apache:masterSep 17, 2020

DaanHoogland added the type:bug label Oct 21, 2020

DaanHoogland added the Severity:Minor label Nov 10, 2020

Pearl1594 mentioned this pull request Nov 16, 2020
Fix: Listing projects comprising of only the user's on listAll=true #4469
Merged
5 tasks

Uh oh!

Conversation

Pearl1594 commented Sep 10, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

How Has This Been Tested?

Uh oh!

Pearl1594 commented Sep 10, 2020

Uh oh!

blueorangutan commented Sep 10, 2020

Uh oh!

yadvrSep 10, 2020

Choose a reason for hiding this comment

Uh oh!

Pearl1594Sep 10, 2020

Choose a reason for hiding this comment

Uh oh!

yadvrSep 10, 2020

Choose a reason for hiding this comment

Uh oh!

Pearl1594Sep 10, 2020

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Sep 10, 2020

Uh oh!

yadvr commented Sep 10, 2020

Uh oh!

blueorangutan commented Sep 10, 2020

Uh oh!

blueorangutan commented Sep 11, 2020

Uh oh!

Pearl1594 commented Sep 11, 2020

Uh oh!

blueorangutan commented Sep 11, 2020

Uh oh!

blueorangutan commented Sep 11, 2020

Uh oh!

Pearl1594 commented Sep 11, 2020

Uh oh!

blueorangutan commented Sep 11, 2020

Uh oh!

blueorangutan commented Sep 11, 2020

Uh oh!

Pearl1594 commented Sep 11, 2020

Uh oh!

blueorangutan commented Sep 11, 2020

Uh oh!

blueorangutan commented Sep 11, 2020

Uh oh!

Pearl1594 commented Sep 11, 2020

Uh oh!

blueorangutan commented Sep 11, 2020

Uh oh!

blueorangutan commented Sep 12, 2020

Uh oh!

yadvr commented Sep 13, 2020

Uh oh!

blueorangutan commented Sep 13, 2020

Uh oh!

blueorangutan commented Sep 13, 2020

Uh oh!

yadvr commented Sep 15, 2020

Uh oh!

blueorangutan commented Sep 15, 2020

Uh oh!

blueorangutan commented Sep 15, 2020

Uh oh!

yadvr commented Sep 15, 2020

Uh oh!

blueorangutan commented Sep 15, 2020

Uh oh!

blueorangutan commented Sep 15, 2020

Uh oh!

yadvr commented Sep 16, 2020

Uh oh!

blueorangutan commented Sep 16, 2020

Uh oh!

blueorangutan commented Sep 16, 2020

Uh oh!

Pearl1594 commented Sep 10, 2020•
edited
Loading