API key pair restructure by KlausDornsbach · Pull Request #9504 · apache/cloudstack

KlausDornsbach · 2024-08-08T15:04:39Z

Description

API access keypairs are primarily used to support interactions between systems, without the need to create sessions (through user and password authentication). Currently, CloudStack's implementation of API keypairs does not allow you to specify permissions for each keypair, simply using the account's default permissions. Additionally, the number of keypairs is limited to one per user and they have no start and end dates.

An extension of the API keypairs functionality was implemented, adding several new features that increase flexibility and security. It is now possible to specify a subset of permissions (from the base account) for each keypair, as well as create more than one key per user. It is also possible to define start and end dates for the validity of a keypair. A key created without an expiration date will always be valid up until it is deleted. It should be noted that creating API keypairs without specifying permissions just creates an API keypair with all account's base permissions. Also, API keypairs older than this patch will always be viewed as keypairs with full account permissions.

The following endpoints were created:

A new listUserKeys API was added. Through this API the user will be able to specify a single keypairid to fetch its specific properties, or apikeyfilter to return a specific keypair based on an apikey. The user can inform an userid to fetch an user's api keypair list. If no keypairid, apikeyfilter or userid is provided, the API defaults to fetching information on the calling user. The listall property allows for fetching all keypairs in the structure that are visible based on the calling user/keypair permissions, if not specified, it defaults to false, fetching only the apikeys on the level of the calling user/keypair. Also, it is possible to inform showpermissions to list all permissions associated with each returned apikey.

Name	Description	Required	Default
userid	id of the owner of the keypairs	no	none
keypairid	id of the keypair	no	none
listall	list all accessable keypairs	no	false
apikeyfilter	apikey of they keypair	no	none
showpermissions	lists all associated apikey permissions	no	false

The API getUserKeys was modified preserving backwards compatibility. It now fetches the last keypair created for the informed user.

The api registerUserKeys was modified so the new API keypair parameters could be specified on creation:

Name	Description	Required	Example	Default
id	user id	Yes	b8914774-771e-11ee-8e59-5254003754dc	none
name	name of the keypair	No	MyKey	userId + " - API Keypair"
startdate	date when key becomes valid	No	2024-04-09	none
enddate	date when key expires	No	2024-04-09	never expires
description	keypair description	No	read only permissions	none
rules	list of API access rules	No	rules[1].rule=list* rules[1].permission=allow	all user API permissions based on Account Role

A new keypair deletion API was added (deleteUserKeys). It will accept only one required argument, the keypair id.

Name	Description	Required
keypairid	id of the keypair	yes

I also added a listUserKeyRules api, allowing the user to list the rules associated with an API keypair.

Name	Description	Required
keypairid	id of the keypair	yes

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

How Has This Been Tested?

API Key Creation and Basic Testing

Single Key (via UI):

I was able to create API keypairs through the UI through the button on the top right of the user view;
Through Cloud Monkey, validated that the keypair had the same permissions as the base user by calling a series of APIs;

Multiple Keys (via Cloud Monkey):

Created multiple keys;
Validated the operation was successful on the DB;
Tested creating a key that was not valid and would become valid in the future, with success;
Tested creating a key that was valid and would become invalid in the future, with success;
Tested trying to create keys that were already invalid and got errors;
All permissions of the API key pairs were consistent with each key pair tested.

Permissions Validation

Tested the permissions of keyrules listing, keypair listing, keypair deletion and keypair cretion with the following user/account/domain setup:

domain /ROOT
- account root admin
  - user root admin
  - user user1
- domain subdomain
  - account domain admin
    - user domain admin
  - account userAccount
    - user user2
    - user user3

The following table describes the results obtained when the user on the first column attempted to operate on the keypairs of users on the first row (V: operation was possible, F: operation was not possible).

user	rootAdm	user1	domainAdm	user2	user3
rootAdm	V	V	V	V	V
user1	V	V	V	V	V
domainAdm	F	F	V	V	V
user2	F	F	F	V	F
user3	F	F	F	F	V

Migration to `api_keypair` Table

A key was created for a read-only user
The database was updated from version 4.19 to 4.20
The API key data was successfully migrated to the api_keypair table, with the corresponding columns in the user table deleted.
Confirmed correct permission handling of the api key.

General validations

Could not create an API keypair with more permissions than the base keypair.
Deleting system keys was not possible.
After deleting a user or account, the API keypairs were invalidated.

codecov · 2024-08-12T13:09:52Z

Codecov Report

❌ Patch coverage is 16.50738% with 961 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.92%. Comparing base (7ad68aa) to head (b6b1b5f).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...c/main/java/com/cloud/user/AccountManagerImpl.java	31.80%	182 Missing and 11 partials ⚠️
...he/cloudstack/api/response/ApiKeyPairResponse.java	0.00%	127 Missing ⚠️
...ck/api/command/admin/user/RegisterUserKeysCmd.java	0.00%	94 Missing ⚠️
...n/java/org/apache/cloudstack/acl/ApiKeyPairVO.java	25.60%	93 Missing ⚠️
...src/main/java/com/cloud/api/ApiResponseHelper.java	0.00%	61 Missing ⚠️
...g/apache/cloudstack/acl/dao/ApiKeyPairDaoImpl.java	0.00%	43 Missing ⚠️
...dstack/api/command/admin/user/ListUserKeysCmd.java	0.00%	35 Missing ⚠️
...g/apache/cloudstack/acl/ApiKeyPairManagerImpl.java	0.00%	33 Missing ⚠️
...tack/api/command/admin/user/DeleteUserKeysCmd.java	0.00%	30 Missing ⚠️
.../cloudstack/discovery/ApiDiscoveryServiceImpl.java	14.28%	28 Missing and 2 partials ⚠️
... and 31 more

Additional details and impacted files

@@ Coverage Diff @@## main #9504 +/- ##
==========================================
Coverage 17.92% 17.92% - Complexity 16156 16176 +20 
==========================================
Files 5939 5949 +10 Lines 533192 534058 +866 Branches 65239 65301 +62 ==========================================
+ Hits 95591 95748 +157 - Misses 426859 427553 +694 - Partials 10742 10757 +15

Flag	Coverage Δ
uitests	`3.66% <ø> (ø)`
unittests	`19.03% <16.50%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

DaanHoogland · 2024-08-13T06:59:57Z

nice feature @KlausDornsbach , some suggestions,

would it make sense to be able to delete a kay based on name?
also is admin allowed to delete keys for other users? (would make sense from a maintainance point of view, would it?)

KlausDornsbach · 2024-08-14T21:36:15Z

Hey @DaanHoogland, thanks for taking a look!

It would make sense to be able to delete a keypair by name, we would just need to block users from creating multiple API keypairs with the same name.

At the moment an admin is allowed to delete keypairs from users it has access, for example, a root admin user could delete any keypair in the platform, domain admin users can delete any keypair in the domain, normal users can only delete their own keys. These permissions are also true for visualization and creation APIs.

DaanHoogland · 2024-08-16T07:38:35Z

It would make sense to be able to delete a keypair by name, we would just need to block users from creating multiple API keypairs with the same name.

Well, I think a unique constraint on UserId/KeyPairName makes sense also from a usability sense.

At the moment an admin is allowed to delete keypairs from users it has access, for example, a root admin user could delete any keypair in the platform, domain admin users can delete any keypair in the domain, normal users can only delete their own keys. These permissions are also true for visualization and creation APIs.

👍

rajujith · 2024-08-19T12:04:24Z

@blueorangutan package

github-actions · 2024-08-20T10:54:33Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland · 2024-08-20T14:27:45Z

@blueorangutan package

blueorangutan · 2024-08-20T14:30:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-08-20T15:00:10Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10714

rajujith · 2024-08-21T04:21:38Z

@blueorangutan package

blueorangutan · 2024-08-21T04:22:03Z

@rajujith a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-08-21T04:49:20Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10720

rajujith · 2024-08-22T04:29:16Z

@blueorangutan package

blueorangutan · 2024-08-22T04:30:03Z

@rajujith a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-08-22T05:36:21Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10736

DaanHoogland · 2024-08-22T12:52:49Z

@blueorangutan test

GutoVeronezi · 2026-01-15T14:36:31Z

good functionality and looks good so far, however, this should go into a non-LTS first and only after extensive testing. it is a cross cutting concern and hence brings risks adherent to such code

I totally agree with you; LTS are supposed to be as stable as possible. We should also apply the same criteria to every other PR with extensive or critical changes as well 😃

DaanHoogland · 2026-01-15T16:44:08Z

good functionality and looks good so far, however, this should go into a non-LTS first and only after extensive testing. it is a cross cutting concern and hence brings risks adherent to such code
I totally agree with you; LTS are supposed to be as stable as possible. We should also apply the same criteria to every other PR with extensive or critical changes as well 😃

I was kind of inviting this to 23, but yes, if it is a cross-cutting concern it should go in the non-LTS release first. Extensive or critical are gliding scales (and maybe cross-cutting as well, but certainly less so)

GutoVeronezi · 2026-01-15T17:30:38Z

I was kind of inviting this to 23, but yes, if it is a cross-cutting concern it should go in the non-LTS release first. Extensive or critical are gliding scales (and maybe cross-cutting as well, but certainly less so)

I was under impression that 4.23 would be a regular release (the wiki states so);

Thus, it would make sense to address this PR on release 4.23. 5 months seems enough time to review, test, and mature the PR. I also plan to test it in a few weeks.

DaanHoogland · 2026-01-16T07:46:50Z

As our ex-colleague always says @GutoVeronezi , “we are in violent agreement”!

github-actions · 2026-01-27T10:08:12Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

bernardodemarco · 2026-03-02T21:21:14Z

@blueorangutan package

blueorangutan · 2026-03-02T21:22:04Z

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-02T22:18:29Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16988

sureshanaparti · 2026-03-04T15:17:05Z

@blueorangutan package

blueorangutan · 2026-03-04T15:20:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

github-actions · 2026-03-04T16:02:31Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

blueorangutan · 2026-03-04T16:41:39Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17001

bernardodemarco · 2026-03-04T20:27:19Z

@blueorangutan package

blueorangutan · 2026-03-04T20:28:04Z

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-04T21:21:11Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17004

bernardodemarco · 2026-03-05T13:40:11Z

@blueorangutan package

blueorangutan · 2026-03-05T13:42:04Z

@bernardodemarco a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

bernardodemarco · 2026-03-05T13:44:21Z

@shwstppr @sureshanaparti @DaanHoogland @weizhouapache @GutoVeronezi @winterhazel @JoaoJandre @hsato03 @erikbocks, guys, this one is finally ready for review and testing. Below are the test descriptions that I have performed:

N°	Test	Description	Expected result?
1	Key pair without explicit rules	Verified that when creating a key pair without explicit rules, all the account's role permissions are inferred as the permissions of the key pair.	Yes
2	Key pair with explicit rules	Verified that when creating a key pair with explicit rules, it is only granted access to the key pair's rules.	Yes
3	Change account's role rules after API keys have been created.	Verified that the key pair never has access to more APIs that its role has.	Yes
4	Key pairs creation with more permissions that the caller's role	Verified that it is not possible to create key pairs with more permission than the caller's set of permissions.	Yes
5	Key pair with start and end dates.	Verified that the `startdate` and `enddate` attributes of the key pair are correctly respected.	Yes
6	Key pairs creation from the UI.	Accessed the details page of an user and successfully generated an API key pair for it.	Yes
7	Invalidation of API key pairs after user/account deletion	Verified that after deleting accounts and users, their API key pairs are correctly deleted.	Yes
8	Invalidation of API key pairs when they are disabled	Verified that it is not possible to use the API key pairs when they are disabled for their accounts and users.	Yes
9	High-level workflows of the APIs `getUserKeys`, `registerUserKeys`, `deleteUserKeys`, `listUserKeyRules`, `listUserKeys`	Verified the execution of such APIs, including access validation checks.	Yes
10	Deletion of system keys	Tried to delete the keys of the `baremetal-system-account` account and verified that an error message was returned. Verified the possibility of removing the key pair when the `baremetal.provision.done.notification` is disabled.	Yes
11	Migration to the new DB's schema	Verified that the migration of active API key pairs from the `cloud.user` table to the `cloud.api_keypair` one was performed successfully.	Yes

bernardodemarco · 2026-03-05T13:45:08Z

@shwstppr @sureshanaparti @DaanHoogland @weizhouapache can we run the integration tests for the PR, please?

btw, the simulator tests are failing across multiple PRs (see #12683 and #12680, for instance). Thus, they do not seem to be related with the current PR.

blueorangutan · 2026-03-05T14:39:51Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17010

DaanHoogland · 2026-03-05T19:28:10Z

@blueorangutan package

blueorangutan · 2026-03-05T19:30:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2026-03-05T20:33:45Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 17014

DaanHoogland · 2026-03-06T08:09:49Z

@blueorangutan test

blueorangutan · 2026-03-06T08:12:03Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2026-03-06T23:45:52Z

[SF] Trillian test result (tid-15573)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 53151 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr9504-t15573-kvm-ol8.zip
Smoke tests completed. 145 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_03_ping_in_ssvm_success	`Failure`	14.66	test_diagnostics.py
test_04_extract_Iso	`Failure`	1.10	test_iso.py
ContextSuite context=TestListIdsParams>:teardown	`Error`	1.17	test_list_ids_parameter.py
test_01_snapshot_root_disk	`Error`	5.16	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	47.14	test_snapshots.py
test_02_list_snapshots_with_removed_data_store	`Error`	47.15	test_snapshots.py
ContextSuite context=TestSnapshotStandaloneBackup>:teardown	`Error`	32.48	test_snapshots.py
test_01_snapshot_usage	`Error`	22.94	test_usage.py
test_01_vpn_usage	`Error`	1.12	test_usage.py

boring-cyborgBot added the component:api label Aug 8, 2024

DaanHoogland closed this Aug 12, 2024

DaanHoogland reopened this Aug 12, 2024

DaanHoogland added the status:needs-testing label Aug 13, 2024

rajujith self-assigned this Aug 19, 2024

DaanHoogland reviewed Aug 20, 2024
View reviewed changes

Comment threadapi/src/main/java/org/apache/cloudstack/api/command/admin/user/ListUserKeysCmd.java Outdated
Comment threadengine/schema/src/main/java/org/apache/cloudstack/acl/ApiKeyPairPermissionVO.java Outdated

github-actionsBot added the status:has-conflicts label Aug 20, 2024

github-actionsBot removed the status:has-conflicts label Aug 20, 2024

apache deleted a comment from blueorangutan Aug 20, 2024

DaanHoogland reviewed Jan 21, 2026
View reviewed changes

Comment threadserver/src/main/java/com/cloud/api/ApiServer.java Outdated

API key pair restructure
4718ca7

improve log message
b6b1b5f

bernardodemarco mentioned this pull request May 22, 2026
UI for API Key Pair Management #13225
Merged
9 tasks

Uh oh!

Conversation

KlausDornsbach commented Aug 8, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

How Has This Been Tested?

API Key Creation and Basic Testing

Permissions Validation

Migration to api_keypair Table

General validations

Uh oh!

codecovBot commented Aug 12, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

DaanHoogland commented Aug 13, 2024

Uh oh!

KlausDornsbach commented Aug 14, 2024

Uh oh!

DaanHoogland commented Aug 16, 2024

Uh oh!

rajujith commented Aug 19, 2024

Uh oh!

Uh oh!

Uh oh!

github-actionsBot commented Aug 20, 2024

Uh oh!

DaanHoogland commented Aug 20, 2024

Uh oh!

blueorangutan commented Aug 20, 2024

Uh oh!

blueorangutan commented Aug 20, 2024

Uh oh!

rajujith commented Aug 21, 2024

Uh oh!

blueorangutan commented Aug 21, 2024

Uh oh!

blueorangutan commented Aug 21, 2024

Uh oh!

rajujith commented Aug 22, 2024

Uh oh!

blueorangutan commented Aug 22, 2024

Uh oh!

blueorangutan commented Aug 22, 2024

Uh oh!

DaanHoogland commented Aug 22, 2024

Uh oh!

GutoVeronezi commented Jan 15, 2026

Uh oh!

DaanHoogland commented Jan 15, 2026

Uh oh!

GutoVeronezi commented Jan 15, 2026

Uh oh!

DaanHoogland commented Jan 16, 2026

Uh oh!

Uh oh!

github-actionsBot commented Jan 27, 2026

Uh oh!

bernardodemarco commented Mar 2, 2026

Uh oh!

blueorangutan commented Mar 2, 2026

Uh oh!

blueorangutan commented Mar 2, 2026

Uh oh!

sureshanaparti commented Mar 4, 2026

Uh oh!

blueorangutan commented Mar 4, 2026

Uh oh!

github-actionsBot commented Mar 4, 2026

Uh oh!

blueorangutan commented Mar 4, 2026

Uh oh!

bernardodemarco commented Mar 4, 2026

Uh oh!

blueorangutan commented Mar 4, 2026

Uh oh!

KlausDornsbach commented Aug 8, 2024•
edited
Loading

Migration to `api_keypair` Table

codecovBot commented Aug 12, 2024•
edited
Loading