[Hilo] New module onboard

[yuc-Li]

Related command

Description

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature

---

This checklist is used to make sure that common guidelines for a pull request are followed.

• The PR title and description has followed the guideline in Submitting Pull Requests.

• I adhere to the Command Guidelines.

• I adhere to the Error Handling Guidelines.

[azure-client-tools-bot-prd]

❌AzureCLI-FullTest

️✔️acr

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️acs

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️advisor

️✔️latest

️✔️3.11

️✔️3.9

️✔️ams

️✔️latest

️✔️3.11

️✔️3.9

️✔️apim

️✔️latest

️✔️3.11

️✔️3.9

️✔️appconfig

️✔️latest

️✔️3.11

️✔️3.9

️✔️appservice

️✔️latest

️✔️3.11

️✔️3.9

️✔️aro

️✔️latest

️✔️3.11

️✔️3.9

️✔️backup

️✔️latest

️✔️3.11

️✔️3.9

️✔️batch

️✔️latest

️✔️3.11

️✔️3.9

️✔️batchai

️✔️latest

️✔️3.11

️✔️3.9

️✔️billing

️✔️latest

️✔️3.11

️✔️3.9

️✔️botservice

️✔️latest

️✔️3.11

️✔️3.9

️✔️cdn

️✔️latest

️✔️3.11

️✔️3.9

️✔️cloud

️✔️latest

️✔️3.11

️✔️3.9

️✔️cognitiveservices

️✔️latest

️✔️3.11

️✔️3.9

️✔️config

️✔️latest

️✔️3.11

️✔️3.9

️✔️configure

️✔️latest

️✔️3.11

️✔️3.9

️✔️consumption

️✔️latest

️✔️3.11

️✔️3.9

️✔️container

️✔️latest

️✔️3.11

️✔️3.9

️✔️containerapp

️✔️latest

️✔️3.11

️✔️3.9

️✔️core

️✔️2018-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️cosmosdb

️✔️latest

️✔️3.11

️✔️3.9

️✔️databoxedge

️✔️2019-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️dla

️✔️latest

️✔️3.11

️✔️3.9

️✔️dls

️✔️latest

️✔️3.11

️✔️3.9

️✔️dms

️✔️latest

️✔️3.11

️✔️3.9

️✔️eventgrid

️✔️latest

️✔️3.11

️✔️3.9

️✔️eventhubs

️✔️latest

️✔️3.11

️✔️3.9

️✔️feedback

️✔️latest

️✔️3.11

️✔️3.9

️✔️find

️✔️latest

️✔️3.11

️✔️3.9

️✔️hdinsight

️✔️latest

️✔️3.11

️✔️3.9

❌hdinsightonaks

❌latest

🔄3.11

❌3.9

Type	Test Case	Error Message	Line
Failed	test_create_cluster_with_hive	self = <azure.cli.testsdk.base.ExecutionResult object at 0x7f44e58d8490> cli_ctx = <azure.cli.core.mock.DummyCli object at 0x7f44e5d50ca0> command = 'hdinsightonaks cluster create  -n testclipool --cluster-pool-name hilopool-000002 -g cli -l westus3 --cluster-type Tr...on_password_secret':'sqlpassword','metastore_warehouse_dir':'abfs://test1@hilostorage.dfs.core.windows.net/'}"]' expect_failure = False     def in_process_execute(self, cli_ctx, command, expect_failure=False):         from io import StringIO         from vcr.errors import CannotOverwriteExistingCassetteException              if command.startswith('az '):             command = command[3:]              stdout_buf = StringIO()         logging_buf = StringIO()         try:             # issue: stderr cannot be redirect in this form, as a result some failure information             # is lost when command fails. >           self.exit_code = cli_ctx.invoke(shlex.split(command), out_file=stdout_buf) or 0 src/azure-cli-testsdk/azure/cli/testsdk/base.py:301:                                           env/lib/python3.9/site-packages/knack/cli.py:245: in invoke     exit_code = self.exception_handler(ex) src/azure-cli-core/azure/cli/core/init.py:127: in exception_handler     return handle_exception(ex) src/azure-cli-testsdk/azure/cli/testsdk/patches.py:33: in handle_main_exception     raise ex env/lib/python3.9/site-packages/knack/cli.py:233: in invoke     cmd_result = self.invocation.execute(args) src/azure-cli-core/azure/cli/core/commands/init.py:663: in execute     raise ex src/azure-cli-core/azure/cli/core/commands/init.py:726: in run_jobs_serially     results.append(self.run_job(expanded_arg, cmd_copy)) src/azure-cli-core/azure/cli/core/commands/init.py:708: in run_job     result = LongRunningOperation(cmd_copy.cli_ctx, 'Starting {}'.format(cmd_copy.name))(result) src/azure-cli-core/azure/cli/core/commands/init.py:1000: in call     result = poller.result() src/azure-cli-core/azure/cli/core/aaz/poller.py:108: in result     self.wait(timeout) env/lib/python3.9/site-packages/azure/core/tracing/decorator.py:78: in wrapper_use_tracer     return func(*args, **kwargs) src/azure-cli-core/azure/cli/core/aaz/poller.py:130: in wait     raise self.exception src/azure-cli-core/azure/cli/core/aaz/poller.py:83: in start     for polling_method in self.polling_generator: src/azure-cli/azure/cli/command_modules/hdinsightonaks/aaz/latest/hdinsightonaks/cluster/create.py:832: in execute_operations     yield self.ClustersCreate(ctx=self.ctx)() src/azure-cli/azure/cli/command_modules/hdinsightonaks/aaz/latest/hdinsightonaks/cluster/create.py:852: in call     session = self.client.send_request(request=request, stream=False, **kwargs) src/azure-cli-core/azure/cli/core/aaz/client.py:86: in send_request     session = self.pipeline.run(request, stream=stream, **kwargs) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:211: in run     return first_node.send(pipeline_request)  # type: ignore env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/mgmt/core/policies/base.py:47: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/policies/redirect.py:158: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/policies/retry.py:446: in send     response = self.next.send(request) src/azure-cli-core/azure/cli/core/aaz/http_policy.py:112: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:71: in send     response = self.next.send(request) env/lib/python3.9/site-packages/azure/core/pipeline/base.py:103: in send     self.sender.send(request.http_request, **request.context.options), env/lib/python3.9/site-packages/azure/core/pipeline/transport/requests_basic.py:327: in send     response = self.session.request(  # type: ignore env/lib/python3.9/site-packages/requests/sessions.py:589: in request     resp = self.send(prep, **send_kwargs) env/lib/python3.9/site-packages/requests/sessions.py:703: in send     r = adapter.send(request, **kwargs) env/lib/python3.9/site-packages/requests/adapters.py:486: in send     resp = conn.urlopen( env/lib/python3.9/site-packages/urllib3/connectionpool.py:715: in urlopen     httplib_response = self.make_request( env/lib/python3.9/site-packages/urllib3/connectionpool.py:458: in make_request     httplib_response = conn.getresponse(buffering=True)                                          self = <vcr.patch.VCRRequestsHTTPSConnection/mnt/vss/work/1/s/src/azure-cli/azure/cli/command_modules/hdinsightonaks/tests/latest/recordings/test_create_cluster_with_hive.yaml object at 0x7f44e58781f0>  = False, kwargs = {'buffering': True}     def getresponse(self, =False, **kwargs):         """Retrieve the response"""         # Check to see if the cassette has a response for this request. If so,         # then return it         if self.cassette.can_play_response_for(self.vcr_request):             log.info(f"Playing response for {self.vcr_request} from cassette")             response = self.cassette.play_response(self.vcr_request)             return VCRHTTPResponse(response)         else:             if self.cassette.write_protected and self.cassette.filter_request(self.vcr_request): >               raise CannotOverwriteExistingCassetteException(                     cassette=self.cassette,                     failed_request=self.vcr_request,                 ) E               vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/mnt/vss/work/1/s/src/azure-cli/azure/cli/command_modules/hdinsightonaks/tests/latest/recordings/test_create_cluster_with_hive.yaml') in your current record mode ('once'). E               No match for the request (<Request (PUT) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli/providers/Microsoft.HDInsight/clusterpools/hilopool-000002/clusters/testclipool?api-version=2023-06-01-preview>)&nbsp;was&nbsp;found. E               Found 1 similar requests with 1 different matcher(s) : E                E               1 - (<Request (PUT) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli/providers/Microsoft.HDInsight/clusterpools/testclipool/clusters/hilo-000003?api-version=2023-06-01-preview>). E               Matchers succeeded : ['method', 'scheme', 'host', 'port', 'custom_request_query_matcher'] E               Matchers failed : E               path - assertion failure : E               /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli/providers/Microsoft.HDInsight/clusterpools/hilopool-000002/clusters/testclipool != /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli/providers/Microsoft.HDInsight/clusterpools/testclipool/clusters/hilo-000003 env/lib/python3.9/site-packages/vcr/stubs/init.py:263: CannotOverwriteExistingCassetteException During handling of the above exception, another exception occurred: self = <azure.cli.command_modules.hdinsightonaks.tests.latest.test_hdinsightonaks_cluster.HdinsightonaksClusterScenario testMethod=test_create_cluster_with_hive>     @ResourceGroupPreparer(name_prefix='hilocli-', location=location, random_name_length=12)     def test_create_cluster_with_hive(self):         self.kwargs.update({                 "loc": self.location,                 "poolName": self.create_random_name(prefix='hilopool-', length=18),                 "clusterName": "testclipool", # self.create_random_name(prefix='hilo-', length=18),                 "clusterType": "Trino",                 "computeNodeProfile": self.cmd('az hdinsightonaks cluster node-profile create --count 5 --type Worker --vm-size Standard_D8d_v5').get_output_in_json(),    # Create a cluster node-profile object.                      "keyVaultResourceId": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/PSGroup/providers/Microsoft.KeyVault/vaults/sqlpass",                 "trinoHiveCatalogOption": self.cmd('az hdinsightonaks cluster trino-hive-catalog create --catalog-name ' + self.catalogName <br>                                                                + ' --metastore-db-connection-url ' + self.metastoreDbConnectionURL + ' --metastore-db-connection-user-name ' + self.metastoreDbUserName <br>                                                                + ' --metastore-db-connection-password-secret ' + self.metastoreDbPasswordSecret + ' --metastore-warehouse-dir ' + self.metastoreWarehouseDir).get_output_in_json(),                      "secret_reference": self.cmd('az hdinsightonaks cluster secret create --secret-name ' + self.secretName + ' --reference-name ' +  self.secretName).get_output_in_json()             })         # Get trino cluster version and ossVersion.         trino_versions = self.cmd('az hdinsightonaks available-cluster-version list -l {loc} --query "[?clusterType=='Trino']"').get_output_in_json()              # Create a cluster pool.         # self.cmd('az hdinsightonaks clusterpool create -g {rg} -n {poolName} -l {loc} --workernode-size Standard_E4s_v3', checks=[         #     self.check("name", '{poolName}'),         #     self.check("location", '{loc}'),         #     self.check("status", 'Running')         # ])              # Create a Trino cluster.         create_command = 'az hdinsightonaks cluster create  -n {clusterName} --cluster-pool-name {poolName} -g cli -l {loc} --cluster-type {clusterType} --cluster-version ' <br>              + trino_versions[0]["clusterVersion"] + ' --oss-version ' + trino_versions[0]["ossVersion"] + ' --nodes ' + '{computeNodeProfile}' <br>                +' '+ HdinsightonaksClusterScenario.authorization_info() + ' --secret-reference {secret_reference} --key-vault-id {keyVaultResourceId}' + ' --trino-hive-catalog {trinoHiveCatalogOption}'      >       self.cmd(create_command,checks=[             self.check("name", '{clusterName}'),             self.check("location", '{loc}'),         ]) src/azure-cli/azure/cli/command_modules/hdinsightonaks/tests/latest/test_hdinsightonaks_cluster.py:147:                                           src/azure-cli-testsdk/azure/cli/testsdk/base.py:175: in cmd     return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks) src/azure-cli-testsdk/azure/cli/testsdk/base.py:250: in init     self.in_process_execute(cli_ctx, command, expect_failure=expect_failure)                                          self = <azure.cli.testsdk.base.ExecutionResult object at 0x7f44e58d8490> cli_ctx = <azure.cli.core.mock.DummyCli object at 0x7f44e5d50ca0> command = 'hdinsightonaks cluster create  -n testclipool --cluster-pool-name hilopool-000002 -g cli -l westus3 --cluster-type Tr...on_password_secret':'sqlpassword','metastore_warehouse_dir':'abfs://test1@hilostorage.dfs.core.windows.net/'}"]' expect_failure = False     def _in_process_execute(self, cli_ctx, command, expect_failure=False):         from io import StringIO         from vcr.errors import CannotOverwriteExistingCassetteException              if command.startswith('az '):             command = command[3:]              stdout_buf = StringIO()         logging_buf = StringIO()         try:             # issue: stderr cannot be redirect in this form, as a result some failure information             # is lost when command fails.             self.exit_code = cli_ctx.invoke(shlex.split(command), out_file=stdout_buf) or 0             self.output = stdout_buf.getvalue()             self.applog = logging_buf.getvalue()              except CannotOverwriteExistingCassetteException as ex: >           raise AssertionError(ex) E           AssertionError: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/hdinsightonaks/tests/latest/recordings/test_create_cluster_with_hive.yaml') in your current record mode ('once'). E           No match for the request (<Request (PUT) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli/providers/Microsoft.HDInsight/clusterpools/hilopool-000002/clusters/testclipool?api-version=2023-06-01-preview>)&nbsp;was&nbsp;found. E           Found 1 similar requests with 1 different matcher(s) : E            E           1 - (<Request (PUT) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli/providers/Microsoft.HDInsight/clusterpools/testclipool/clusters/hilo-000003?api-version=2023-06-01-preview>). E           Matchers succeeded : ['method', 'scheme', 'host', 'port', '_custom_request_query_matcher'] E           Matchers failed : E           path - assertion failure : E           /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli/providers/Microsoft.HDInsight/clusterpools/hilopool-000002/clusters/testclipool != /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli/providers/Microsoft.HDInsight/clusterpools/testclipool/clusters/hilo-000003 src/azure-cli-testsdk/azure/cli/testsdk/base.py:306: AssertionError	azure/cli/command_modules/hdinsightonaks/tests/latest/test_hdinsightonaks_cluster.py:115

️✔️identity

️✔️latest

️✔️3.11

️✔️3.9

️✔️iot

️✔️2019-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️keyvault

️✔️2018-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️kusto

️✔️latest

️✔️3.11

️✔️3.9

️✔️lab

️✔️latest

️✔️3.11

️✔️3.9

️✔️managedservices

️✔️latest

️✔️3.11

️✔️3.9

️✔️maps

️✔️latest

️✔️3.11

️✔️3.9

️✔️marketplaceordering

️✔️latest

️✔️3.11

️✔️3.9

️✔️monitor

️✔️latest

️✔️3.11

️✔️3.9

🔄mysql

🔄latest

️✔️3.11

🔄3.9

️✔️netappfiles

️✔️latest

️✔️3.11

️✔️3.9

️✔️network

️✔️2018-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️policyinsights

️✔️latest

️✔️3.11

️✔️3.9

️✔️privatedns

️✔️latest

️✔️3.11

️✔️3.9

️✔️profile

️✔️latest

️✔️3.11

️✔️3.9

️✔️rdbms

️✔️latest

️✔️3.11

️✔️3.9

️✔️redis

️✔️latest

️✔️3.11

️✔️3.9

️✔️relay

️✔️latest

️✔️3.11

️✔️3.9

️✔️resource

️✔️2018-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️role

️✔️latest

️✔️3.11

️✔️3.9

️✔️search

️✔️latest

️✔️3.11

️✔️3.9

️✔️security

️✔️latest

️✔️3.11

️✔️3.9

️✔️servicebus

️✔️latest

️✔️3.11

️✔️3.9

️✔️serviceconnector

️✔️latest

️✔️3.11

️✔️3.9

️✔️servicefabric

️✔️latest

️✔️3.11

️✔️3.9

️✔️signalr

️✔️latest

️✔️3.11

️✔️3.9

️✔️sql

️✔️latest

️✔️3.11

️✔️3.9

️✔️sqlvm

️✔️latest

️✔️3.11

️✔️3.9

️✔️storage

️✔️2018-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️synapse

️✔️latest

️✔️3.11

️✔️3.9

️✔️telemetry

️✔️2018-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

️✔️util

️✔️latest

️✔️3.11

️✔️3.9

️✔️vm

️✔️2018-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.11

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.11

️✔️3.9

️✔️latest

️✔️3.11

️✔️3.9

[azure-client-tools-bot-prd]

️✔️AzureCLI-BreakingChangeTest

️✔️Non Breaking Changes

[Yong Zhang (yonzhan)]

Thank you for your contribution! We will review the pull request and get back to you soon.

[kai ru (kairu-ms)]

Hi yuc-Li, thanks for your contribution. Recommend releasing it as a cli extension which is more flexible to maintain and release.

[yuc-Li]

Hi yuc-Li, thanks for your contribution. Recommend releasing it as a cli extension which is more flexible to maintain and release.

Hi kai ru (@kairu-ms) Thank you for your reminder. I will try to convert this into an extension for releasing.

[kai ru (kairu-ms)]

Azure/azure-cli-extensions#6926