Create runtime clone to manually kick off full test runs

[steveisok]

This change allows devs to manually kick off full test runs on the configurations that only execute smoke tests per PR.

/azp runtime-manual will do the trick

[ghost]

Tagging subscribers to this area: @Anipik, @safern, @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Issue Details

---

null

Author:	steveisok
Assignees:	-
Labels:	area-Infrastructure-libraries
Milestone:	-

[jkoritzinsky]

This new pipeline has no differences from the runtime pipeline (at least in the YAML). What's the use case?

If this is the new "full" pipeline, then I would highly prefer if we were to manually pre-evaluate the 'isFullMatrix' variable and make the runtime-manual pipeline have the jobs from when it is true and the runtime pipeline have the jobs from when it is false. Otherwise we're adding complexity on complexity and making our build systems more difficult to understand instead of reducing complexity.

[safern]

These are already run as part of PRs; I guess the difference is that we would run the full tests not only the smoke tests?

[safern]

@jkoritzinsky this is the main difference. It will only run the mono jobs that were turned into smoke tests like this one, but with full test mode.

[steveisok]

Yes, that is correct.

[jkoritzinsky]

I would highly prefer if this pipeline could just include these jobs instead of adding another variable into the template.

We were already at the point where it's extremely difficult to understand what is triggered where. For example, this new pipeline would include the following job as the equivalent for the job this comment is attached to:

## Build for Browser/wasm, with EnableAggressiveTrimming=true#
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/global-build-job.ymlhelixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.ymlbuildConfig: ReleaseruntimeFlavor: monoplatforms:
- Browser_wasmjobParameters:
testGroup: innerloopnameSuffix: AllSubsets_Mono_EATbuildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=false /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=falsetimeoutInMinutes: 180# extra steps, run testsextraStepsTemplate: /eng/pipelines/libraries/helix.ymlextraStepsParameters:
creator: dotnet-bottestRunNamePrefixSuffix: Mono_$(_BuildConfig)extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true /p:RunSmokeTestsOnly=falsescenarios:
- normal

[steveisok]

This new pipeline has no differences from the runtime pipeline (at least in the YAML). What's the use case?

If this is the new "full" pipeline, then I would highly prefer if we were to manually pre-evaluate the 'isFullMatrix' variable and make the runtime-manual pipeline have the jobs from when it is true and the runtime pipeline have the jobs from when it is false. Otherwise we're adding complexity on complexity and making our build systems more difficult to understand instead of reducing complexity.

The intent is to allow devs to manually trigger full test runs for all legs. The reason why this is important is that some configurations only run System.Runtime tests on PR's. If a dev wants to add a test to wasm, for example, there is no good way until the rolling build to actually get a pass / fail indication.

I expect to have a follow up PR to make this a bit more streamlined. Merging this PR will give us something usable while we work out the best way.

[jkoritzinsky]

The intent is to allow devs to manually trigger full test runs for all legs. The reason why this is important is that some configurations only run System.Runtime tests on PR's. If a dev wants to add a test to wasm, for example, there is no good way until the rolling build to actually get a pass / fail indication.

If that is the use case, then this pipeline should explicitly define only the exact jobs that it runs. If it needs to run only a subset of the jobs, we should include only those jobs in the pipeline YAML. We should not just keep adding new variables to define when we run or skip different subsets of jobs, especially for manually triggered pipelines.

And following on the conversation we had that led to #61588, we shouldn't keep perpetuating the "some jobs in PR, some in CI" design as it adds complexity and makes knowing how to run a given job in PR vs CI difficult. We should instead have these jobs for these platforms on a separate pipeline that is CI or manual/comment trigger only.

I expect to have a follow up PR to make this a bit more streamlined. Merging this PR will give us something usable while we work out the best way.

I fear that this is one of those scenarios where "later" will continue to be pushed back due to more important work and our CI infrastructure will continue to become increasingly inscrutable to anyone who is not deeply involved in it regularly.

[steveisok]

If that is the use case, then this pipeline should explicitly define only the exact jobs that it runs. If it needs to run only a subset of the jobs, we should include only those jobs in the pipeline YAML. We should not just keep adding new variables to define when we run or skip different subsets of jobs, especially for manually triggered pipelines.

I'm after whatever will be an acceptable interim solution. If duplicating parts of runtime.yml will get us there, fine by me.

I fear that this is one of those scenarios where "later" will continue to be pushed back due to more important work and our CI infrastructure will continue to become increasingly inscrutable to anyone who is not deeply involved in it regularly.

With the mono configurations mostly under smoke tests, I'm not interested in punting on the right solution. I'd like to get it right.

[jkoritzinsky]

@steveisok I think the work to get a working build without adding complexity is not expensive enough to warrant the current proposed interim solution. The code block below should include all of the jobs from the shared pipeline conditioned on isManualOrIsNotPR. As a result, the following snippet should define all of the jobs needed in the runtime-manual.yml pipeline if the current pipeline definitions are accurate.

variables:
- template: /eng/pipelines/common/variables.ymljobs:
## Build for Browser/wasm, with EnableAggressiveTrimming=true#
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/global-build-job.ymlhelixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.ymlbuildConfig: ReleaseruntimeFlavor: monoplatforms:
- Browser_wasmvariables:
# map dependencies variables to local variables
- name: librariesContainsChangevalue: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
- name: monoContainsChangevalue: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ]jobParameters:
testGroup: innerloopnameSuffix: AllSubsets_Mono_EATbuildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=false /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=falsetimeoutInMinutes: 180# extra steps, run testsextraStepsTemplate: /eng/pipelines/libraries/helix.ymlextraStepsParameters:
creator: dotnet-bottestRunNamePrefixSuffix: Mono_$(_BuildConfig)extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true /p:RunSmokeTestsOnly=falsescenarios:
- normal## Build for Browser/wasm with RunAOTCompilation=true#
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/global-build-job.ymlhelixQueuesTemplate: /eng/pipelines/libraries/helix-queues-setup.ymlbuildConfig: ReleaseruntimeFlavor: monoplatforms:
- Browser_wasmvariables:
# map dependencies variables to local variables
- name: librariesContainsChangevalue: $[ dependencies.evaluate_paths.outputs['SetPathVars_libraries.containsChange'] ]
- name: monoContainsChangevalue: $[ dependencies.evaluate_paths.outputs['SetPathVars_mono.containsChange'] ]jobParameters:
testGroup: innerloopnameSuffix: AllSubsets_Mono_AOTbuildArgs: -s mono+libs+host+packs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=false /p:EnableAggressiveTrimming=true /p:BuildAOTTestsOnHelix=true /p:RunAOTCompilation=truetimeoutInMinutes: 180# extra steps, run testsextraStepsTemplate: /eng/pipelines/libraries/helix.ymlextraStepsParameters:
creator: dotnet-bottestRunNamePrefixSuffix: Mono_$(_BuildConfig)extraHelixArguments: /p:NeedsToBuildWasmAppsOnHelix=true /p:RunSmokeTestsOnly=falsescenarios:
- normal## Build the whole product using Mono and run runtime tests#
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/global-build-job.ymlhelixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.ymlbuildConfig: ReleaseruntimeFlavor: monoplatforms:
- Browser_wasmvariables:
- ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}:
- name: _HelixSourcevalue: pr/dotnet/runtime/$(Build.SourceBranch)
- ${{ if and(eq(variables['System.TeamProject'], 'public'), ne(variables['Build.Reason'], 'PullRequest')) }}:
- name: _HelixSourcevalue: ci/dotnet/runtime/$(Build.SourceBranch)
- name: timeoutPerTestInMinutesvalue: 10
- name: timeoutPerTestCollectionInMinutesvalue: 200jobParameters:
testGroup: innerloopnameSuffix: AllSubsets_Mono_RuntimeTestsbuildArgs: -s mono+libs -c $(_BuildConfig)timeoutInMinutes: 180# NOTE: Per PR test execution is not recommended for mobile runtime tests# extra steps, run testsextraStepsTemplate: /eng/pipelines/common/templates/runtimes/wasm-runtime-and-send-to-helix.ymlextraStepsParameters:
creator: dotnet-bottestRunNamePrefixSuffix: Mono_$(_BuildConfig)## Build the whole product using Mono for Android and run runtime tests with Android emulator#
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/global-build-job.ymlhelixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.ymlbuildConfig: ReleaseruntimeFlavor: monoplatforms:
- Android_x64variables:
- ${{ if and(eq(variables['System.TeamProject'], 'public'), eq(variables['Build.Reason'], 'PullRequest')) }}:
- name: _HelixSourcevalue: pr/dotnet/runtime/$(Build.SourceBranch)
- ${{ if and(eq(variables['System.TeamProject'], 'public'), ne(variables['Build.Reason'], 'PullRequest')) }}:
- name: _HelixSourcevalue: ci/dotnet/runtime/$(Build.SourceBranch)
- name: timeoutPerTestInMinutesvalue: 60
- name: timeoutPerTestCollectionInMinutesvalue: 180jobParameters:
testGroup: innerloopnameSuffix: AllSubsets_Mono_RuntimeTestsbuildArgs: -s mono+libs -c $(_BuildConfig)timeoutInMinutes: 240# NOTE: Per PR test execution is not recommended for mobile runtime tests# extra steps, run testsextraStepsTemplate: /eng/pipelines/common/templates/runtimes/android-runtime-and-send-to-helix.ymlextraStepsParameters:
creator: dotnet-bottestRunNamePrefixSuffix: Mono_$(_BuildConfig)## Mono Test builds with CoreCLR runtime tests using live libraries debug build
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/templates/runtimes/build-test-job.ymlbuildConfig: releaseruntimeFlavor: monoplatforms:
- CoreClrTestBuildHost # Either OSX_x64 or Linux_x64jobParameters:
testGroup: innerloopliveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}liveRuntimeBuildConfig: release## Mono CoreCLR runtime Test executions using live libraries in jit mode
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.ymlbuildConfig: releaseruntimeFlavor: monoplatforms:
- OSX_x64
- Linux_arm64helixQueueGroup: prhelixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.ymljobParameters:
testGroup: innerloopliveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}liveRuntimeBuildConfig: releaseruntimeVariant: minijit## Mono CoreCLR runtime Test executions using live libraries in interpreter mode
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.ymlbuildConfig: releaseruntimeFlavor: monoplatforms:
- OSX_x64
- Linux_arm64helixQueueGroup: prhelixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.ymljobParameters:
testGroup: innerloopliveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}liveRuntimeBuildConfig: releaseruntimeVariant: monointerpreter## Mono CoreCLR runtime Test executions using live libraries and LLVM AOT#
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.ymlbuildConfig: releaseruntimeFlavor: monoplatforms:
- Linux_x64# Disabled pending outcome of https://github.com/dotnet/runtime/issues/60234 investigation#- Linux_arm64helixQueueGroup: prhelixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.ymljobParameters:
testGroup: innerloopliveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}liveRuntimeBuildConfig: releaseruntimeVariant: llvmaot## Mono CoreCLR runtime Test executions using live libraries and LLVM Full AOT#
- template: /eng/pipelines/common/platform-matrix.ymlparameters:
jobTemplate: /eng/pipelines/common/templates/runtimes/run-test-job.ymlbuildConfig: releaseruntimeFlavor: monoplatforms:
- Linux_x64# Disabled pending outcome of https://github.com/dotnet/runtime/issues/60234 investigation#- Linux_arm64helixQueueGroup: prhelixQueuesTemplate: /eng/pipelines/coreclr/templates/helix-queues-setup.ymljobParameters:
testGroup: innerloopliveLibrariesBuildConfig: ${{ variables.debugOnPrReleaseOnRolling }}liveRuntimeBuildConfig: releaseruntimeVariant: llvmfullaot

Please change the runtime-manual pipeline to use this snippet (+ any changes required to get it to work) and change the runtime.yml pipeline back to the way it is on main.

[jkoritzinsky]

Let's make sure to run this pipeline at least once before merging so we know it works.

[steveisok]

/azp run runtime-manual

[azure-pipelines]

Azure Pipelines failed to run 1 pipeline(s).

[steveisok]

Let's make sure to run this pipeline at least once before merging so we know it works.

I wondered if there were any runtime test deps and it looks like there are. They may be acceptable to exclude for the time being.

[steveisok]

/azp run runtime-manual

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[lewing]

👏

[safern]

Wild idea, if we are using a separate template, why not include these as part of staging-manual? That way mono devs an people that care about this don't have to trigger two separate pipelines?

[steveisok]

Wild idea, if we are using a separate template, why not include these as part of staging-manual? That way mono devs an people that care about this don't have to trigger two separate pipelines?

I'd rather call it runtime-manual and fold runtime-staging-manual into it. With the way the PR builds have changed, there are less reasons to have builds in runtime-staging.

Good idea :-)

[safern]

I'd rather call it runtime-manual and fold runtime-staging-manual into it. With the way the PR builds have changed, there are less reasons to have builds in runtime-staging.

Like that idea.

[steveisok]

/azp run runtime-manual

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).