Use AzDO Pipeline Resources to define our build Docker containers by jkoritzinsky · Pull Request #75473 · dotnet/runtime

jkoritzinsky · 2022-09-12T19:15:19Z

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

ghost · 2022-09-12T19:15:36Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

Author:	jkoritzinsky
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

…iner resources. Also update all of our pipelines (we have a lot) to use this new format. This enables a few things: 1. We can pass the ROOTFS_DIR environment variable as part of the container. 2. Docker container names are better centralized in an easier to read and easier to parse file. Fix typo in template name Pipeline template parameters are declared differently for some reason (and are strongly typed) Fix runtime and runtime-llvm pipeilne defs Explicitly reference mcr.microsoft.com

hoyosjs

Do you have an official build?

jkoritzinsky · 2022-10-06T17:37:29Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2013530&view=results

jkoritzinsky · 2022-10-07T17:47:22Z

Files #76750 for the timeout

Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

Fallout from #75473, container handling happens in platform-matrix.yml now.

jkoritzinsky · 2022-09-12T19:15:19Z

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

ghost · 2022-09-12T19:15:36Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

Author:	jkoritzinsky
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

…iner resources. Also update all of our pipelines (we have a lot) to use this new format. This enables a few things: 1. We can pass the ROOTFS_DIR environment variable as part of the container. 2. Docker container names are better centralized in an easier to read and easier to parse file. Fix typo in template name Pipeline template parameters are declared differently for some reason (and are strongly typed) Fix runtime and runtime-llvm pipeilne defs Explicitly reference mcr.microsoft.com

hoyosjs

Do you have an official build?

jkoritzinsky · 2022-10-06T17:37:29Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2013530&view=results

jkoritzinsky · 2022-10-07T17:47:22Z

Files #76750 for the timeout

Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

Fallout from #75473, container handling happens in platform-matrix.yml now.

jkoritzinsky · 2022-09-12T19:15:19Z

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

ghost · 2022-09-12T19:15:36Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

Author:	jkoritzinsky
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

…iner resources. Also update all of our pipelines (we have a lot) to use this new format. This enables a few things: 1. We can pass the ROOTFS_DIR environment variable as part of the container. 2. Docker container names are better centralized in an easier to read and easier to parse file. Fix typo in template name Pipeline template parameters are declared differently for some reason (and are strongly typed) Fix runtime and runtime-llvm pipeilne defs Explicitly reference mcr.microsoft.com

hoyosjs

Do you have an official build?

jkoritzinsky · 2022-10-06T17:37:29Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2013530&view=results

jkoritzinsky · 2022-10-07T17:47:22Z

Files #76750 for the timeout

Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

Fallout from #75473, container handling happens in platform-matrix.yml now.

jkoritzinsky · 2022-09-12T19:15:19Z

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

ghost · 2022-09-12T19:15:36Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

Author:	jkoritzinsky
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

…iner resources. Also update all of our pipelines (we have a lot) to use this new format. This enables a few things: 1. We can pass the ROOTFS_DIR environment variable as part of the container. 2. Docker container names are better centralized in an easier to read and easier to parse file. Fix typo in template name Pipeline template parameters are declared differently for some reason (and are strongly typed) Fix runtime and runtime-llvm pipeilne defs Explicitly reference mcr.microsoft.com

hoyosjs

Do you have an official build?

jkoritzinsky · 2022-10-06T17:37:29Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2013530&view=results

jkoritzinsky · 2022-10-07T17:47:22Z

Files #76750 for the timeout

Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

Fallout from #75473, container handling happens in platform-matrix.yml now.

jkoritzinsky · 2022-09-12T19:15:19Z

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

ghost · 2022-09-12T19:15:36Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

Author:	jkoritzinsky
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

…iner resources. Also update all of our pipelines (we have a lot) to use this new format. This enables a few things: 1. We can pass the ROOTFS_DIR environment variable as part of the container. 2. Docker container names are better centralized in an easier to read and easier to parse file. Fix typo in template name Pipeline template parameters are declared differently for some reason (and are strongly typed) Fix runtime and runtime-llvm pipeilne defs Explicitly reference mcr.microsoft.com

hoyosjs

Do you have an official build?

jkoritzinsky · 2022-10-06T17:37:29Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2013530&view=results

jkoritzinsky · 2022-10-07T17:47:22Z

Files #76750 for the timeout

Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

Fallout from #75473, container handling happens in platform-matrix.yml now.

jkoritzinsky · 2022-09-12T19:15:19Z

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

ghost · 2022-09-12T19:15:36Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

Author:	jkoritzinsky
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

…iner resources. Also update all of our pipelines (we have a lot) to use this new format. This enables a few things: 1. We can pass the ROOTFS_DIR environment variable as part of the container. 2. Docker container names are better centralized in an easier to read and easier to parse file. Fix typo in template name Pipeline template parameters are declared differently for some reason (and are strongly typed) Fix runtime and runtime-llvm pipeilne defs Explicitly reference mcr.microsoft.com

hoyosjs

Do you have an official build?

jkoritzinsky · 2022-10-06T17:37:29Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2013530&view=results

jkoritzinsky · 2022-10-07T17:47:22Z

Files #76750 for the timeout

Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

Fallout from #75473, container handling happens in platform-matrix.yml now.

jkoritzinsky · 2022-09-12T19:15:19Z

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

ghost · 2022-09-12T19:15:36Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

Author:	jkoritzinsky
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

…iner resources. Also update all of our pipelines (we have a lot) to use this new format. This enables a few things: 1. We can pass the ROOTFS_DIR environment variable as part of the container. 2. Docker container names are better centralized in an easier to read and easier to parse file. Fix typo in template name Pipeline template parameters are declared differently for some reason (and are strongly typed) Fix runtime and runtime-llvm pipeilne defs Explicitly reference mcr.microsoft.com

hoyosjs

Do you have an official build?

jkoritzinsky · 2022-10-06T17:37:29Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2013530&view=results

jkoritzinsky · 2022-10-07T17:47:22Z

Files #76750 for the timeout

Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

Fallout from #75473, container handling happens in platform-matrix.yml now.

jkoritzinsky · 2022-09-12T19:15:19Z

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

ghost · 2022-09-12T19:15:36Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Refactor all of the docker images we use in our build into AzDO container resources.

Also update all of our pipelines (we have a lot) to use this new format.

This enables a few things:

We can pass the ROOTFS_DIR environment variable as part of the container.
Docker container names are better centralized in an easier to read and easier to parse file.
Reduce duplication of defining container resources for containers used in individual steps (instead of job-wide).
Enable an easy mechanism to reference specific containers that we use job-wide in specific steps.

The vast majority of changes in this PR are whitespace changes. I highly recommend viewing changes either side-by-side (to validate YAML whitespace changes) or with whitespace changes hidden (for the more interesting logic changes in the YAML).

Author:	jkoritzinsky
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

…iner resources. Also update all of our pipelines (we have a lot) to use this new format. This enables a few things: 1. We can pass the ROOTFS_DIR environment variable as part of the container. 2. Docker container names are better centralized in an easier to read and easier to parse file. Fix typo in template name Pipeline template parameters are declared differently for some reason (and are strongly typed) Fix runtime and runtime-llvm pipeilne defs Explicitly reference mcr.microsoft.com

hoyosjs

Do you have an official build?

jkoritzinsky · 2022-10-06T17:37:29Z

Official build at https://dev.azure.com/dnceng/internal/_build/results?buildId=2013530&view=results

jkoritzinsky · 2022-10-07T17:47:22Z

Files #76750 for the timeout

Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

Fallout from #75473, container handling happens in platform-matrix.yml now.

jkoritzinsky requested a review from radical as a code owner September 12, 2022 19:15

jkoritzinsky added the area-Infrastructure label Sep 12, 2022

ghost assigned jkoritzinsky Sep 12, 2022

jkoritzinsky force-pushed the pipeline-container-resources branch from 1c682f1 to 972bcdfCompare September 12, 2022 19:37

jkoritzinsky force-pushed the pipeline-container-resources branch from 606928f to faee2c3Compare October 3, 2022 21:45

jkoritzinsky requested a review from a teamOctober 3, 2022 23:04

akoeplinger reviewed Oct 4, 2022
View reviewed changes

Comment threadeng/pipelines/runtime.yml Outdated
Comment threadeng/pipelines/coreclr/perf_slow.yml Outdated

Fix bad diffs
47c7378

akoeplinger approved these changes Oct 4, 2022
View reviewed changes

Comment threadeng/pipelines/common/templates/pipeline-with-resources.yml

Add spacing
3ce2aaf

hoyosjs approved these changes Oct 4, 2022
View reviewed changes

Merge branch 'main' into pipeline-container-resources
948f983

jkoritzinsky mentioned this pull request Oct 7, 2022
Chrome debugger tests timeout #76750
Closed

jkoritzinsky merged commit 3c99def into dotnet:mainOct 7, 2022

jkoritzinsky deleted the pipeline-container-resources branch October 7, 2022 17:47

directhex added a commit to directhex/runtime that referenced this pull request Oct 7, 2022
Fix for pipelines broken indotnet#75473
ac9f655

directhex added a commit that referenced this pull request Oct 7, 2022
Fix for pipelines broken in#75473(#76758)
0058c73

DrewScoggins mentioned this pull request Oct 12, 2022
Remove container specification #76943
Merged

akoeplinger added a commit to akoeplinger/runtime that referenced this pull request Nov 2, 2022
Use correct container definitions in perf_slow.yml
198577b
Fallout from dotnet#75473, container handling happens in platform-matrix.yml now.

akoeplinger mentioned this pull request Nov 2, 2022
Use correct container definitions in perf_slow.yml #77774
Merged

ViktorHofer pushed a commit that referenced this pull request Nov 3, 2022
Use correct container definitions in perf_slow.yml (#77774)
c9216a7
Fallout from #75473, container handling happens in platform-matrix.yml now.

ghost locked as resolved and limited conversation to collaborators Nov 6, 2022