Add official signed build pipeline

[dagood]

Adds runtime-official.yml.

The new pipeline builds the subsets in release mode. It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts. After the installer build, a utility stage runs to sign and prepare the artifacts for publish. Then Arcade validation and publish stages run.

I tweaked Libraries publish a bit to let AllConfigurations bits get published. testhost doesn't seem to get built that way, causing a copy step to fail, plus publishing in an additional layout to get the OOB packages signed.

Feedback items on the runtime live CI build (#749) are not generally addressed in this PR. I changed skiptests from a parameter to a variable so that I could skip Installer tests, but not much else.

I've run this all the way through green, here: https://dev.azure.com/dnceng/internal/_build/results?buildId=460871

That build didn't have any channels auto-assigned to the branch, so no publishing happened. I'm working on trying out a validation channel in a new build now.

Tested publishing via the validation channel here: https://dev.azure.com/dnceng/internal/_build/results?buildId=461236. Looks reasonable to me. It didn't publish to dotnetcli blob storage but I believe Arcade will do that based on the channel the build's assigned to. master is already set up to default-assign to .NET Core 5 Dev, so building after merging will try that out.

@dotnet/runtime-infrastructure @mmitche

[safern]

Other than a few comments/questions, LGTM.

[safern]

@dagood I notices that the allConfigurations artifacts are downloaded in every installer build step... is that expected?

Should they only be downloaded in 1 build?

Which build is the one that will sign them?

[dagood]

That's normal, that's what I meant by fan out in this bit of the description:

It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Which build is the one that will sign them?

All the allconfigurations nupkgs are recursively signed by the "Prepare for Publish" stage. If the installer jobs need to pull out anything and put them in the Windows installers, those specific bits are signed then (because the Windows installers can't be recursively signed later). I don't know off the top of my head what's being pulled in this way though.

There might be some time saving potential there, but AFAIK we don't know yet if unnecessary round trips or redundant signing is worse yet.

[safern]

Thanks for explaining.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

[dagood]

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

@jkoritzinsky might, I think at least Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets.

[safern]

The reason why I ask is because the drop of allConfigurations packages is heavy. So we might instead be able to just upload to a different drop the ones we know are needed.

[jkoritzinsky]

The two packages @dagood mentioned are the packages that we need.

[dagood]

Adds runtime-official.yml.

The new pipeline builds the subsets in release mode. It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts. After the installer build, a utility stage runs to sign and prepare the artifacts for publish. Then Arcade validation and publish stages run.

I tweaked Libraries publish a bit to let AllConfigurations bits get published. testhost doesn't seem to get built that way, causing a copy step to fail, plus publishing in an additional layout to get the OOB packages signed.

Feedback items on the runtime live CI build (#749) are not generally addressed in this PR. I changed skiptests from a parameter to a variable so that I could skip Installer tests, but not much else.

I've run this all the way through green, here: https://dev.azure.com/dnceng/internal/_build/results?buildId=460871

That build didn't have any channels auto-assigned to the branch, so no publishing happened. I'm working on trying out a validation channel in a new build now.

Tested publishing via the validation channel here: https://dev.azure.com/dnceng/internal/_build/results?buildId=461236. Looks reasonable to me. It didn't publish to dotnetcli blob storage but I believe Arcade will do that based on the channel the build's assigned to. master is already set up to default-assign to .NET Core 5 Dev, so building after merging will try that out.

@dotnet/runtime-infrastructure @mmitche

[safern]

Other than a few comments/questions, LGTM.

[safern]

@dagood I notices that the allConfigurations artifacts are downloaded in every installer build step... is that expected?

Should they only be downloaded in 1 build?

Which build is the one that will sign them?

[dagood]

That's normal, that's what I meant by fan out in this bit of the description:

It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Which build is the one that will sign them?

All the allconfigurations nupkgs are recursively signed by the "Prepare for Publish" stage. If the installer jobs need to pull out anything and put them in the Windows installers, those specific bits are signed then (because the Windows installers can't be recursively signed later). I don't know off the top of my head what's being pulled in this way though.

There might be some time saving potential there, but AFAIK we don't know yet if unnecessary round trips or redundant signing is worse yet.

[safern]

Thanks for explaining.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

[dagood]

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

@jkoritzinsky might, I think at least Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets.

[safern]

The reason why I ask is because the drop of allConfigurations packages is heavy. So we might instead be able to just upload to a different drop the ones we know are needed.

[jkoritzinsky]

The two packages @dagood mentioned are the packages that we need.

[dagood]

Adds runtime-official.yml.

The new pipeline builds the subsets in release mode. It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts. After the installer build, a utility stage runs to sign and prepare the artifacts for publish. Then Arcade validation and publish stages run.

I tweaked Libraries publish a bit to let AllConfigurations bits get published. testhost doesn't seem to get built that way, causing a copy step to fail, plus publishing in an additional layout to get the OOB packages signed.

Feedback items on the runtime live CI build (#749) are not generally addressed in this PR. I changed skiptests from a parameter to a variable so that I could skip Installer tests, but not much else.

I've run this all the way through green, here: https://dev.azure.com/dnceng/internal/_build/results?buildId=460871

That build didn't have any channels auto-assigned to the branch, so no publishing happened. I'm working on trying out a validation channel in a new build now.

Tested publishing via the validation channel here: https://dev.azure.com/dnceng/internal/_build/results?buildId=461236. Looks reasonable to me. It didn't publish to dotnetcli blob storage but I believe Arcade will do that based on the channel the build's assigned to. master is already set up to default-assign to .NET Core 5 Dev, so building after merging will try that out.

@dotnet/runtime-infrastructure @mmitche

[safern]

Other than a few comments/questions, LGTM.

[safern]

@dagood I notices that the allConfigurations artifacts are downloaded in every installer build step... is that expected?

Should they only be downloaded in 1 build?

Which build is the one that will sign them?

[dagood]

That's normal, that's what I meant by fan out in this bit of the description:

It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Which build is the one that will sign them?

All the allconfigurations nupkgs are recursively signed by the "Prepare for Publish" stage. If the installer jobs need to pull out anything and put them in the Windows installers, those specific bits are signed then (because the Windows installers can't be recursively signed later). I don't know off the top of my head what's being pulled in this way though.

There might be some time saving potential there, but AFAIK we don't know yet if unnecessary round trips or redundant signing is worse yet.

[safern]

Thanks for explaining.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

[dagood]

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

@jkoritzinsky might, I think at least Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets.

[safern]

The reason why I ask is because the drop of allConfigurations packages is heavy. So we might instead be able to just upload to a different drop the ones we know are needed.

[jkoritzinsky]

The two packages @dagood mentioned are the packages that we need.

[dagood]

Adds runtime-official.yml.

The new pipeline builds the subsets in release mode. It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts. After the installer build, a utility stage runs to sign and prepare the artifacts for publish. Then Arcade validation and publish stages run.

I tweaked Libraries publish a bit to let AllConfigurations bits get published. testhost doesn't seem to get built that way, causing a copy step to fail, plus publishing in an additional layout to get the OOB packages signed.

Feedback items on the runtime live CI build (#749) are not generally addressed in this PR. I changed skiptests from a parameter to a variable so that I could skip Installer tests, but not much else.

I've run this all the way through green, here: https://dev.azure.com/dnceng/internal/_build/results?buildId=460871

That build didn't have any channels auto-assigned to the branch, so no publishing happened. I'm working on trying out a validation channel in a new build now.

Tested publishing via the validation channel here: https://dev.azure.com/dnceng/internal/_build/results?buildId=461236. Looks reasonable to me. It didn't publish to dotnetcli blob storage but I believe Arcade will do that based on the channel the build's assigned to. master is already set up to default-assign to .NET Core 5 Dev, so building after merging will try that out.

@dotnet/runtime-infrastructure @mmitche

[safern]

Other than a few comments/questions, LGTM.

[safern]

@dagood I notices that the allConfigurations artifacts are downloaded in every installer build step... is that expected?

Should they only be downloaded in 1 build?

Which build is the one that will sign them?

[dagood]

That's normal, that's what I meant by fan out in this bit of the description:

It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Which build is the one that will sign them?

All the allconfigurations nupkgs are recursively signed by the "Prepare for Publish" stage. If the installer jobs need to pull out anything and put them in the Windows installers, those specific bits are signed then (because the Windows installers can't be recursively signed later). I don't know off the top of my head what's being pulled in this way though.

There might be some time saving potential there, but AFAIK we don't know yet if unnecessary round trips or redundant signing is worse yet.

[safern]

Thanks for explaining.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

[dagood]

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

@jkoritzinsky might, I think at least Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets.

[safern]

The reason why I ask is because the drop of allConfigurations packages is heavy. So we might instead be able to just upload to a different drop the ones we know are needed.

[jkoritzinsky]

The two packages @dagood mentioned are the packages that we need.

[dagood]

Adds runtime-official.yml.

The new pipeline builds the subsets in release mode. It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts. After the installer build, a utility stage runs to sign and prepare the artifacts for publish. Then Arcade validation and publish stages run.

I tweaked Libraries publish a bit to let AllConfigurations bits get published. testhost doesn't seem to get built that way, causing a copy step to fail, plus publishing in an additional layout to get the OOB packages signed.

Feedback items on the runtime live CI build (#749) are not generally addressed in this PR. I changed skiptests from a parameter to a variable so that I could skip Installer tests, but not much else.

I've run this all the way through green, here: https://dev.azure.com/dnceng/internal/_build/results?buildId=460871

That build didn't have any channels auto-assigned to the branch, so no publishing happened. I'm working on trying out a validation channel in a new build now.

Tested publishing via the validation channel here: https://dev.azure.com/dnceng/internal/_build/results?buildId=461236. Looks reasonable to me. It didn't publish to dotnetcli blob storage but I believe Arcade will do that based on the channel the build's assigned to. master is already set up to default-assign to .NET Core 5 Dev, so building after merging will try that out.

@dotnet/runtime-infrastructure @mmitche

[safern]

Other than a few comments/questions, LGTM.

[safern]

@dagood I notices that the allConfigurations artifacts are downloaded in every installer build step... is that expected?

Should they only be downloaded in 1 build?

Which build is the one that will sign them?

[dagood]

That's normal, that's what I meant by fan out in this bit of the description:

It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Which build is the one that will sign them?

All the allconfigurations nupkgs are recursively signed by the "Prepare for Publish" stage. If the installer jobs need to pull out anything and put them in the Windows installers, those specific bits are signed then (because the Windows installers can't be recursively signed later). I don't know off the top of my head what's being pulled in this way though.

There might be some time saving potential there, but AFAIK we don't know yet if unnecessary round trips or redundant signing is worse yet.

[safern]

Thanks for explaining.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

[dagood]

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

@jkoritzinsky might, I think at least Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets.

[safern]

The reason why I ask is because the drop of allConfigurations packages is heavy. So we might instead be able to just upload to a different drop the ones we know are needed.

[jkoritzinsky]

The two packages @dagood mentioned are the packages that we need.

[dagood]

Adds runtime-official.yml.

The new pipeline builds the subsets in release mode. It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts. After the installer build, a utility stage runs to sign and prepare the artifacts for publish. Then Arcade validation and publish stages run.

I tweaked Libraries publish a bit to let AllConfigurations bits get published. testhost doesn't seem to get built that way, causing a copy step to fail, plus publishing in an additional layout to get the OOB packages signed.

Feedback items on the runtime live CI build (#749) are not generally addressed in this PR. I changed skiptests from a parameter to a variable so that I could skip Installer tests, but not much else.

I've run this all the way through green, here: https://dev.azure.com/dnceng/internal/_build/results?buildId=460871

That build didn't have any channels auto-assigned to the branch, so no publishing happened. I'm working on trying out a validation channel in a new build now.

Tested publishing via the validation channel here: https://dev.azure.com/dnceng/internal/_build/results?buildId=461236. Looks reasonable to me. It didn't publish to dotnetcli blob storage but I believe Arcade will do that based on the channel the build's assigned to. master is already set up to default-assign to .NET Core 5 Dev, so building after merging will try that out.

@dotnet/runtime-infrastructure @mmitche

[safern]

Other than a few comments/questions, LGTM.

[safern]

@dagood I notices that the allConfigurations artifacts are downloaded in every installer build step... is that expected?

Should they only be downloaded in 1 build?

Which build is the one that will sign them?

[dagood]

That's normal, that's what I meant by fan out in this bit of the description:

It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Which build is the one that will sign them?

All the allconfigurations nupkgs are recursively signed by the "Prepare for Publish" stage. If the installer jobs need to pull out anything and put them in the Windows installers, those specific bits are signed then (because the Windows installers can't be recursively signed later). I don't know off the top of my head what's being pulled in this way though.

There might be some time saving potential there, but AFAIK we don't know yet if unnecessary round trips or redundant signing is worse yet.

[safern]

Thanks for explaining.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

[dagood]

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

@jkoritzinsky might, I think at least Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets.

[safern]

The reason why I ask is because the drop of allConfigurations packages is heavy. So we might instead be able to just upload to a different drop the ones we know are needed.

[jkoritzinsky]

The two packages @dagood mentioned are the packages that we need.

[dagood]

Adds runtime-official.yml.

The new pipeline builds the subsets in release mode. It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts. After the installer build, a utility stage runs to sign and prepare the artifacts for publish. Then Arcade validation and publish stages run.

I tweaked Libraries publish a bit to let AllConfigurations bits get published. testhost doesn't seem to get built that way, causing a copy step to fail, plus publishing in an additional layout to get the OOB packages signed.

Feedback items on the runtime live CI build (#749) are not generally addressed in this PR. I changed skiptests from a parameter to a variable so that I could skip Installer tests, but not much else.

I've run this all the way through green, here: https://dev.azure.com/dnceng/internal/_build/results?buildId=460871

That build didn't have any channels auto-assigned to the branch, so no publishing happened. I'm working on trying out a validation channel in a new build now.

Tested publishing via the validation channel here: https://dev.azure.com/dnceng/internal/_build/results?buildId=461236. Looks reasonable to me. It didn't publish to dotnetcli blob storage but I believe Arcade will do that based on the channel the build's assigned to. master is already set up to default-assign to .NET Core 5 Dev, so building after merging will try that out.

@dotnet/runtime-infrastructure @mmitche

[safern]

Other than a few comments/questions, LGTM.

[safern]

@dagood I notices that the allConfigurations artifacts are downloaded in every installer build step... is that expected?

Should they only be downloaded in 1 build?

Which build is the one that will sign them?

[dagood]

That's normal, that's what I meant by fan out in this bit of the description:

It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Which build is the one that will sign them?

All the allconfigurations nupkgs are recursively signed by the "Prepare for Publish" stage. If the installer jobs need to pull out anything and put them in the Windows installers, those specific bits are signed then (because the Windows installers can't be recursively signed later). I don't know off the top of my head what's being pulled in this way though.

There might be some time saving potential there, but AFAIK we don't know yet if unnecessary round trips or redundant signing is worse yet.

[safern]

Thanks for explaining.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

[dagood]

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

@jkoritzinsky might, I think at least Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets.

[safern]

The reason why I ask is because the drop of allConfigurations packages is heavy. So we might instead be able to just upload to a different drop the ones we know are needed.

[jkoritzinsky]

The two packages @dagood mentioned are the packages that we need.

[dagood]

Adds runtime-official.yml.

The new pipeline builds the subsets in release mode. It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts. After the installer build, a utility stage runs to sign and prepare the artifacts for publish. Then Arcade validation and publish stages run.

I tweaked Libraries publish a bit to let AllConfigurations bits get published. testhost doesn't seem to get built that way, causing a copy step to fail, plus publishing in an additional layout to get the OOB packages signed.

Feedback items on the runtime live CI build (#749) are not generally addressed in this PR. I changed skiptests from a parameter to a variable so that I could skip Installer tests, but not much else.

I've run this all the way through green, here: https://dev.azure.com/dnceng/internal/_build/results?buildId=460871

That build didn't have any channels auto-assigned to the branch, so no publishing happened. I'm working on trying out a validation channel in a new build now.

Tested publishing via the validation channel here: https://dev.azure.com/dnceng/internal/_build/results?buildId=461236. Looks reasonable to me. It didn't publish to dotnetcli blob storage but I believe Arcade will do that based on the channel the build's assigned to. master is already set up to default-assign to .NET Core 5 Dev, so building after merging will try that out.

@dotnet/runtime-infrastructure @mmitche

[safern]

Other than a few comments/questions, LGTM.

[safern]

@dagood I notices that the allConfigurations artifacts are downloaded in every installer build step... is that expected?

Should they only be downloaded in 1 build?

Which build is the one that will sign them?

[dagood]

That's normal, that's what I meant by fan out in this bit of the description:

It includes an AllConfigurations libraries job, which fans out into each installer job in addition to the platform-specific artifacts.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Which build is the one that will sign them?

All the allconfigurations nupkgs are recursively signed by the "Prepare for Publish" stage. If the installer jobs need to pull out anything and put them in the Windows installers, those specific bits are signed then (because the Windows installers can't be recursively signed later). I don't know off the top of my head what's being pulled in this way though.

There might be some time saving potential there, but AFAIK we don't know yet if unnecessary round trips or redundant signing is worse yet.

[safern]

Thanks for explaining.

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

[dagood]

Some of the AllConfigurations packages are used to build the platform-specific installers assets, so all jobs need the bits.

Do we know which ones?

@jkoritzinsky might, I think at least Microsoft.NETCore.Platforms and Microsoft.NETCore.Targets.

[safern]

The reason why I ask is because the drop of allConfigurations packages is heavy. So we might instead be able to just upload to a different drop the ones we know are needed.

[jkoritzinsky]

The two packages @dagood mentioned are the packages that we need.