Publish dotnet/runtime using standard Arcade Publish tooling - #111934

Merged
jkoritzinsky merged 84 commits into
dotnet:mainfrom
jkoritzinsky:standard-arcade-publish
Mar 7, 2025
Merged

Publish dotnet/runtime using standard Arcade Publish tooling#111934
jkoritzinsky merged 84 commits into
dotnet:mainfrom
jkoritzinsky:standard-arcade-publish

Conversation

@jkoritzinsky

@jkoritzinskyjkoritzinsky commented Jan 28, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:

  • In a given build, publish all packages that were produced in the current build that contain that job's target RID.
  • For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
  • For some jobs (specified in runtime-official), publish all packages produced.
  • In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
  • Outside the VMR, skip publishing all "packages not to be published" packages.

Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793 As per #111934 (comment), we are no longer blocked on this.

Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.

Fixes#102672
Fixes#1719

@jkoritzinsky
Remove prepare-signed-artifacts and publish from each leg directly.
0364ee2
@jkoritzinsky
Publish in a separate stage to make dependency tracking easy
7e284a4
@jkoritzinsky
Set _SignType so we can pass -sign
d2c9cf8
@jkoritzinsky
Remove old validation target
3338d4d
@jkoritzinsky
!drop remove -sign so we can figure out publishing first
01bca60
@jkoritzinsky
Remove InitialTargets
88ee36c
@jkoritzinsky
Only generate product version files once. Fix asset manifest extensio…
d2fa244
…n. Don't copy workloads packages as they're already in the right format now.
@jkoritzinsky
Completely restructure how we publish to be more explicit that specif…
f93b7e4
…ic jobs publish specific packages
@jkoritzinsky
Move "non-RID-specific" special packages logic into Publishing.props
8ee9ac1
@jkoritzinsky
Make sure we build the list of skipped packages after we add the RID-…
4455301
…specific packages
@jkoritzinsky
Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…
e36e87e
…es. Use that when possible for unique names and produce our own names when necessary.
@jkoritzinsky
Extract handling hostOS and hostArch into another props file
747bffe
@jkoritzinsky
Fix import and adjust condition for SB
fbe3515
@jkoritzinsky
Fix asset manifest name uniqueness to ensue we don't overwrite asset …
6bbbc4c
…manifests.
@jkoritzinsky
Set AssetManifestFilePath again
c1068eb
@jkoritzinsky
Fix PublishAllPackages condition and make sure to publish the VS redi…
9e61e42
…st packages
@jkoritzinsky
Explicitly look up packages as we add zips and other blob artifacts t…
05fe23b
…o the Artifacts items in Signing.props
@jkoritzinsky
Search in the right folder.
d8b032a
@jkoritzinsky
Look up zips in VSSetup for workloads
2ac88a1
@jkoritzinsky
Add missing semicolon
552bb1d
@jkoritzinsky
Include VSSetup artifacts separately
9247027
@jkoritzinsky
Re-enable signing (depends on Microbuild signing on Linux/Mac)
1eb6a85
@jkoritzinsky
Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…
72af6e0
…-publish
@jkoritzinskyjkoritzinsky added blocked Issue/PR is blocked on something - see comments area-Infrastructure labels Jan 28, 2025
@build-analysisbuild-analysisBot mentioned this pull request Jan 29, 2025
Closed
ViktorHofer
ViktorHofer reviewed Jan 29, 2025
View reviewed changes
Comment threadeng/pipelines/runtime-official.yml Outdated
@jkoritzinsky

jkoritzinsky commented Mar 6, 2025

Copy link
Copy Markdown
MemberAuthor

I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.

Here's my basic approach:

  • Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
  • Runtime verticals won't sign in the runtime official build
  • We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
  • We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.

I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results

@jkoritzinskyjkoritzinsky removed the blocked Issue/PR is blocked on something - see comments label Mar 6, 2025
@jkoritzinsky

jkoritzinsky commented Mar 7, 2025

Copy link
Copy Markdown
MemberAuthor

New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results

@jkoritzinskyjkoritzinsky mentioned this pull request Mar 7, 2025
Merged
ViktorHofer
ViktorHofer reviewed Mar 7, 2025
View reviewed changes
Comment threadeng/DotNetBuild.props
Comment on lines +45 to +52
<PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
<!-- Source-build always needs all artifacts to be published. -->
<EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
<!-- Short-stack builds should always only publish RID-specific artifacts. -->
<EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
<!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
<EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
</PropertyGroup>

@ViktorHoferViktorHoferMar 7, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?

@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)

@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the VMR controls are in, I'll come back and remove these.

@ViktorHoferViktorHoferMar 7, 2025

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.

ViktorHofer
ViktorHofer reviewed Mar 7, 2025
View reviewed changes
Comment threadeng/Signing.props Outdated
@jkoritzinsky@ViktorHofer
Update Signing.props
5312597
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
@build-analysisbuild-analysisBot mentioned this pull request Mar 7, 2025
Closed
3 tasks
@build-analysisbuild-analysisBot mentioned this pull request Mar 7, 2025
Closed
@jkoritzinsky

jkoritzinsky commented Mar 7, 2025

Copy link
Copy Markdown
MemberAuthor

/ba-g timeouts unrelated

@jkoritzinsky
jkoritzinsky merged commit 891ff91 into dotnet:mainMar 7, 2025
@jkoritzinsky
jkoritzinsky deleted the standard-arcade-publish branch March 7, 2025 22:55
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Apr 7, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@ViktorHoferViktorHoferViktorHofer approved these changes
@mmitchemmitchemmitche approved these changes
@akoeplingerakoeplingerAwaiting requested review from akoeplinger

Assignees

@jkoritzinskyjkoritzinsky

Labels

area-Infrastructure

Projects

Runtime Infra
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Duplicate ref packs are getting uploaded Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts

3 participants

@jkoritzinsky@ViktorHofer@mmitche
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Publish dotnet/runtime using standard Arcade Publish tooling - #111934
Merged
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Mar 7, 2025
Merged
Publish dotnet/runtime using standard Arcade Publish tooling#111934
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Conversation
@jkoritzinsky
@jkoritzinskyjkoritzinsky
commented
Jan 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:
  • In a given build, publish all packages that were produced in the current build that contain that job's target RID.
  • For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
  • For some jobs (specified in runtime-official), publish all packages produced.
  • In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
  • Outside the VMR, skip publishing all "packages not to be published" packages.
Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793 As per #111934 (comment), we are no longer blocked on this.
Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.
Fixes#102672
Fixes#1719
@jkoritzinsky
Remove prepare-signed-artifacts and publish from each leg directly.
0364ee2
@jkoritzinsky
Publish in a separate stage to make dependency tracking easy
7e284a4
@jkoritzinsky
Set _SignType so we can pass -sign
d2c9cf8
@jkoritzinsky
Remove old validation target
3338d4d
@jkoritzinsky
!drop remove -sign so we can figure out publishing first
01bca60
@jkoritzinsky
Remove InitialTargets
88ee36c
@jkoritzinsky
Only generate product version files once. Fix asset manifest extensio…
d2fa244
…n. Don't copy workloads packages as they're already in the right format now.
@jkoritzinsky
Completely restructure how we publish to be more explicit that specif…
f93b7e4
…ic jobs publish specific packages
@jkoritzinsky
Move "non-RID-specific" special packages logic into Publishing.props
8ee9ac1
@jkoritzinsky
Make sure we build the list of skipped packages after we add the RID-…
4455301
…specific packages
@jkoritzinsky
Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…
e36e87e
…es. Use that when possible for unique names and produce our own names when necessary.
@jkoritzinsky
Extract handling hostOS and hostArch into another props file
747bffe
@jkoritzinsky
Fix import and adjust condition for SB
fbe3515
@jkoritzinsky
Fix asset manifest name uniqueness to ensue we don't overwrite asset …
6bbbc4c
…manifests.
@jkoritzinsky
Set AssetManifestFilePath again
c1068eb
@jkoritzinsky
Fix PublishAllPackages condition and make sure to publish the VS redi…
9e61e42
…st packages
@jkoritzinsky
Explicitly look up packages as we add zips and other blob artifacts t…
05fe23b
…o the Artifacts items in Signing.props
@jkoritzinsky
Search in the right folder.
d8b032a
@jkoritzinsky
Look up zips in VSSetup for workloads
2ac88a1
@jkoritzinsky
Add missing semicolon
552bb1d
@jkoritzinsky
Include VSSetup artifacts separately
9247027
@jkoritzinsky
Re-enable signing (depends on Microbuild signing on Linux/Mac)
1eb6a85
@jkoritzinsky
Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…
72af6e0
…-publish
@jkoritzinskyjkoritzinsky
added

blocked
Issue/PR is blocked on something - see comments
area-Infrastructure

labels
Jan 28, 2025
@build-analysisbuild-analysisBot
mentioned this pull request
Jan 29, 2025

Closed

ViktorHofer
ViktorHofer
reviewed
Jan 29, 2025
View reviewed changes
Comment threadeng/pipelines/runtime-official.yml
Outdated

@jkoritzinsky
jkoritzinsky
commented
Mar 6, 2025

Copy link


Copy Markdown

MemberAuthor
I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.
Here's my basic approach:
  • Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
  • Runtime verticals won't sign in the runtime official build
  • We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
  • We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.
I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results
@jkoritzinskyjkoritzinsky
removed
the 
blocked
Issue/PR is blocked on something - see comments
label
Mar 6, 2025
@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results
@jkoritzinskyjkoritzinsky
mentioned this pull request
Mar 7, 2025
 Merged

ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/DotNetBuild.props

Comment on lines

+45
 to 
+52

 <PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
 <!-- Source-build always needs all artifacts to be published. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
 <!-- Short-stack builds should always only publish RID-specific artifacts. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
 <!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
 </PropertyGroup>
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the VMR controls are in, I'll come back and remove these.
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.
ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/Signing.props
Outdated

@jkoritzinsky@ViktorHofer
Update Signing.props
5312597
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

3 tasks
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
/ba-g timeouts unrelated
@jkoritzinsky
jkoritzinsky
merged commit 891ff91
into
dotnet:mainMar 7, 2025
@jkoritzinsky
jkoritzinsky
deleted the

standard-arcade-publish

branch
March 7, 2025 22:55
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 7, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@ViktorHoferViktorHoferViktorHofer approved these changes
@mmitchemmitchemmitche approved these changes
@akoeplingerakoeplingerAwaiting requested review from akoeplinger

Assignees

@jkoritzinskyjkoritzinsky

Labels

area-Infrastructure

Projects

Runtime Infra

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate ref packs are getting uploaded

Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts


3 participants

@jkoritzinsky@ViktorHofer@mmitche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Publish dotnet/runtime using standard Arcade Publish tooling - #111934
Merged
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Mar 7, 2025
Merged
Publish dotnet/runtime using standard Arcade Publish tooling#111934
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Conversation
@jkoritzinsky
@jkoritzinskyjkoritzinsky
commented
Jan 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:
  • In a given build, publish all packages that were produced in the current build that contain that job's target RID.
  • For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
  • For some jobs (specified in runtime-official), publish all packages produced.
  • In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
  • Outside the VMR, skip publishing all "packages not to be published" packages.
Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793 As per #111934 (comment), we are no longer blocked on this.
Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.
Fixes#102672
Fixes#1719
@jkoritzinsky
Remove prepare-signed-artifacts and publish from each leg directly.
0364ee2
@jkoritzinsky
Publish in a separate stage to make dependency tracking easy
7e284a4
@jkoritzinsky
Set _SignType so we can pass -sign
d2c9cf8
@jkoritzinsky
Remove old validation target
3338d4d
@jkoritzinsky
!drop remove -sign so we can figure out publishing first
01bca60
@jkoritzinsky
Remove InitialTargets
88ee36c
@jkoritzinsky
Only generate product version files once. Fix asset manifest extensio…
d2fa244
…n. Don't copy workloads packages as they're already in the right format now.
@jkoritzinsky
Completely restructure how we publish to be more explicit that specif…
f93b7e4
…ic jobs publish specific packages
@jkoritzinsky
Move "non-RID-specific" special packages logic into Publishing.props
8ee9ac1
@jkoritzinsky
Make sure we build the list of skipped packages after we add the RID-…
4455301
…specific packages
@jkoritzinsky
Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…
e36e87e
…es. Use that when possible for unique names and produce our own names when necessary.
@jkoritzinsky
Extract handling hostOS and hostArch into another props file
747bffe
@jkoritzinsky
Fix import and adjust condition for SB
fbe3515
@jkoritzinsky
Fix asset manifest name uniqueness to ensue we don't overwrite asset …
6bbbc4c
…manifests.
@jkoritzinsky
Set AssetManifestFilePath again
c1068eb
@jkoritzinsky
Fix PublishAllPackages condition and make sure to publish the VS redi…
9e61e42
…st packages
@jkoritzinsky
Explicitly look up packages as we add zips and other blob artifacts t…
05fe23b
…o the Artifacts items in Signing.props
@jkoritzinsky
Search in the right folder.
d8b032a
@jkoritzinsky
Look up zips in VSSetup for workloads
2ac88a1
@jkoritzinsky
Add missing semicolon
552bb1d
@jkoritzinsky
Include VSSetup artifacts separately
9247027
@jkoritzinsky
Re-enable signing (depends on Microbuild signing on Linux/Mac)
1eb6a85
@jkoritzinsky
Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…
72af6e0
…-publish
@jkoritzinskyjkoritzinsky
added

blocked
Issue/PR is blocked on something - see comments
area-Infrastructure

labels
Jan 28, 2025
@build-analysisbuild-analysisBot
mentioned this pull request
Jan 29, 2025

Closed

ViktorHofer
ViktorHofer
reviewed
Jan 29, 2025
View reviewed changes
Comment threadeng/pipelines/runtime-official.yml
Outdated

@jkoritzinsky
jkoritzinsky
commented
Mar 6, 2025

Copy link


Copy Markdown

MemberAuthor
I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.
Here's my basic approach:
  • Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
  • Runtime verticals won't sign in the runtime official build
  • We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
  • We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.
I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results
@jkoritzinskyjkoritzinsky
removed
the 
blocked
Issue/PR is blocked on something - see comments
label
Mar 6, 2025
@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results
@jkoritzinskyjkoritzinsky
mentioned this pull request
Mar 7, 2025
 Merged

ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/DotNetBuild.props

Comment on lines

+45
 to 
+52

 <PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
 <!-- Source-build always needs all artifacts to be published. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
 <!-- Short-stack builds should always only publish RID-specific artifacts. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
 <!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
 </PropertyGroup>
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the VMR controls are in, I'll come back and remove these.
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.
ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/Signing.props
Outdated

@jkoritzinsky@ViktorHofer
Update Signing.props
5312597
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

3 tasks
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
/ba-g timeouts unrelated
@jkoritzinsky
jkoritzinsky
merged commit 891ff91
into
dotnet:mainMar 7, 2025
@jkoritzinsky
jkoritzinsky
deleted the

standard-arcade-publish

branch
March 7, 2025 22:55
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 7, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@ViktorHoferViktorHoferViktorHofer approved these changes
@mmitchemmitchemmitche approved these changes
@akoeplingerakoeplingerAwaiting requested review from akoeplinger

Assignees

@jkoritzinskyjkoritzinsky

Labels

area-Infrastructure

Projects

Runtime Infra

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate ref packs are getting uploaded

Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts


3 participants

@jkoritzinsky@ViktorHofer@mmitche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Publish dotnet/runtime using standard Arcade Publish tooling - #111934
Merged
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Mar 7, 2025
Merged
Publish dotnet/runtime using standard Arcade Publish tooling#111934
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Conversation
@jkoritzinsky
@jkoritzinskyjkoritzinsky
commented
Jan 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:
  • In a given build, publish all packages that were produced in the current build that contain that job's target RID.
  • For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
  • For some jobs (specified in runtime-official), publish all packages produced.
  • In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
  • Outside the VMR, skip publishing all "packages not to be published" packages.
Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793 As per #111934 (comment), we are no longer blocked on this.
Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.
Fixes#102672
Fixes#1719
@jkoritzinsky
Remove prepare-signed-artifacts and publish from each leg directly.
0364ee2
@jkoritzinsky
Publish in a separate stage to make dependency tracking easy
7e284a4
@jkoritzinsky
Set _SignType so we can pass -sign
d2c9cf8
@jkoritzinsky
Remove old validation target
3338d4d
@jkoritzinsky
!drop remove -sign so we can figure out publishing first
01bca60
@jkoritzinsky
Remove InitialTargets
88ee36c
@jkoritzinsky
Only generate product version files once. Fix asset manifest extensio…
d2fa244
…n. Don't copy workloads packages as they're already in the right format now.
@jkoritzinsky
Completely restructure how we publish to be more explicit that specif…
f93b7e4
…ic jobs publish specific packages
@jkoritzinsky
Move "non-RID-specific" special packages logic into Publishing.props
8ee9ac1
@jkoritzinsky
Make sure we build the list of skipped packages after we add the RID-…
4455301
…specific packages
@jkoritzinsky
Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…
e36e87e
…es. Use that when possible for unique names and produce our own names when necessary.
@jkoritzinsky
Extract handling hostOS and hostArch into another props file
747bffe
@jkoritzinsky
Fix import and adjust condition for SB
fbe3515
@jkoritzinsky
Fix asset manifest name uniqueness to ensue we don't overwrite asset …
6bbbc4c
…manifests.
@jkoritzinsky
Set AssetManifestFilePath again
c1068eb
@jkoritzinsky
Fix PublishAllPackages condition and make sure to publish the VS redi…
9e61e42
…st packages
@jkoritzinsky
Explicitly look up packages as we add zips and other blob artifacts t…
05fe23b
…o the Artifacts items in Signing.props
@jkoritzinsky
Search in the right folder.
d8b032a
@jkoritzinsky
Look up zips in VSSetup for workloads
2ac88a1
@jkoritzinsky
Add missing semicolon
552bb1d
@jkoritzinsky
Include VSSetup artifacts separately
9247027
@jkoritzinsky
Re-enable signing (depends on Microbuild signing on Linux/Mac)
1eb6a85
@jkoritzinsky
Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…
72af6e0
…-publish
@jkoritzinskyjkoritzinsky
added

blocked
Issue/PR is blocked on something - see comments
area-Infrastructure

labels
Jan 28, 2025
@build-analysisbuild-analysisBot
mentioned this pull request
Jan 29, 2025

Closed

ViktorHofer
ViktorHofer
reviewed
Jan 29, 2025
View reviewed changes
Comment threadeng/pipelines/runtime-official.yml
Outdated

@jkoritzinsky
jkoritzinsky
commented
Mar 6, 2025

Copy link


Copy Markdown

MemberAuthor
I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.
Here's my basic approach:
  • Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
  • Runtime verticals won't sign in the runtime official build
  • We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
  • We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.
I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results
@jkoritzinskyjkoritzinsky
removed
the 
blocked
Issue/PR is blocked on something - see comments
label
Mar 6, 2025
@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results
@jkoritzinskyjkoritzinsky
mentioned this pull request
Mar 7, 2025
 Merged

ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/DotNetBuild.props

Comment on lines

+45
 to 
+52

 <PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
 <!-- Source-build always needs all artifacts to be published. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
 <!-- Short-stack builds should always only publish RID-specific artifacts. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
 <!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
 </PropertyGroup>
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the VMR controls are in, I'll come back and remove these.
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.
ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/Signing.props
Outdated

@jkoritzinsky@ViktorHofer
Update Signing.props
5312597
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

3 tasks
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
/ba-g timeouts unrelated
@jkoritzinsky
jkoritzinsky
merged commit 891ff91
into
dotnet:mainMar 7, 2025
@jkoritzinsky
jkoritzinsky
deleted the

standard-arcade-publish

branch
March 7, 2025 22:55
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 7, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@ViktorHoferViktorHoferViktorHofer approved these changes
@mmitchemmitchemmitche approved these changes
@akoeplingerakoeplingerAwaiting requested review from akoeplinger

Assignees

@jkoritzinskyjkoritzinsky

Labels

area-Infrastructure

Projects

Runtime Infra

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate ref packs are getting uploaded

Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts


3 participants

@jkoritzinsky@ViktorHofer@mmitche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Publish dotnet/runtime using standard Arcade Publish tooling - #111934
Merged
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Mar 7, 2025
Merged
Publish dotnet/runtime using standard Arcade Publish tooling#111934
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Conversation
@jkoritzinsky
@jkoritzinskyjkoritzinsky
commented
Jan 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:
  • In a given build, publish all packages that were produced in the current build that contain that job's target RID.
  • For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
  • For some jobs (specified in runtime-official), publish all packages produced.
  • In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
  • Outside the VMR, skip publishing all "packages not to be published" packages.
Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793 As per #111934 (comment), we are no longer blocked on this.
Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.
Fixes#102672
Fixes#1719
@jkoritzinsky
Remove prepare-signed-artifacts and publish from each leg directly.
0364ee2
@jkoritzinsky
Publish in a separate stage to make dependency tracking easy
7e284a4
@jkoritzinsky
Set _SignType so we can pass -sign
d2c9cf8
@jkoritzinsky
Remove old validation target
3338d4d
@jkoritzinsky
!drop remove -sign so we can figure out publishing first
01bca60
@jkoritzinsky
Remove InitialTargets
88ee36c
@jkoritzinsky
Only generate product version files once. Fix asset manifest extensio…
d2fa244
…n. Don't copy workloads packages as they're already in the right format now.
@jkoritzinsky
Completely restructure how we publish to be more explicit that specif…
f93b7e4
…ic jobs publish specific packages
@jkoritzinsky
Move "non-RID-specific" special packages logic into Publishing.props
8ee9ac1
@jkoritzinsky
Make sure we build the list of skipped packages after we add the RID-…
4455301
…specific packages
@jkoritzinsky
Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…
e36e87e
…es. Use that when possible for unique names and produce our own names when necessary.
@jkoritzinsky
Extract handling hostOS and hostArch into another props file
747bffe
@jkoritzinsky
Fix import and adjust condition for SB
fbe3515
@jkoritzinsky
Fix asset manifest name uniqueness to ensue we don't overwrite asset …
6bbbc4c
…manifests.
@jkoritzinsky
Set AssetManifestFilePath again
c1068eb
@jkoritzinsky
Fix PublishAllPackages condition and make sure to publish the VS redi…
9e61e42
…st packages
@jkoritzinsky
Explicitly look up packages as we add zips and other blob artifacts t…
05fe23b
…o the Artifacts items in Signing.props
@jkoritzinsky
Search in the right folder.
d8b032a
@jkoritzinsky
Look up zips in VSSetup for workloads
2ac88a1
@jkoritzinsky
Add missing semicolon
552bb1d
@jkoritzinsky
Include VSSetup artifacts separately
9247027
@jkoritzinsky
Re-enable signing (depends on Microbuild signing on Linux/Mac)
1eb6a85
@jkoritzinsky
Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…
72af6e0
…-publish
@jkoritzinskyjkoritzinsky
added

blocked
Issue/PR is blocked on something - see comments
area-Infrastructure

labels
Jan 28, 2025
@build-analysisbuild-analysisBot
mentioned this pull request
Jan 29, 2025

Closed

ViktorHofer
ViktorHofer
reviewed
Jan 29, 2025
View reviewed changes
Comment threadeng/pipelines/runtime-official.yml
Outdated

@jkoritzinsky
jkoritzinsky
commented
Mar 6, 2025

Copy link


Copy Markdown

MemberAuthor
I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.
Here's my basic approach:
  • Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
  • Runtime verticals won't sign in the runtime official build
  • We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
  • We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.
I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results
@jkoritzinskyjkoritzinsky
removed
the 
blocked
Issue/PR is blocked on something - see comments
label
Mar 6, 2025
@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results
@jkoritzinskyjkoritzinsky
mentioned this pull request
Mar 7, 2025
 Merged

ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/DotNetBuild.props

Comment on lines

+45
 to 
+52

 <PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
 <!-- Source-build always needs all artifacts to be published. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
 <!-- Short-stack builds should always only publish RID-specific artifacts. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
 <!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
 </PropertyGroup>
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the VMR controls are in, I'll come back and remove these.
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.
ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/Signing.props
Outdated

@jkoritzinsky@ViktorHofer
Update Signing.props
5312597
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

3 tasks
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
/ba-g timeouts unrelated
@jkoritzinsky
jkoritzinsky
merged commit 891ff91
into
dotnet:mainMar 7, 2025
@jkoritzinsky
jkoritzinsky
deleted the

standard-arcade-publish

branch
March 7, 2025 22:55
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 7, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@ViktorHoferViktorHoferViktorHofer approved these changes
@mmitchemmitchemmitche approved these changes
@akoeplingerakoeplingerAwaiting requested review from akoeplinger

Assignees

@jkoritzinskyjkoritzinsky

Labels

area-Infrastructure

Projects

Runtime Infra

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate ref packs are getting uploaded

Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts


3 participants

@jkoritzinsky@ViktorHofer@mmitche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Publish dotnet/runtime using standard Arcade Publish tooling - #111934
Merged
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Mar 7, 2025
Merged
Publish dotnet/runtime using standard Arcade Publish tooling#111934
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Conversation
@jkoritzinsky
@jkoritzinskyjkoritzinsky
commented
Jan 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:
  • In a given build, publish all packages that were produced in the current build that contain that job's target RID.
  • For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
  • For some jobs (specified in runtime-official), publish all packages produced.
  • In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
  • Outside the VMR, skip publishing all "packages not to be published" packages.
Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793 As per #111934 (comment), we are no longer blocked on this.
Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.
Fixes#102672
Fixes#1719
@jkoritzinsky
Remove prepare-signed-artifacts and publish from each leg directly.
0364ee2
@jkoritzinsky
Publish in a separate stage to make dependency tracking easy
7e284a4
@jkoritzinsky
Set _SignType so we can pass -sign
d2c9cf8
@jkoritzinsky
Remove old validation target
3338d4d
@jkoritzinsky
!drop remove -sign so we can figure out publishing first
01bca60
@jkoritzinsky
Remove InitialTargets
88ee36c
@jkoritzinsky
Only generate product version files once. Fix asset manifest extensio…
d2fa244
…n. Don't copy workloads packages as they're already in the right format now.
@jkoritzinsky
Completely restructure how we publish to be more explicit that specif…
f93b7e4
…ic jobs publish specific packages
@jkoritzinsky
Move "non-RID-specific" special packages logic into Publishing.props
8ee9ac1
@jkoritzinsky
Make sure we build the list of skipped packages after we add the RID-…
4455301
…specific packages
@jkoritzinsky
Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…
e36e87e
…es. Use that when possible for unique names and produce our own names when necessary.
@jkoritzinsky
Extract handling hostOS and hostArch into another props file
747bffe
@jkoritzinsky
Fix import and adjust condition for SB
fbe3515
@jkoritzinsky
Fix asset manifest name uniqueness to ensue we don't overwrite asset …
6bbbc4c
…manifests.
@jkoritzinsky
Set AssetManifestFilePath again
c1068eb
@jkoritzinsky
Fix PublishAllPackages condition and make sure to publish the VS redi…
9e61e42
…st packages
@jkoritzinsky
Explicitly look up packages as we add zips and other blob artifacts t…
05fe23b
…o the Artifacts items in Signing.props
@jkoritzinsky
Search in the right folder.
d8b032a
@jkoritzinsky
Look up zips in VSSetup for workloads
2ac88a1
@jkoritzinsky
Add missing semicolon
552bb1d
@jkoritzinsky
Include VSSetup artifacts separately
9247027
@jkoritzinsky
Re-enable signing (depends on Microbuild signing on Linux/Mac)
1eb6a85
@jkoritzinsky
Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…
72af6e0
…-publish
@jkoritzinskyjkoritzinsky
added

blocked
Issue/PR is blocked on something - see comments
area-Infrastructure

labels
Jan 28, 2025
@build-analysisbuild-analysisBot
mentioned this pull request
Jan 29, 2025

Closed

ViktorHofer
ViktorHofer
reviewed
Jan 29, 2025
View reviewed changes
Comment threadeng/pipelines/runtime-official.yml
Outdated

@jkoritzinsky
jkoritzinsky
commented
Mar 6, 2025

Copy link


Copy Markdown

MemberAuthor
I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.
Here's my basic approach:
  • Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
  • Runtime verticals won't sign in the runtime official build
  • We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
  • We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.
I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results
@jkoritzinskyjkoritzinsky
removed
the 
blocked
Issue/PR is blocked on something - see comments
label
Mar 6, 2025
@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results
@jkoritzinskyjkoritzinsky
mentioned this pull request
Mar 7, 2025
 Merged

ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/DotNetBuild.props

Comment on lines

+45
 to 
+52

 <PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
 <!-- Source-build always needs all artifacts to be published. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
 <!-- Short-stack builds should always only publish RID-specific artifacts. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
 <!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
 </PropertyGroup>
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the VMR controls are in, I'll come back and remove these.
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.
ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/Signing.props
Outdated

@jkoritzinsky@ViktorHofer
Update Signing.props
5312597
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

3 tasks
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
/ba-g timeouts unrelated
@jkoritzinsky
jkoritzinsky
merged commit 891ff91
into
dotnet:mainMar 7, 2025
@jkoritzinsky
jkoritzinsky
deleted the

standard-arcade-publish

branch
March 7, 2025 22:55
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 7, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@ViktorHoferViktorHoferViktorHofer approved these changes
@mmitchemmitchemmitche approved these changes
@akoeplingerakoeplingerAwaiting requested review from akoeplinger

Assignees

@jkoritzinskyjkoritzinsky

Labels

area-Infrastructure

Projects

Runtime Infra

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate ref packs are getting uploaded

Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts


3 participants

@jkoritzinsky@ViktorHofer@mmitche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Publish dotnet/runtime using standard Arcade Publish tooling - #111934
Merged
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Mar 7, 2025
Merged
Publish dotnet/runtime using standard Arcade Publish tooling#111934
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Conversation
@jkoritzinsky
@jkoritzinskyjkoritzinsky
commented
Jan 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:
  • In a given build, publish all packages that were produced in the current build that contain that job's target RID.
  • For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
  • For some jobs (specified in runtime-official), publish all packages produced.
  • In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
  • Outside the VMR, skip publishing all "packages not to be published" packages.
Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793 As per #111934 (comment), we are no longer blocked on this.
Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.
Fixes#102672
Fixes#1719
@jkoritzinsky
Remove prepare-signed-artifacts and publish from each leg directly.
0364ee2
@jkoritzinsky
Publish in a separate stage to make dependency tracking easy
7e284a4
@jkoritzinsky
Set _SignType so we can pass -sign
d2c9cf8
@jkoritzinsky
Remove old validation target
3338d4d
@jkoritzinsky
!drop remove -sign so we can figure out publishing first
01bca60
@jkoritzinsky
Remove InitialTargets
88ee36c
@jkoritzinsky
Only generate product version files once. Fix asset manifest extensio…
d2fa244
…n. Don't copy workloads packages as they're already in the right format now.
@jkoritzinsky
Completely restructure how we publish to be more explicit that specif…
f93b7e4
…ic jobs publish specific packages
@jkoritzinsky
Move "non-RID-specific" special packages logic into Publishing.props
8ee9ac1
@jkoritzinsky
Make sure we build the list of skipped packages after we add the RID-…
4455301
…specific packages
@jkoritzinsky
Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…
e36e87e
…es. Use that when possible for unique names and produce our own names when necessary.
@jkoritzinsky
Extract handling hostOS and hostArch into another props file
747bffe
@jkoritzinsky
Fix import and adjust condition for SB
fbe3515
@jkoritzinsky
Fix asset manifest name uniqueness to ensue we don't overwrite asset …
6bbbc4c
…manifests.
@jkoritzinsky
Set AssetManifestFilePath again
c1068eb
@jkoritzinsky
Fix PublishAllPackages condition and make sure to publish the VS redi…
9e61e42
…st packages
@jkoritzinsky
Explicitly look up packages as we add zips and other blob artifacts t…
05fe23b
…o the Artifacts items in Signing.props
@jkoritzinsky
Search in the right folder.
d8b032a
@jkoritzinsky
Look up zips in VSSetup for workloads
2ac88a1
@jkoritzinsky
Add missing semicolon
552bb1d
@jkoritzinsky
Include VSSetup artifacts separately
9247027
@jkoritzinsky
Re-enable signing (depends on Microbuild signing on Linux/Mac)
1eb6a85
@jkoritzinsky
Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…
72af6e0
…-publish
@jkoritzinskyjkoritzinsky
added

blocked
Issue/PR is blocked on something - see comments
area-Infrastructure

labels
Jan 28, 2025
@build-analysisbuild-analysisBot
mentioned this pull request
Jan 29, 2025

Closed

ViktorHofer
ViktorHofer
reviewed
Jan 29, 2025
View reviewed changes
Comment threadeng/pipelines/runtime-official.yml
Outdated

@jkoritzinsky
jkoritzinsky
commented
Mar 6, 2025

Copy link


Copy Markdown

MemberAuthor
I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.
Here's my basic approach:
  • Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
  • Runtime verticals won't sign in the runtime official build
  • We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
  • We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.
I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results
@jkoritzinskyjkoritzinsky
removed
the 
blocked
Issue/PR is blocked on something - see comments
label
Mar 6, 2025
@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results
@jkoritzinskyjkoritzinsky
mentioned this pull request
Mar 7, 2025
 Merged

ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/DotNetBuild.props

Comment on lines

+45
 to 
+52

 <PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
 <!-- Source-build always needs all artifacts to be published. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
 <!-- Short-stack builds should always only publish RID-specific artifacts. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
 <!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
 </PropertyGroup>
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the VMR controls are in, I'll come back and remove these.
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.
ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/Signing.props
Outdated

@jkoritzinsky@ViktorHofer
Update Signing.props
5312597
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

3 tasks
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
/ba-g timeouts unrelated
@jkoritzinsky
jkoritzinsky
merged commit 891ff91
into
dotnet:mainMar 7, 2025
@jkoritzinsky
jkoritzinsky
deleted the

standard-arcade-publish

branch
March 7, 2025 22:55
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 7, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@ViktorHoferViktorHoferViktorHofer approved these changes
@mmitchemmitchemmitche approved these changes
@akoeplingerakoeplingerAwaiting requested review from akoeplinger

Assignees

@jkoritzinskyjkoritzinsky

Labels

area-Infrastructure

Projects

Runtime Infra

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate ref packs are getting uploaded

Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts


3 participants

@jkoritzinsky@ViktorHofer@mmitche

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Publish dotnet/runtime using standard Arcade Publish tooling - #111934
Merged
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Mar 7, 2025
Merged
Publish dotnet/runtime using standard Arcade Publish tooling#111934
jkoritzinsky merged 84 commits into
dotnet:mainfrom 
jkoritzinsky:standard-arcade-publish
Conversation
@jkoritzinsky
@jkoritzinskyjkoritzinsky
commented
Jan 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Publish the assets from dotnet/runtime using the standard Arcade publish tooling. This tooling doesn't allow any duplicate packages, so change our publishing logic to the following:
  • In a given build, publish all packages that were produced in the current build that contain that job's target RID.
  • For some target RIDs, also publish an additional subset of packages (specified in Publishing.props).
  • For some jobs (specified in runtime-official), publish all packages produced.
  • In the VMR, mark all "packages not to be published" as "Vertical" visibility so they're still available upstack.
  • Outside the VMR, skip publishing all "packages not to be published" packages.
Blocked on MicroBuild fixes for Linux and Mac: dotnet/source-build#4793 As per #111934 (comment), we are no longer blocked on this.
Verified in https://dev.azure.com/dnceng/internal/_build/results?buildId=2629649&view=results that all of the same assets that we expect to produce in a runtime official build are produced on this branch.
Fixes#102672
Fixes#1719
@jkoritzinsky
Remove prepare-signed-artifacts and publish from each leg directly.
0364ee2
@jkoritzinsky
Publish in a separate stage to make dependency tracking easy
7e284a4
@jkoritzinsky
Set _SignType so we can pass -sign
d2c9cf8
@jkoritzinsky
Remove old validation target
3338d4d
@jkoritzinsky
!drop remove -sign so we can figure out publishing first
01bca60
@jkoritzinsky
Remove InitialTargets
88ee36c
@jkoritzinsky
Only generate product version files once. Fix asset manifest extensio…
d2fa244
…n. Don't copy workloads packages as they're already in the right format now.
@jkoritzinsky
Completely restructure how we publish to be more explicit that specif…
f93b7e4
…ic jobs publish specific packages
@jkoritzinsky
Move "non-RID-specific" special packages logic into Publishing.props
8ee9ac1
@jkoritzinsky
Make sure we build the list of skipped packages after we add the RID-…
4455301
…specific packages
@jkoritzinsky
Arcade now handles TargetOS/TargetArchitecture for asset manifest nam…
e36e87e
…es. Use that when possible for unique names and produce our own names when necessary.
@jkoritzinsky
Extract handling hostOS and hostArch into another props file
747bffe
@jkoritzinsky
Fix import and adjust condition for SB
fbe3515
@jkoritzinsky
Fix asset manifest name uniqueness to ensue we don't overwrite asset …
6bbbc4c
…manifests.
@jkoritzinsky
Set AssetManifestFilePath again
c1068eb
@jkoritzinsky
Fix PublishAllPackages condition and make sure to publish the VS redi…
9e61e42
…st packages
@jkoritzinsky
Explicitly look up packages as we add zips and other blob artifacts t…
05fe23b
…o the Artifacts items in Signing.props
@jkoritzinsky
Search in the right folder.
d8b032a
@jkoritzinsky
Look up zips in VSSetup for workloads
2ac88a1
@jkoritzinsky
Add missing semicolon
552bb1d
@jkoritzinsky
Include VSSetup artifacts separately
9247027
@jkoritzinsky
Re-enable signing (depends on Microbuild signing on Linux/Mac)
1eb6a85
@jkoritzinsky
Merge branch 'main' of github.com:dotnet/runtime into standard-arcade…
72af6e0
…-publish
@jkoritzinskyjkoritzinsky
added

blocked
Issue/PR is blocked on something - see comments
area-Infrastructure

labels
Jan 28, 2025
@build-analysisbuild-analysisBot
mentioned this pull request
Jan 29, 2025

Closed

ViktorHofer
ViktorHofer
reviewed
Jan 29, 2025
View reviewed changes
Comment threadeng/pipelines/runtime-official.yml
Outdated

@jkoritzinsky
jkoritzinsky
commented
Mar 6, 2025

Copy link


Copy Markdown

MemberAuthor
I've updated this PR to provide a mechanism for us to merge it today and get rid of the long-running patch.
Here's my basic approach:
  • Instead of publishing to AzDO, each runtime vertical will instead publish to disk with a layout like the artifacts directory.
  • Runtime verticals won't sign in the runtime official build
  • We bring back the final join job. This job downloads all of the artifacts from each runtime vertical into its artifacts directory and then does a standard arcade sign/publish with the same signing rules as exist in main.
  • We still get rid of IntermediateArtifacts and instead use separate pipeline artifacts for each vertical.
I have a passing official build from this PR here: https://dev.azure.com/dnceng/internal/_build/results?buildId=2657037&view=results
@jkoritzinskyjkoritzinsky
removed
the 
blocked
Issue/PR is blocked on something - see comments
label
Mar 6, 2025
@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
New official build for a314fc3 at https://dev.azure.com/dnceng/internal/_build/results?buildId=2657219&view=results
@jkoritzinskyjkoritzinsky
mentioned this pull request
Mar 7, 2025
 Merged

ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/DotNetBuild.props

Comment on lines

+45
 to 
+52

 <PropertyGroup Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">
 <!-- Source-build always needs all artifacts to be published. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(DotNetBuildSourceOnly)' != ''">false</EnableDefaultRidSpecificArtifacts>
 <!-- Short-stack builds should always only publish RID-specific artifacts. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(ShortStack)' == 'true'">true</EnableDefaultRidSpecificArtifacts>
 <!-- If no override is specified, don't use RID-specific publishing. Instead, publish everything. -->
 <EnableDefaultRidSpecificArtifacts Condition="'$(EnableDefaultRidSpecificArtifacts)' == ''">false</EnableDefaultRidSpecificArtifacts>
 </PropertyGroup>
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These settings make sense to me but wouldn't they better fit into the VMR orchestrator as they aren't specific to runtime?
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they do fit better in the VMR. However, we need to put them here first so they don't break the VMR until I change the VMR to pass these options down (also in PR)
@jkoritzinskyjkoritzinskyMar 7, 2025

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once the VMR controls are in, I'll come back and remove these.
@ViktorHoferViktorHoferMar 7, 2025

Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider filing a tracking issue and linking to it. Also, so that others can follow along and understand the potential for improvement in these code paths.
ViktorHofer
ViktorHofer
reviewed
Mar 7, 2025
View reviewed changes
Comment threadeng/Signing.props
Outdated

@jkoritzinsky@ViktorHofer
Update Signing.props
5312597
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

3 tasks
@build-analysisbuild-analysisBot
mentioned this pull request
Mar 7, 2025

Closed

@jkoritzinsky
jkoritzinsky
commented
Mar 7, 2025

Copy link


Copy Markdown

MemberAuthor
/ba-g timeouts unrelated
@jkoritzinsky
jkoritzinsky
merged commit 891ff91
into
dotnet:mainMar 7, 2025
@jkoritzinsky
jkoritzinsky
deleted the

standard-arcade-publish

branch
March 7, 2025 22:55
@github-actionsgithub-actionsBot
locked and limited conversation to collaborators
Apr 7, 2025
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers

@ViktorHoferViktorHoferViktorHofer approved these changes
@mmitchemmitchemmitche approved these changes
@akoeplingerakoeplingerAwaiting requested review from akoeplinger

Assignees

@jkoritzinskyjkoritzinsky

Labels

area-Infrastructure

Projects

Runtime Infra

Status: Done


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Duplicate ref packs are getting uploaded

Migrate official build "Prepare Signed Artifacts" from Build to Pipeline Artifacts


3 participants

@jkoritzinsky@ViktorHofer@mmitche