source-build: support building runtime using non-portable runtime packages. by tmds · Pull Request #75597 · dotnet/runtime

tmds · 2022-09-14T09:26:10Z

Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

cc @omajid @MichaelSimons @crummel @ViktorHofer @ericstj @jkotas

ghost · 2022-09-14T09:26:18Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

tmds · 2022-09-14T09:33:21Z

This isn't working fully.

For some reason ILCompiler.csproj and crossgen2.csproj go look for Microsoft.AspNetCore.App.Runtime.<rid>.

 /home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Runtime.linux-x64 with version (= 7.0.0-rc.2.22452.11) [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102: - Found 1 version(s) in previously-source-built [ Nearest version: 7.0.0-rc.1.22404.6 ] [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102: - Found 0 version(s) in source-built [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102: - Found 0 version(s) in prebuilt [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/ILCompiler/ILCompiler.csproj : error NU1102: - Found 0 version(s) in reference-packages [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102: Unable to find package Microsoft.AspNetCore.App.Runtime.linux-x64 with version (= 7.0.0-rc.2.22452.11) [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102: - Found 1 version(s) in previously-source-built [ Nearest version: 7.0.0-rc.1.22404.6 ] [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102: - Found 0 version(s) in source-built [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102: - Found 0 version(s) in prebuilt [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]
/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/src/coreclr/tools/aot/crossgen2/crossgen2.csproj : error NU1102: - Found 0 version(s) in reference-packages [/home/tmds/tarball/src/runtime/artifacts/source-build/self/src/Build.proj]

I don't understand where this reference comes from, but probably we don't actually need this?
@ericstj @ViktorHofer how can I handle this?

tmds · 2022-09-15T07:05:08Z

For some reason ILCompiler.csproj and crossgen2.csproj go look for Microsoft.AspNetCore.App.Runtime..

Apparently the reference happens as soon as you add <RuntimeIdentifier> based on the KnownFrameworkReferences.
The fix is to remove the reference for Microsoft.AspNetCore.App.

ghost · 2022-09-15T17:52:46Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

cc @omajid @MichaelSimons @crummel @ViktorHofer @ericstj @jkotas

Author:	tmds
Assignees:	-
Labels:	`area-Infrastructure`, `community-contribution`
Milestone:	-

ghost · 2022-09-15T18:23:22Z

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

cc @omajid @MichaelSimons @crummel @ViktorHofer @ericstj @jkotas

Author:	tmds
Assignees:	-
Labels:	`area-Infrastructure-coreclr`, `community-contribution`
Milestone:	-

ghost · 2022-09-15T18:23:51Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable).

With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

cc @omajid @MichaelSimons @crummel @ViktorHofer @ericstj @jkotas

Author:	tmds
Assignees:	-
Labels:	`area-Infrastructure`, `community-contribution`
Milestone:	-

tmds · 2022-09-20T06:35:50Z

@ViktorHofer @jkotas I took a step back and wrote up an issue that describes the goal: dotnet/source-build#3027. And made a PR that includes these changes as a patch to help validate it: dotnet/installer#14549.

@ViktorHofer I will address your feedback this week.

Besides this, we also need #74721 (comment).

ViktorHofer · 2022-09-20T08:57:03Z

@jkotas who would be the right person to review the ILCompiler and Crossgen2 changes?

jkotas · 2022-09-20T18:47:17Z

@jkotas who would be the right person to review the ILCompiler and Crossgen2 changes?

I can take a look or route as necessary once the feedback is addressed.

…kages. Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable). With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build.

tmds · 2022-10-11T12:41:37Z

CI failed because of an issue with the dnceng nuget feed.

@ViktorHofer can you trigger CI, and also take another look at this PR?

ViktorHofer · 2022-10-11T14:23:46Z

Closing and reopening usually does the trick and retriggers all the legs.

tmds · 2022-10-12T06:23:02Z

@ViktorHofer since this has Condition=" '$(DotNetBuildFromSource)' == 'true' " and the integration test in dotnet/installer#14549 is passing, I think we can consider CI green on this.

ViktorHofer · 2022-10-12T08:43:53Z

Added a style nit commit. Also, to retrigger CI as there were more than 20 failing legs.

ViktorHofer · 2022-10-12T18:52:36Z

Thanks a lot, Tom!

…kages (backport of dotnet#75597)

tmds · 2022-11-09T13:01:54Z

@ViktorHofer can you initiate backports to 7.0 for this and the other 2 related PRs: #74504, #76068.
Can the bot handle these?

ViktorHofer · 2022-11-09T16:20:58Z

We just merged some of these into release/6.0. Ideally, we would have backported these changes at the same into release/6.0 and release/7.0 as we now need another set of approvals.

That said, yes, I will cherry-pick them together into a single PR and add the servicing template and ask for an approval.

…kages (backport of #75597) (#77509)

tmds · 2022-11-14T07:33:07Z

Sorry for the overhead, and thank you for taking care of it!

…kages. (#75597) * source-build: support building runtime using non-portable runtime packages. Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable). With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build. * Avoid references to Microsoft.AspNetCore.App.<rid>. * crossgen2: set RuntimeIdentifier explicitly. * PR feedback * Revert "crossgen2: set RuntimeIdentifier explicitly." This reverts commit 10be274. * PR feedback. * Revert changes to ILCompiler.csproj. * Update Directory.Build.targets Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>

* Use generated runtime.json when building shared framework packages. (#76068) * Use generated runtime.json when building shared framework packages. * Don't UpdateRuntimeJson on Build. * PR feedback. * Update Microsoft.NETCore.Platforms.csproj Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> * source-build: support building runtime using non-portable runtime packages. (#75597) * source-build: support building runtime using non-portable runtime packages. Currently source-build performs a 'runtime-portable' build that produces 'linux-{arch}' packages that are used when building target runtime (non-portable). With this change, we can use the non-portable packages that are produced by a previous (non-portable) 'runtime' build. This helps eliminate the 'runtime-portable' build. * Avoid references to Microsoft.AspNetCore.App.<rid>. * crossgen2: set RuntimeIdentifier explicitly. * PR feedback * Revert "crossgen2: set RuntimeIdentifier explicitly." This reverts commit 10be274. * PR feedback. * Revert changes to ILCompiler.csproj. * Update Directory.Build.targets Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> * source-build: set the properties to build for mono runtime. (#77786) * Pass TargetRid and SourceBuildNonPortable to the native scripts (#74504) * Pass SourceBuild TargetRid and SourceBuildPortable args through the native script. * Rename -packagerid to -distrorid. * Fix init-distro-rid.sh. * Use OutputRid instead of PackageRid. * Drop unnecessary Condition. * Try passing --distrorid as OutputRid. * NativeExport: use PackageRID for AppHostRuntimeIdentifier. * Rename --distrorid to --outputrid. * Fix SourceBuild.props. * Undo changes to init-distro-rid.sh. * Let source-build leg build 'banana-x64' rid. * SourceBuild.props: don't force a RuntimeOS. * SourceBuild.props: set AdditionalRuntimeIdentifierParent. * Build banana.24-x64 instead. * Fix SourceBuild.props. * SourceBuild.props: try fix AdditionalRuntimeIdentifier* usage. * source-build.yml: add runtimeOS parameter. * SourceBuild.props: derive RuntimeOS from NETCoreSdkRuntimeIdentifier. * Undo using NETCoreSdkRuntimeIdentifier. * Update eng/pipelines/common/global-build-job.yml Co-authored-by: Jeremy Koritzinsky <jkoritzinsky@gmail.com> Co-authored-by: Jeremy Koritzinsky <jkoritzinsky@gmail.com> Co-authored-by: Tom Deseyn <tom.deseyn@gmail.com> Co-authored-by: Jeremy Koritzinsky <jkoritzinsky@gmail.com>

ghost added the community-contribution Indicates that the PR has been added by a community member label Sep 14, 2022

tmds commented Sep 14, 2022
View reviewed changes

Comment threadsrc/coreclr/tools/aot/ILCompiler/ILCompiler.csproj Outdated

tmds mentioned this pull request Sep 14, 2022
Microsoft.NETCore.App.Crossgen2.sfxproj in source-build #74721
Closed

tmds mentioned this pull request Sep 15, 2022
Eliminate the 'runtime-portable' build dotnet/source-build#3027
Closed
5 tasks

jkotas requested a review from ViktorHofer September 15, 2022 17:52

jkotas added the area-Infrastructure label Sep 15, 2022

ViktorHofer added area-Infrastructure-coreclr Only use for closed issues and removed area-Infrastructure labels Sep 15, 2022

ViktorHofer added area-Infrastructure and removed area-Infrastructure-coreclr Only use for closed issues labels Sep 15, 2022

ViktorHofer reviewed Sep 15, 2022
View reviewed changes

Comment threadDirectory.Build.targets Outdated

ViktorHofer reviewed Sep 15, 2022
View reviewed changes

Comment threadDirectory.Build.targets Outdated

ViktorHofer reviewed Sep 15, 2022
View reviewed changes

Comment threadDirectory.Build.targets Outdated

ViktorHofer reviewed Sep 15, 2022
View reviewed changes

Comment threadDirectory.Build.targets Outdated

tmds mentioned this pull request Sep 20, 2022
Eliminate portable runtime build. dotnet/installer#14549
Merged

MichaelSimons reviewed Sep 20, 2022
View reviewed changes

Comment threadsrc/coreclr/tools/aot/crossgen2/crossgen2.csproj Outdated

hoyosjs reviewed Sep 21, 2022
View reviewed changes

Comment threadDirectory.Build.targets Outdated

tmds added 4 commits September 28, 2022 12:51

Avoid references to Microsoft.AspNetCore.App.<rid>.
73d3229

crossgen2: set RuntimeIdentifier explicitly.
10be274

PR feedback
4e5e662

build-analysisBot mentioned this pull request Oct 11, 2022
503's dotnet/arcade#10943
Closed
2 tasks

ViktorHofer closed this Oct 11, 2022

ViktorHofer reopened this Oct 11, 2022

This was referenced Oct 11, 2022
Azure artifacts restore failures dotnet/arcade#11233
Closed
Tracking issue for CI build timeouts #76454
Closed
Nuget restore error caused by 503s dotnet/arcade#11239
Closed

tmds closed this Oct 12, 2022

tmds reopened this Oct 12, 2022

build-analysisBot mentioned this pull request Oct 12, 2022
Tracking Nuget 429s dotnet/arcade#10885
Closed
2 tasks

ViktorHofer reviewed Oct 12, 2022
View reviewed changes

Comment threadDirectory.Build.targets Outdated

Update Directory.Build.targets
cefa413

ViktorHofer approved these changes Oct 12, 2022
View reviewed changes

ViktorHofer merged commit e2bee8a into dotnet:mainOct 12, 2022

ayakael mentioned this pull request Oct 26, 2022
[release/6.0] source-build: support building runtime using non-portable runtime packages (backport of #75597) #77509
Merged

ayakael added a commit to ayakael/runtime that referenced this pull request Nov 5, 2022
source-build: support building runtime using non-portable runtime pac…
884c70d
…kages (backport of dotnet#75597)

carlossanlop pushed a commit that referenced this pull request Nov 9, 2022
source-build: support building runtime using non-portable runtime pac…
d47faf2
…kages (backport of #75597) (#77509)

ghost locked as resolved and limited conversation to collaborators Dec 14, 2022

Uh oh!

Conversation

tmds commented Sep 14, 2022

Uh oh!

ghost commented Sep 14, 2022

Uh oh!

Uh oh!

tmds commented Sep 14, 2022

Uh oh!

tmds commented Sep 15, 2022

Uh oh!

ghost commented Sep 15, 2022

Uh oh!

ghost commented Sep 15, 2022

Uh oh!

ghost commented Sep 15, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tmds commented Sep 20, 2022

Uh oh!

ViktorHofer commented Sep 20, 2022

Uh oh!

Uh oh!

jkotas commented Sep 20, 2022

Uh oh!

Uh oh!

tmds commented Oct 11, 2022

Uh oh!

ViktorHofer commented Oct 11, 2022

Uh oh!

tmds commented Oct 12, 2022

Uh oh!

Uh oh!

ViktorHofer commented Oct 12, 2022

Uh oh!

ViktorHofer commented Oct 12, 2022

Uh oh!

tmds commented Nov 9, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ViktorHofer commented Nov 9, 2022

Uh oh!

tmds commented Nov 14, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tmds commented Nov 9, 2022•
edited
Loading