Rev Alpine WithNode from 3.9 to 3.13 by am11 · Pull Request #57324 · dotnet/runtime

am11 · 2021-08-12T22:17:23Z

Also,

update helix queues from 3.12 to 3.13
add 3.14 helix queues
update performance job to install cargo from main branch

ghost · 2021-08-12T22:17:28Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

Updating to 3.14 now will buy us ~2 years before this latest version reaches EOL.
Upstream support for 3.14 was added: dotnet/dotnet-buildtools-prereqs-docker@d90babf

Fixes #56672

Author:	am11
Assignees:	-
Labels:	`area-Infrastructure`
Milestone:	-

am11 · 2021-08-12T22:17:52Z

cc @jkotas, @janvorli we can also update helix queues to test the latest Alpine Linux version, but I haven't changed that as part of this PR.

jkotas · 2021-08-12T22:25:36Z

According to https://github.com/dotnet/core/blob/main/release-notes/6.0/supported-os.md#linux, we want to support Alpine 3.13+ for .NET 6. Are binaries built on Alpine 3.14 going to run on Alpine 3.13?

am11 · 2021-08-12T22:28:09Z

Are binaries built on Alpine 3.14 going to run on Alpine 3.13?

They should work (as time64 changes were included in 3.13). I will test it.

am11 · 2021-08-12T23:52:32Z

Locally I was able to build the main branch without any changes:

docker run -v $(pwd):/runtime \
-it mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf \
/runtime/src/coreclr/build-runtime.sh checked x64

but on CI it was failing to find python while it is clearly there, but when i worked around it, it is now failing with Failed to get PGO data package path. There is nothing in logs pointing to clues what's wrong, so it might be something specific to pipeline YAML that needs updating?

hoyosjs · 2021-08-13T01:11:11Z

@am11 looks like the container doesn't have curl? (and maybe doing the volume mount masks this as you may have a .dotnet directory?

am11 · 2021-08-13T01:20:10Z

@am11 looks like the container doesn't have curl? (and maybe doing the volume mount masks this as you may have a .dotnet directory?

@hoyosjs, I tried it with clean git repo. Even without mounting, git clone <runtime url> followed by build-runtime.sh release x64 succeeds in that container.

hoyosjs · 2021-08-13T20:18:43Z

I tried with copied IDs from the build logs and I was still able to get the SDK.

am11 · 2021-08-14T02:51:51Z

Yes, the docker container mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf is the one failing CI leg is using which I tested earlier. It has all the required stuff and capable of building runtime. Something else in AzDo is apparently going wrong. Nothing jumping out at me from logs of Initialize containers step, all looks normal until Build CoreCLR Runtime step, which seems to be using a different environment.

ViktorHofer · 2021-08-16T15:16:02Z

@am11 please let us know if we can help anyhow.

am11 · 2021-08-16T15:45:15Z

@ViktorHofer, it is still a mystery to me as CI error is not reproducible locally. e.g. this leg https://dev.azure.com/dnceng/public/_build/results?buildId=1291489&view=logs&jobId=0e64859c-a870-5a74-6b6e-333fc1003298&j=0e64859c-a870-5a74-6b6e-333fc1003298&t=5da9eef6-c013-592b-ccae-2e46c52786d0 is apparently using container mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf and failing on build step due to missing curl. Although the said container has curl, python along rest of the dependencies.

I just (temporarily) deleted > /dev/null 2>&1 to reveal the missing curl error from

runtime/src/coreclr/build-runtime.sh

Lines 55 to 58 in 6acb72b

"$__RepoRootDir/eng/common/msbuild.sh" /clp:nosummary $__ArcadeScriptArgs$OptDataProjectFilePath$RestoreArg /t:DumpPgoDataPackagePath \ 
${__CommonMSBuildArgs} /p:PgoDataPackagePathOutputFile=${PgoDataPackagePathOutputFile} \ 
 -bl:"$__LogsDir/PgoVersionRead_$__ConfigTriplet.binlog"> /dev/null 2>&1
local exit_code="$?"

If we test it locally with same command used by this failing CI leg:

docker run \
-it mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf \
sh -c 'git clone https://github.com/dotnet/runtime --single-branch --depth 1 /runtime &&
/runtime/src/coreclr/build-runtime.sh checked x64'

it builds coreclr successfully. Do you see anything unusual from the logs that could be causing Build CoreCLR Runtime step to use different environment than the container?

ViktorHofer · 2021-08-16T16:01:34Z

I'll take a look. Let me also cc @safern and @jkoritzinsky

ViktorHofer · 2021-08-20T14:51:12Z

@hoyosjs @jkoritzinsky any other ideas? I'm currently a bit short on time, maybe you have some additional ideas?

ViktorHofer · 2021-09-01T16:06:49Z

@MattGal do you think you can help here?

MattGal · 2021-09-01T16:12:01Z

@MattGal do you think you can help here?

No guarantees but sure, I'll take a look

MattGal · 2021-09-01T17:08:07Z

@ViktorHofer I can't reproduce this running main build inside a mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.14-WithNode-20210812132848-d90babf container, it may be a transient error?

hoyosjs · 2021-09-01T17:47:47Z

~~Sigh. Looks like transient - cache or something. Kicked off again and the containers initialized just fine...~~ spoke too soon. Once more failed with the same error as three weeks back.

MattGal · 2021-09-01T17:54:35Z

Super weird but does seem to be noticed and known externally: dotnet/install-scripts#206.

I'm running the same image you used to run the same CI build from a Windows box. I wonder if you started from an Ubuntu 18.04/20.04 if you could get a repro.

hoyosjs · 2021-09-01T18:15:02Z

I couldn't repro on a physical Ubuntu box with that container on top.

MattGal · 2021-09-01T18:22:02Z

I couldn't repro on a physical Ubuntu box with that container on top.

Maybe just try alpine-3.13-WithNode-20210812132854-ddfc481? since , even if this works after a bunch of retries you don't want to merge like this...

am11 · 2021-09-01T21:04:07Z

even if this works after a bunch of retries

Flakiness is really not the problem. On other systems, you can build using this new container reliably and in AzDO it is always failing to run build step inside the container (it is apparently running build step on host system rather than the container). Logs are not indicating any problem.

Also, * update helix queues from 3.12 to 3.13 * add 3.14 helix queues * update performance job to install cargo from main branch

safern · 2021-09-24T19:29:25Z


 # Linux musl x64
 - ${{ if eq(parameters.platform, 'Linux_musl_x64') }}:
- - ${{ if eq(parameters.jobParameters.isFullMatrix, false) }}:


can we test on 3.13 on fullMatrix == true? Meaning on rolling CI?

Yup, didn't mean to press delete on rolling ones 🤣

safern

Thanks 🎉

hoyosjs

Thanks a lot @am11

safern · 2021-09-24T22:18:37Z

It seems like you have a yaml syntax error 😢 : https://dev.azure.com/dnceng/public/_build/results?buildId=1384687&view=results

aik-jahoda · 2021-09-29T15:08:38Z

There is a JSON failure on Alpine.314.Arm64.Open. @safern do you think the failure is related to the change or we can merge this PR?

danmoseley · 2021-09-29T15:24:13Z

The JSON crash has a dump. We should look at that dump -- there is a how-to-debug-dump.md file there to help.

The regex issue is known and fixed already.

am11 · 2021-09-29T15:46:58Z

There is a JSON failure on Alpine.314.Arm64.Open

Isn't it #57198?

safern · 2021-09-29T18:53:18Z

I think we should at least look at the dump to make sure if it is: #57198 before merging something that could potentially introduce a failure to CI.

am11 · 2021-09-29T19:18:22Z

The dump link on that issue is expired and Eirik had no success with clrstack when link was active (#57198 (comment)). Not sure how I can determine it.

Maybe rerun the job to see if the issue persist?

safern · 2021-09-29T19:19:16Z

Maybe rerun the job to see if the issue persist?

Sounds good to me.

danmoseley · 2021-09-29T19:30:35Z

Dump links from #57324 are still active, they're only hours old.

Console Logs

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57324-merge-9e9d1cbb6f224c0492/System.Text.Json.Tests/1/how-to-debug-dump.md?sv=2019-07-07&se=2021-10-19T14%3A08%3A15Z&sr=c&sp=rl&sig=wKHlTG%2F2YMVkeicpVl0W%2FlyaZ4RfBE74SXPTyvUs6b0%3D

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-57324-merge-9e9d1cbb6f224c0492/System.Text.Json.Tests/1/core.1001.23?sv=2019-07-07&se=2021-10-19T14%3A08%3A15Z&sr=c&sp=rl&sig=wKHlTG%2F2YMVkeicpVl0W%2FlyaZ4RfBE74SXPTyvUs6b0%3D

from
https://dev.azure.com/dnceng/public/_build/results?buildId=1385273&view=ms.vss-test-web.build-test-results-tab&runId=40500360&paneView=dotnet-dnceng.dnceng-build-release-tasks.helix-test-information-tab&resultId=185830

I believe it requires a certain access token to pull part of what you need down so @am11 you maybe can't.

am11 · 2021-09-29T19:34:06Z

Dump links from #57324 are still active, they're only hours old.

Yup, I can download dump from this PR but the one from #57198 (which I would like to compare with) is expired.

danmoseley · 2021-09-29T19:50:13Z

Ah gotcha. Is there anything that can be gotten out of this dump?

am11 · 2021-09-29T19:51:52Z

I am working on it. 🙂

am11 · 2021-09-29T22:03:37Z

clrstack results are posted here: #57198 (comment). I have the container saved, in case more info is needed.

aik-jahoda · 2021-09-30T08:14:08Z

The regex issue is known and fixed already.

clrstack results are posted here: #57198 (comment). I have the container saved, in case more info is needed.

Sounds like both test failures are known and tracked/fixed already. Can we merge this PR?

ghost added area-Infrastructure community-contribution Indicates that the PR has been added by a community member labels Aug 12, 2021

am11 marked this pull request as draft August 12, 2021 22:28

ViktorHofer mentioned this pull request Aug 16, 2021
Alpine builds use alpine-3.9 docker image that is EOL #56672
Closed

ViktorHofer added this to the 6.0.0 milestone Aug 20, 2021

ViktorHofer mentioned this pull request Aug 23, 2021
Update unsupported distros in helix queues to latest and add missing ones #57947
Closed
14 tasks

am11 force-pushed the feature/ci/alpine3.14 branch 2 times, most recently from 37bf805 to 2888eafCompare September 2, 2021 00:53

am11 changed the title ~~Rev Alpine WithNode from 3.9 to 3.14~~Rev Alpine WithNode from 3.9 to 3.13Sep 2, 2021

runfoappBot mentioned this pull request Sep 24, 2021
System.Text.RegularExpressions.Tests.RegexMatchTests.Match_ExcessPrefix Failing in CI #59541
Closed

safern reviewed Sep 24, 2021
View reviewed changes

Comment threadeng/pipelines/coreclr/templates/helix-queues-setup.yml Outdated

safern reviewed Sep 24, 2021
View reviewed changes

Comment threadeng/pipelines/libraries/helix-queues-setup.yml Outdated

Rev Alpine WithNode from 3.9 to 3.13
7e1fb3e
Also, * update helix queues from 3.12 to 3.13 * add 3.14 helix queues * update performance job to install cargo from main branch

am11 force-pushed the feature/ci/alpine3.14 branch from 174c287 to 7e1fb3eCompare September 24, 2021 19:23

safern reviewed Sep 24, 2021
View reviewed changes

Add Alpine 3.13 in rolling CI
f0fd764

safern approved these changes Sep 24, 2021
View reviewed changes

hoyosjs approved these changes Sep 24, 2021
View reviewed changes

Add missing closing paren
a1e698c

This was referenced Sep 29, 2021
Segfault in tracing/eventpipe/reverseouter/reverseouter/reverseouter.sh on Ubuntu. #59758
Closed
System.Text.Json.Tests crashing with segfault in CI #57198
Closed

hoyosjs merged commit a50e1e6 into dotnet:mainSep 30, 2021

am11 deleted the feature/ci/alpine3.14 branch September 30, 2021 10:30

kunalspathak mentioned this pull request Oct 7, 2021
[Perf] Changes at 9/30/2021 9:36:23 AM dotnet/perf-autofiling-issues#1672
Closed

CarnaViire mentioned this pull request Oct 8, 2021
Test failures: System.Diagnostics.Tests.ProcessStartInfoTests on windows.11.amd64.clientpre #60191
Open

ghost locked as resolved and limited conversation to collaborators Nov 3, 2021

Uh oh!

Conversation

am11 commented Aug 12, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Aug 12, 2021

Uh oh!

am11 commented Aug 12, 2021

Uh oh!

jkotas commented Aug 12, 2021

Uh oh!

am11 commented Aug 12, 2021

Uh oh!

am11 commented Aug 12, 2021

Uh oh!

hoyosjs commented Aug 13, 2021

Uh oh!

am11 commented Aug 13, 2021

Uh oh!

hoyosjs commented Aug 13, 2021

Uh oh!

am11 commented Aug 14, 2021

Uh oh!

ViktorHofer commented Aug 16, 2021

Uh oh!

am11 commented Aug 16, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ViktorHofer commented Aug 16, 2021

Uh oh!

ViktorHofer commented Aug 20, 2021

Uh oh!

ViktorHofer commented Sep 1, 2021

Uh oh!

MattGal commented Sep 1, 2021

Uh oh!

MattGal commented Sep 1, 2021

Uh oh!

hoyosjs commented Sep 1, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MattGal commented Sep 1, 2021

Uh oh!

hoyosjs commented Sep 1, 2021

Uh oh!

MattGal commented Sep 1, 2021

Uh oh!

am11 commented Sep 1, 2021

Uh oh!

Uh oh!

Uh oh!

safernSep 24, 2021

Choose a reason for hiding this comment

Uh oh!

am11Sep 24, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

safern left a comment

Choose a reason for hiding this comment

Uh oh!

hoyosjs left a comment

Choose a reason for hiding this comment

Uh oh!

safern commented Sep 24, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aik-jahoda commented Sep 29, 2021

Uh oh!

danmoseley commented Sep 29, 2021

Uh oh!

am11 commented Sep 29, 2021

Uh oh!

safern commented Sep 29, 2021

Uh oh!

am11 commented Sep 29, 2021

Uh oh!

safern commented Sep 29, 2021

Uh oh!

danmoseley commented Sep 29, 2021• edited by safern Loading Uh oh! There was an error while loading. Please reload this page.

am11 commented Aug 12, 2021•
edited
Loading

am11 commented Aug 16, 2021•
edited
Loading

hoyosjs commented Sep 1, 2021•
edited
Loading

am11Sep 24, 2021•
edited
Loading

safern commented Sep 24, 2021•
edited
Loading

danmoseley commented Sep 29, 2021•
edited by safern
Loading