Enable building .NET Core osx-arm64 in CI - #43187

Merged
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS
Oct 9, 2020
Merged

Enable building .NET Core osx-arm64 in CI#43187
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS

Conversation

@sdmaclea

Copy link
Copy Markdown
Contributor

Enables building of .NET Core osx-arm64

Fixes#41131

/cc @mangod9

@ghost

ghost commented Oct 8, 2020

Copy link
Copy Markdown

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Comment threadeng/pipelines/common/platform-matrix.yml Outdated
Comment threadeng/pipelines/common/xplat-setup.yml
Comment threadeng/install-native-dependencies.sh Outdated
@@ -1,5 +1,10 @@
#!/usr/bin/env bash

if [ "$1" = "OSX" ] && [ "$2" = "arm64" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is a great place to begin doing this. This should probably go in the yaml as it's tied to AzDO pools, and in documentation we should specify it as a dependency for osx-arm64.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the rationale. Isn't this file just for installing dependencies for CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this just feels like hiding things in scripts. This is not an installation. At least it's not obvious to me that an installation script hardcodes a path to something that's only true in AzDO (for example, what if a customer runs this expecting to brew install the deps in their box).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hoyosjs & I are in disagreement here... Any other opinions... Before we try to drive to consensus?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build step that calls this is called Install Build Dependencies. It seems logical to configure dependencies here too. If we must I can add another configure dependencies step, but it seems overkill.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdo seems overly specific. Are you OK with setup-native-dependencies-4ci.sh & Setup Build Dependencies?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose azdo just because it's that: the xcode path for AzDO agents. It's not the same path as helix test agents necessarily or anything (my box doesn't have that path even though I use the same version of XCode now). However, I don't feel too strongly, mostly care about "setup" and "ci/not-local".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be people that run this to help them set native dependencies locally?

@sdmacleasdmacleaOct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen any instructions suggesting using this for developer setup.

Edit:
I guess since this is likely temporary I can just make it a separate build step.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added azDO to the CI install-native-dependencies.sh command line and made the xcode-select conditional on the azDO argument. I also added brief header comments.

I initial went down the path of renaming install -> setup, but I felt like it added confusion. setup in my mind doesn't imply install. However setup/configurations is a subset of install. So I reverted it all and went with this approach.

Hope this is a satisfactory compromise.

Comment threadeng/pipelines/common/xplat-setup.yml
@hoyosjs

Copy link
Copy Markdown
Member

Also unclear if related, somehow we ended with this job:

 - job: installer_coreclr__OSX_arm64_ReleasedisplayName: Installer Build and Test coreclr OSX_arm64 ReleasedependsOn:
- checkout
- coreclr__product_build_OSX_arm64_release
- libraries_build_OSX_arm64_Debug

@safern do release tests for installer depend on debug libraries bits?

@hoyosjs

Copy link
Copy Markdown
Member

The error remaining is just we are missing passing the architecture to the installer part of the build. The autodetection thinks it's x64 (given that's the host's architecture).

@safern

Copy link
Copy Markdown
Member

@safern do release tests for installer depend on debug libraries bits?

In PR some of them do, because we build some flavors of libraries in Debug on PR and Release for CI and official build.

There is a variable that controls this... debugOnPrReleaseOnRolling
.https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml#L617

Comment threadeng/pipelines/installer/jobs/base-job.yml Outdated
@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I am running an internal build of this here https://dev.azure.com/dnceng/internal/_build/results?buildId=847071&view=results. I suspect I will need to add something additional to publish the new packages, but that could be a subsequent PR.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

Looks like the internal build completed fine. Looks to me (a novice) like the osx-arm64 packages are properly publishing.

CI on this PR and the internal run are green. This is ready for a final review.

@mangod9mangod9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for enabling this.

@hoyosjs

Copy link
Copy Markdown
Member

The Runtime packs look good to me
image The assemblies are crossgen'd and all.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I'm going to merge this. I'll work on a follow up PR to add the crossBuild parameter.

If there is anything else which you would like resolved. Feel free to comment on this or that PR, and I will work to get it resolved.

@sdmaclea
sdmaclea merged commit 2a52f75 into dotnet:masterOct 9, 2020
@sdmaclea

sdmaclea commented Oct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

/cc @richlander This should enable nightly runtime osx-arm64 builds.

- Linux_arm64
- Linux_musl_arm64
- ${{ if eq(variables['includeOsxOuterloop'], true) }}:
- OSX_arm64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this... I think we don't have a helix queue setup for OSX_arm64, so I guess this broke the libraries outerloop build. I can fix this in a separate PR and we can enable them once we have a queue for it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we still dont have stable hardware for this we dont expect to enable test runs for osx_arm64 just yet. Can we leave it disabled till we have that?

@safernsafernOct 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I created a PR for that: #43242

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool.. thanks.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this...

@safern Thanks for being genereous, but this was clearly my fault. I forget there are three main pipelines all the time. I get the rolling outerloop and the official build pipeline confused and map them to a single pipeline...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we have so many ymls that it is sometimes hard to get it right and the easier way to add a new flavor is to try and look where we're already building osx_x64 and just add entries for the new platform there, so confusing for all I think 😄

@safernsafern left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't enable mono on that new platform. @akoeplinger@steveisok do we want to do that?

Overall looks good to me. Thanks for doing this @sdmaclea

@ghostghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osx-arm64 enable CoreCLR native component builds in CI.

6 participants

@sdmaclea@hoyosjs@safern@janvorli@wfurt@mangod9
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Enable building .NET Core osx-arm64 in CI - #43187

Merged
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS
Oct 9, 2020
Merged

Enable building .NET Core osx-arm64 in CI#43187
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS

Conversation

@sdmaclea

Copy link
Copy Markdown
Contributor

Enables building of .NET Core osx-arm64

Fixes#41131

/cc @mangod9

@ghost

ghost commented Oct 8, 2020

Copy link
Copy Markdown

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Comment threadeng/pipelines/common/platform-matrix.yml Outdated
Comment threadeng/pipelines/common/xplat-setup.yml
Comment threadeng/install-native-dependencies.sh Outdated
@@ -1,5 +1,10 @@
#!/usr/bin/env bash

if [ "$1" = "OSX" ] && [ "$2" = "arm64" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is a great place to begin doing this. This should probably go in the yaml as it's tied to AzDO pools, and in documentation we should specify it as a dependency for osx-arm64.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the rationale. Isn't this file just for installing dependencies for CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this just feels like hiding things in scripts. This is not an installation. At least it's not obvious to me that an installation script hardcodes a path to something that's only true in AzDO (for example, what if a customer runs this expecting to brew install the deps in their box).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hoyosjs & I are in disagreement here... Any other opinions... Before we try to drive to consensus?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build step that calls this is called Install Build Dependencies. It seems logical to configure dependencies here too. If we must I can add another configure dependencies step, but it seems overkill.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdo seems overly specific. Are you OK with setup-native-dependencies-4ci.sh & Setup Build Dependencies?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose azdo just because it's that: the xcode path for AzDO agents. It's not the same path as helix test agents necessarily or anything (my box doesn't have that path even though I use the same version of XCode now). However, I don't feel too strongly, mostly care about "setup" and "ci/not-local".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be people that run this to help them set native dependencies locally?

@sdmacleasdmacleaOct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen any instructions suggesting using this for developer setup.

Edit:
I guess since this is likely temporary I can just make it a separate build step.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added azDO to the CI install-native-dependencies.sh command line and made the xcode-select conditional on the azDO argument. I also added brief header comments.

I initial went down the path of renaming install -> setup, but I felt like it added confusion. setup in my mind doesn't imply install. However setup/configurations is a subset of install. So I reverted it all and went with this approach.

Hope this is a satisfactory compromise.

Comment threadeng/pipelines/common/xplat-setup.yml
@hoyosjs

Copy link
Copy Markdown
Member

Also unclear if related, somehow we ended with this job:

 - job: installer_coreclr__OSX_arm64_ReleasedisplayName: Installer Build and Test coreclr OSX_arm64 ReleasedependsOn:
- checkout
- coreclr__product_build_OSX_arm64_release
- libraries_build_OSX_arm64_Debug

@safern do release tests for installer depend on debug libraries bits?

@hoyosjs

Copy link
Copy Markdown
Member

The error remaining is just we are missing passing the architecture to the installer part of the build. The autodetection thinks it's x64 (given that's the host's architecture).

@safern

Copy link
Copy Markdown
Member

@safern do release tests for installer depend on debug libraries bits?

In PR some of them do, because we build some flavors of libraries in Debug on PR and Release for CI and official build.

There is a variable that controls this... debugOnPrReleaseOnRolling
.https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml#L617

Comment threadeng/pipelines/installer/jobs/base-job.yml Outdated
@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I am running an internal build of this here https://dev.azure.com/dnceng/internal/_build/results?buildId=847071&view=results. I suspect I will need to add something additional to publish the new packages, but that could be a subsequent PR.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

Looks like the internal build completed fine. Looks to me (a novice) like the osx-arm64 packages are properly publishing.

CI on this PR and the internal run are green. This is ready for a final review.

@mangod9mangod9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for enabling this.

@hoyosjs

Copy link
Copy Markdown
Member

The Runtime packs look good to me
image The assemblies are crossgen'd and all.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I'm going to merge this. I'll work on a follow up PR to add the crossBuild parameter.

If there is anything else which you would like resolved. Feel free to comment on this or that PR, and I will work to get it resolved.

@sdmaclea
sdmaclea merged commit 2a52f75 into dotnet:masterOct 9, 2020
@sdmaclea

sdmaclea commented Oct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

/cc @richlander This should enable nightly runtime osx-arm64 builds.

- Linux_arm64
- Linux_musl_arm64
- ${{ if eq(variables['includeOsxOuterloop'], true) }}:
- OSX_arm64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this... I think we don't have a helix queue setup for OSX_arm64, so I guess this broke the libraries outerloop build. I can fix this in a separate PR and we can enable them once we have a queue for it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we still dont have stable hardware for this we dont expect to enable test runs for osx_arm64 just yet. Can we leave it disabled till we have that?

@safernsafernOct 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I created a PR for that: #43242

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool.. thanks.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this...

@safern Thanks for being genereous, but this was clearly my fault. I forget there are three main pipelines all the time. I get the rolling outerloop and the official build pipeline confused and map them to a single pipeline...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we have so many ymls that it is sometimes hard to get it right and the easier way to add a new flavor is to try and look where we're already building osx_x64 and just add entries for the new platform there, so confusing for all I think 😄

@safernsafern left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't enable mono on that new platform. @akoeplinger@steveisok do we want to do that?

Overall looks good to me. Thanks for doing this @sdmaclea

@ghostghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osx-arm64 enable CoreCLR native component builds in CI.

6 participants

@sdmaclea@hoyosjs@safern@janvorli@wfurt@mangod9
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Enable building .NET Core osx-arm64 in CI - #43187

Merged
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS
Oct 9, 2020
Merged

Enable building .NET Core osx-arm64 in CI#43187
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS

Conversation

@sdmaclea

Copy link
Copy Markdown
Contributor

Enables building of .NET Core osx-arm64

Fixes#41131

/cc @mangod9

@ghost

ghost commented Oct 8, 2020

Copy link
Copy Markdown

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Comment threadeng/pipelines/common/platform-matrix.yml Outdated
Comment threadeng/pipelines/common/xplat-setup.yml
Comment threadeng/install-native-dependencies.sh Outdated
@@ -1,5 +1,10 @@
#!/usr/bin/env bash

if [ "$1" = "OSX" ] && [ "$2" = "arm64" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is a great place to begin doing this. This should probably go in the yaml as it's tied to AzDO pools, and in documentation we should specify it as a dependency for osx-arm64.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the rationale. Isn't this file just for installing dependencies for CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this just feels like hiding things in scripts. This is not an installation. At least it's not obvious to me that an installation script hardcodes a path to something that's only true in AzDO (for example, what if a customer runs this expecting to brew install the deps in their box).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hoyosjs & I are in disagreement here... Any other opinions... Before we try to drive to consensus?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build step that calls this is called Install Build Dependencies. It seems logical to configure dependencies here too. If we must I can add another configure dependencies step, but it seems overkill.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdo seems overly specific. Are you OK with setup-native-dependencies-4ci.sh & Setup Build Dependencies?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose azdo just because it's that: the xcode path for AzDO agents. It's not the same path as helix test agents necessarily or anything (my box doesn't have that path even though I use the same version of XCode now). However, I don't feel too strongly, mostly care about "setup" and "ci/not-local".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be people that run this to help them set native dependencies locally?

@sdmacleasdmacleaOct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen any instructions suggesting using this for developer setup.

Edit:
I guess since this is likely temporary I can just make it a separate build step.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added azDO to the CI install-native-dependencies.sh command line and made the xcode-select conditional on the azDO argument. I also added brief header comments.

I initial went down the path of renaming install -> setup, but I felt like it added confusion. setup in my mind doesn't imply install. However setup/configurations is a subset of install. So I reverted it all and went with this approach.

Hope this is a satisfactory compromise.

Comment threadeng/pipelines/common/xplat-setup.yml
@hoyosjs

Copy link
Copy Markdown
Member

Also unclear if related, somehow we ended with this job:

 - job: installer_coreclr__OSX_arm64_ReleasedisplayName: Installer Build and Test coreclr OSX_arm64 ReleasedependsOn:
- checkout
- coreclr__product_build_OSX_arm64_release
- libraries_build_OSX_arm64_Debug

@safern do release tests for installer depend on debug libraries bits?

@hoyosjs

Copy link
Copy Markdown
Member

The error remaining is just we are missing passing the architecture to the installer part of the build. The autodetection thinks it's x64 (given that's the host's architecture).

@safern

Copy link
Copy Markdown
Member

@safern do release tests for installer depend on debug libraries bits?

In PR some of them do, because we build some flavors of libraries in Debug on PR and Release for CI and official build.

There is a variable that controls this... debugOnPrReleaseOnRolling
.https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml#L617

Comment threadeng/pipelines/installer/jobs/base-job.yml Outdated
@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I am running an internal build of this here https://dev.azure.com/dnceng/internal/_build/results?buildId=847071&view=results. I suspect I will need to add something additional to publish the new packages, but that could be a subsequent PR.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

Looks like the internal build completed fine. Looks to me (a novice) like the osx-arm64 packages are properly publishing.

CI on this PR and the internal run are green. This is ready for a final review.

@mangod9mangod9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for enabling this.

@hoyosjs

Copy link
Copy Markdown
Member

The Runtime packs look good to me
image The assemblies are crossgen'd and all.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I'm going to merge this. I'll work on a follow up PR to add the crossBuild parameter.

If there is anything else which you would like resolved. Feel free to comment on this or that PR, and I will work to get it resolved.

@sdmaclea
sdmaclea merged commit 2a52f75 into dotnet:masterOct 9, 2020
@sdmaclea

sdmaclea commented Oct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

/cc @richlander This should enable nightly runtime osx-arm64 builds.

- Linux_arm64
- Linux_musl_arm64
- ${{ if eq(variables['includeOsxOuterloop'], true) }}:
- OSX_arm64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this... I think we don't have a helix queue setup for OSX_arm64, so I guess this broke the libraries outerloop build. I can fix this in a separate PR and we can enable them once we have a queue for it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we still dont have stable hardware for this we dont expect to enable test runs for osx_arm64 just yet. Can we leave it disabled till we have that?

@safernsafernOct 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I created a PR for that: #43242

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool.. thanks.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this...

@safern Thanks for being genereous, but this was clearly my fault. I forget there are three main pipelines all the time. I get the rolling outerloop and the official build pipeline confused and map them to a single pipeline...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we have so many ymls that it is sometimes hard to get it right and the easier way to add a new flavor is to try and look where we're already building osx_x64 and just add entries for the new platform there, so confusing for all I think 😄

@safernsafern left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't enable mono on that new platform. @akoeplinger@steveisok do we want to do that?

Overall looks good to me. Thanks for doing this @sdmaclea

@ghostghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osx-arm64 enable CoreCLR native component builds in CI.

6 participants

@sdmaclea@hoyosjs@safern@janvorli@wfurt@mangod9
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Enable building .NET Core osx-arm64 in CI - #43187

Merged
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS
Oct 9, 2020
Merged

Enable building .NET Core osx-arm64 in CI#43187
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS

Conversation

@sdmaclea

Copy link
Copy Markdown
Contributor

Enables building of .NET Core osx-arm64

Fixes#41131

/cc @mangod9

@ghost

ghost commented Oct 8, 2020

Copy link
Copy Markdown

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Comment threadeng/pipelines/common/platform-matrix.yml Outdated
Comment threadeng/pipelines/common/xplat-setup.yml
Comment threadeng/install-native-dependencies.sh Outdated
@@ -1,5 +1,10 @@
#!/usr/bin/env bash

if [ "$1" = "OSX" ] && [ "$2" = "arm64" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is a great place to begin doing this. This should probably go in the yaml as it's tied to AzDO pools, and in documentation we should specify it as a dependency for osx-arm64.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the rationale. Isn't this file just for installing dependencies for CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this just feels like hiding things in scripts. This is not an installation. At least it's not obvious to me that an installation script hardcodes a path to something that's only true in AzDO (for example, what if a customer runs this expecting to brew install the deps in their box).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hoyosjs & I are in disagreement here... Any other opinions... Before we try to drive to consensus?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build step that calls this is called Install Build Dependencies. It seems logical to configure dependencies here too. If we must I can add another configure dependencies step, but it seems overkill.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdo seems overly specific. Are you OK with setup-native-dependencies-4ci.sh & Setup Build Dependencies?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose azdo just because it's that: the xcode path for AzDO agents. It's not the same path as helix test agents necessarily or anything (my box doesn't have that path even though I use the same version of XCode now). However, I don't feel too strongly, mostly care about "setup" and "ci/not-local".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be people that run this to help them set native dependencies locally?

@sdmacleasdmacleaOct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen any instructions suggesting using this for developer setup.

Edit:
I guess since this is likely temporary I can just make it a separate build step.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added azDO to the CI install-native-dependencies.sh command line and made the xcode-select conditional on the azDO argument. I also added brief header comments.

I initial went down the path of renaming install -> setup, but I felt like it added confusion. setup in my mind doesn't imply install. However setup/configurations is a subset of install. So I reverted it all and went with this approach.

Hope this is a satisfactory compromise.

Comment threadeng/pipelines/common/xplat-setup.yml
@hoyosjs

Copy link
Copy Markdown
Member

Also unclear if related, somehow we ended with this job:

 - job: installer_coreclr__OSX_arm64_ReleasedisplayName: Installer Build and Test coreclr OSX_arm64 ReleasedependsOn:
- checkout
- coreclr__product_build_OSX_arm64_release
- libraries_build_OSX_arm64_Debug

@safern do release tests for installer depend on debug libraries bits?

@hoyosjs

Copy link
Copy Markdown
Member

The error remaining is just we are missing passing the architecture to the installer part of the build. The autodetection thinks it's x64 (given that's the host's architecture).

@safern

Copy link
Copy Markdown
Member

@safern do release tests for installer depend on debug libraries bits?

In PR some of them do, because we build some flavors of libraries in Debug on PR and Release for CI and official build.

There is a variable that controls this... debugOnPrReleaseOnRolling
.https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml#L617

Comment threadeng/pipelines/installer/jobs/base-job.yml Outdated
@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I am running an internal build of this here https://dev.azure.com/dnceng/internal/_build/results?buildId=847071&view=results. I suspect I will need to add something additional to publish the new packages, but that could be a subsequent PR.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

Looks like the internal build completed fine. Looks to me (a novice) like the osx-arm64 packages are properly publishing.

CI on this PR and the internal run are green. This is ready for a final review.

@mangod9mangod9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for enabling this.

@hoyosjs

Copy link
Copy Markdown
Member

The Runtime packs look good to me
image The assemblies are crossgen'd and all.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I'm going to merge this. I'll work on a follow up PR to add the crossBuild parameter.

If there is anything else which you would like resolved. Feel free to comment on this or that PR, and I will work to get it resolved.

@sdmaclea
sdmaclea merged commit 2a52f75 into dotnet:masterOct 9, 2020
@sdmaclea

sdmaclea commented Oct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

/cc @richlander This should enable nightly runtime osx-arm64 builds.

- Linux_arm64
- Linux_musl_arm64
- ${{ if eq(variables['includeOsxOuterloop'], true) }}:
- OSX_arm64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this... I think we don't have a helix queue setup for OSX_arm64, so I guess this broke the libraries outerloop build. I can fix this in a separate PR and we can enable them once we have a queue for it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we still dont have stable hardware for this we dont expect to enable test runs for osx_arm64 just yet. Can we leave it disabled till we have that?

@safernsafernOct 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I created a PR for that: #43242

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool.. thanks.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this...

@safern Thanks for being genereous, but this was clearly my fault. I forget there are three main pipelines all the time. I get the rolling outerloop and the official build pipeline confused and map them to a single pipeline...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we have so many ymls that it is sometimes hard to get it right and the easier way to add a new flavor is to try and look where we're already building osx_x64 and just add entries for the new platform there, so confusing for all I think 😄

@safernsafern left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't enable mono on that new platform. @akoeplinger@steveisok do we want to do that?

Overall looks good to me. Thanks for doing this @sdmaclea

@ghostghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osx-arm64 enable CoreCLR native component builds in CI.

6 participants

@sdmaclea@hoyosjs@safern@janvorli@wfurt@mangod9
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Enable building .NET Core osx-arm64 in CI - #43187

Merged
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS
Oct 9, 2020
Merged

Enable building .NET Core osx-arm64 in CI#43187
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS

Conversation

@sdmaclea

Copy link
Copy Markdown
Contributor

Enables building of .NET Core osx-arm64

Fixes#41131

/cc @mangod9

@ghost

ghost commented Oct 8, 2020

Copy link
Copy Markdown

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Comment threadeng/pipelines/common/platform-matrix.yml Outdated
Comment threadeng/pipelines/common/xplat-setup.yml
Comment threadeng/install-native-dependencies.sh Outdated
@@ -1,5 +1,10 @@
#!/usr/bin/env bash

if [ "$1" = "OSX" ] && [ "$2" = "arm64" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is a great place to begin doing this. This should probably go in the yaml as it's tied to AzDO pools, and in documentation we should specify it as a dependency for osx-arm64.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the rationale. Isn't this file just for installing dependencies for CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this just feels like hiding things in scripts. This is not an installation. At least it's not obvious to me that an installation script hardcodes a path to something that's only true in AzDO (for example, what if a customer runs this expecting to brew install the deps in their box).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hoyosjs & I are in disagreement here... Any other opinions... Before we try to drive to consensus?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build step that calls this is called Install Build Dependencies. It seems logical to configure dependencies here too. If we must I can add another configure dependencies step, but it seems overkill.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdo seems overly specific. Are you OK with setup-native-dependencies-4ci.sh & Setup Build Dependencies?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose azdo just because it's that: the xcode path for AzDO agents. It's not the same path as helix test agents necessarily or anything (my box doesn't have that path even though I use the same version of XCode now). However, I don't feel too strongly, mostly care about "setup" and "ci/not-local".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be people that run this to help them set native dependencies locally?

@sdmacleasdmacleaOct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen any instructions suggesting using this for developer setup.

Edit:
I guess since this is likely temporary I can just make it a separate build step.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added azDO to the CI install-native-dependencies.sh command line and made the xcode-select conditional on the azDO argument. I also added brief header comments.

I initial went down the path of renaming install -> setup, but I felt like it added confusion. setup in my mind doesn't imply install. However setup/configurations is a subset of install. So I reverted it all and went with this approach.

Hope this is a satisfactory compromise.

Comment threadeng/pipelines/common/xplat-setup.yml
@hoyosjs

Copy link
Copy Markdown
Member

Also unclear if related, somehow we ended with this job:

 - job: installer_coreclr__OSX_arm64_ReleasedisplayName: Installer Build and Test coreclr OSX_arm64 ReleasedependsOn:
- checkout
- coreclr__product_build_OSX_arm64_release
- libraries_build_OSX_arm64_Debug

@safern do release tests for installer depend on debug libraries bits?

@hoyosjs

Copy link
Copy Markdown
Member

The error remaining is just we are missing passing the architecture to the installer part of the build. The autodetection thinks it's x64 (given that's the host's architecture).

@safern

Copy link
Copy Markdown
Member

@safern do release tests for installer depend on debug libraries bits?

In PR some of them do, because we build some flavors of libraries in Debug on PR and Release for CI and official build.

There is a variable that controls this... debugOnPrReleaseOnRolling
.https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml#L617

Comment threadeng/pipelines/installer/jobs/base-job.yml Outdated
@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I am running an internal build of this here https://dev.azure.com/dnceng/internal/_build/results?buildId=847071&view=results. I suspect I will need to add something additional to publish the new packages, but that could be a subsequent PR.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

Looks like the internal build completed fine. Looks to me (a novice) like the osx-arm64 packages are properly publishing.

CI on this PR and the internal run are green. This is ready for a final review.

@mangod9mangod9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for enabling this.

@hoyosjs

Copy link
Copy Markdown
Member

The Runtime packs look good to me
image The assemblies are crossgen'd and all.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I'm going to merge this. I'll work on a follow up PR to add the crossBuild parameter.

If there is anything else which you would like resolved. Feel free to comment on this or that PR, and I will work to get it resolved.

@sdmaclea
sdmaclea merged commit 2a52f75 into dotnet:masterOct 9, 2020
@sdmaclea

sdmaclea commented Oct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

/cc @richlander This should enable nightly runtime osx-arm64 builds.

- Linux_arm64
- Linux_musl_arm64
- ${{ if eq(variables['includeOsxOuterloop'], true) }}:
- OSX_arm64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this... I think we don't have a helix queue setup for OSX_arm64, so I guess this broke the libraries outerloop build. I can fix this in a separate PR and we can enable them once we have a queue for it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we still dont have stable hardware for this we dont expect to enable test runs for osx_arm64 just yet. Can we leave it disabled till we have that?

@safernsafernOct 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I created a PR for that: #43242

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool.. thanks.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this...

@safern Thanks for being genereous, but this was clearly my fault. I forget there are three main pipelines all the time. I get the rolling outerloop and the official build pipeline confused and map them to a single pipeline...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we have so many ymls that it is sometimes hard to get it right and the easier way to add a new flavor is to try and look where we're already building osx_x64 and just add entries for the new platform there, so confusing for all I think 😄

@safernsafern left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't enable mono on that new platform. @akoeplinger@steveisok do we want to do that?

Overall looks good to me. Thanks for doing this @sdmaclea

@ghostghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osx-arm64 enable CoreCLR native component builds in CI.

6 participants

@sdmaclea@hoyosjs@safern@janvorli@wfurt@mangod9
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Enable building .NET Core osx-arm64 in CI - #43187

Merged
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS
Oct 9, 2020
Merged

Enable building .NET Core osx-arm64 in CI#43187
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS

Conversation

@sdmaclea

Copy link
Copy Markdown
Contributor

Enables building of .NET Core osx-arm64

Fixes#41131

/cc @mangod9

@ghost

ghost commented Oct 8, 2020

Copy link
Copy Markdown

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Comment threadeng/pipelines/common/platform-matrix.yml Outdated
Comment threadeng/pipelines/common/xplat-setup.yml
Comment threadeng/install-native-dependencies.sh Outdated
@@ -1,5 +1,10 @@
#!/usr/bin/env bash

if [ "$1" = "OSX" ] && [ "$2" = "arm64" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is a great place to begin doing this. This should probably go in the yaml as it's tied to AzDO pools, and in documentation we should specify it as a dependency for osx-arm64.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the rationale. Isn't this file just for installing dependencies for CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this just feels like hiding things in scripts. This is not an installation. At least it's not obvious to me that an installation script hardcodes a path to something that's only true in AzDO (for example, what if a customer runs this expecting to brew install the deps in their box).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hoyosjs & I are in disagreement here... Any other opinions... Before we try to drive to consensus?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build step that calls this is called Install Build Dependencies. It seems logical to configure dependencies here too. If we must I can add another configure dependencies step, but it seems overkill.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdo seems overly specific. Are you OK with setup-native-dependencies-4ci.sh & Setup Build Dependencies?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose azdo just because it's that: the xcode path for AzDO agents. It's not the same path as helix test agents necessarily or anything (my box doesn't have that path even though I use the same version of XCode now). However, I don't feel too strongly, mostly care about "setup" and "ci/not-local".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be people that run this to help them set native dependencies locally?

@sdmacleasdmacleaOct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen any instructions suggesting using this for developer setup.

Edit:
I guess since this is likely temporary I can just make it a separate build step.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added azDO to the CI install-native-dependencies.sh command line and made the xcode-select conditional on the azDO argument. I also added brief header comments.

I initial went down the path of renaming install -> setup, but I felt like it added confusion. setup in my mind doesn't imply install. However setup/configurations is a subset of install. So I reverted it all and went with this approach.

Hope this is a satisfactory compromise.

Comment threadeng/pipelines/common/xplat-setup.yml
@hoyosjs

Copy link
Copy Markdown
Member

Also unclear if related, somehow we ended with this job:

 - job: installer_coreclr__OSX_arm64_ReleasedisplayName: Installer Build and Test coreclr OSX_arm64 ReleasedependsOn:
- checkout
- coreclr__product_build_OSX_arm64_release
- libraries_build_OSX_arm64_Debug

@safern do release tests for installer depend on debug libraries bits?

@hoyosjs

Copy link
Copy Markdown
Member

The error remaining is just we are missing passing the architecture to the installer part of the build. The autodetection thinks it's x64 (given that's the host's architecture).

@safern

Copy link
Copy Markdown
Member

@safern do release tests for installer depend on debug libraries bits?

In PR some of them do, because we build some flavors of libraries in Debug on PR and Release for CI and official build.

There is a variable that controls this... debugOnPrReleaseOnRolling
.https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml#L617

Comment threadeng/pipelines/installer/jobs/base-job.yml Outdated
@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I am running an internal build of this here https://dev.azure.com/dnceng/internal/_build/results?buildId=847071&view=results. I suspect I will need to add something additional to publish the new packages, but that could be a subsequent PR.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

Looks like the internal build completed fine. Looks to me (a novice) like the osx-arm64 packages are properly publishing.

CI on this PR and the internal run are green. This is ready for a final review.

@mangod9mangod9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for enabling this.

@hoyosjs

Copy link
Copy Markdown
Member

The Runtime packs look good to me
image The assemblies are crossgen'd and all.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I'm going to merge this. I'll work on a follow up PR to add the crossBuild parameter.

If there is anything else which you would like resolved. Feel free to comment on this or that PR, and I will work to get it resolved.

@sdmaclea
sdmaclea merged commit 2a52f75 into dotnet:masterOct 9, 2020
@sdmaclea

sdmaclea commented Oct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

/cc @richlander This should enable nightly runtime osx-arm64 builds.

- Linux_arm64
- Linux_musl_arm64
- ${{ if eq(variables['includeOsxOuterloop'], true) }}:
- OSX_arm64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this... I think we don't have a helix queue setup for OSX_arm64, so I guess this broke the libraries outerloop build. I can fix this in a separate PR and we can enable them once we have a queue for it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we still dont have stable hardware for this we dont expect to enable test runs for osx_arm64 just yet. Can we leave it disabled till we have that?

@safernsafernOct 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I created a PR for that: #43242

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool.. thanks.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this...

@safern Thanks for being genereous, but this was clearly my fault. I forget there are three main pipelines all the time. I get the rolling outerloop and the official build pipeline confused and map them to a single pipeline...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we have so many ymls that it is sometimes hard to get it right and the easier way to add a new flavor is to try and look where we're already building osx_x64 and just add entries for the new platform there, so confusing for all I think 😄

@safernsafern left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't enable mono on that new platform. @akoeplinger@steveisok do we want to do that?

Overall looks good to me. Thanks for doing this @sdmaclea

@ghostghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osx-arm64 enable CoreCLR native component builds in CI.

6 participants

@sdmaclea@hoyosjs@safern@janvorli@wfurt@mangod9
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Enable building .NET Core osx-arm64 in CI - #43187

Merged
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS
Oct 9, 2020
Merged

Enable building .NET Core osx-arm64 in CI#43187
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS

Conversation

@sdmaclea

Copy link
Copy Markdown
Contributor

Enables building of .NET Core osx-arm64

Fixes#41131

/cc @mangod9

@ghost

ghost commented Oct 8, 2020

Copy link
Copy Markdown

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Comment threadeng/pipelines/common/platform-matrix.yml Outdated
Comment threadeng/pipelines/common/xplat-setup.yml
Comment threadeng/install-native-dependencies.sh Outdated
@@ -1,5 +1,10 @@
#!/usr/bin/env bash

if [ "$1" = "OSX" ] && [ "$2" = "arm64" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is a great place to begin doing this. This should probably go in the yaml as it's tied to AzDO pools, and in documentation we should specify it as a dependency for osx-arm64.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the rationale. Isn't this file just for installing dependencies for CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this just feels like hiding things in scripts. This is not an installation. At least it's not obvious to me that an installation script hardcodes a path to something that's only true in AzDO (for example, what if a customer runs this expecting to brew install the deps in their box).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hoyosjs & I are in disagreement here... Any other opinions... Before we try to drive to consensus?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build step that calls this is called Install Build Dependencies. It seems logical to configure dependencies here too. If we must I can add another configure dependencies step, but it seems overkill.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdo seems overly specific. Are you OK with setup-native-dependencies-4ci.sh & Setup Build Dependencies?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose azdo just because it's that: the xcode path for AzDO agents. It's not the same path as helix test agents necessarily or anything (my box doesn't have that path even though I use the same version of XCode now). However, I don't feel too strongly, mostly care about "setup" and "ci/not-local".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be people that run this to help them set native dependencies locally?

@sdmacleasdmacleaOct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen any instructions suggesting using this for developer setup.

Edit:
I guess since this is likely temporary I can just make it a separate build step.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added azDO to the CI install-native-dependencies.sh command line and made the xcode-select conditional on the azDO argument. I also added brief header comments.

I initial went down the path of renaming install -> setup, but I felt like it added confusion. setup in my mind doesn't imply install. However setup/configurations is a subset of install. So I reverted it all and went with this approach.

Hope this is a satisfactory compromise.

Comment threadeng/pipelines/common/xplat-setup.yml
@hoyosjs

Copy link
Copy Markdown
Member

Also unclear if related, somehow we ended with this job:

 - job: installer_coreclr__OSX_arm64_ReleasedisplayName: Installer Build and Test coreclr OSX_arm64 ReleasedependsOn:
- checkout
- coreclr__product_build_OSX_arm64_release
- libraries_build_OSX_arm64_Debug

@safern do release tests for installer depend on debug libraries bits?

@hoyosjs

Copy link
Copy Markdown
Member

The error remaining is just we are missing passing the architecture to the installer part of the build. The autodetection thinks it's x64 (given that's the host's architecture).

@safern

Copy link
Copy Markdown
Member

@safern do release tests for installer depend on debug libraries bits?

In PR some of them do, because we build some flavors of libraries in Debug on PR and Release for CI and official build.

There is a variable that controls this... debugOnPrReleaseOnRolling
.https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml#L617

Comment threadeng/pipelines/installer/jobs/base-job.yml Outdated
@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I am running an internal build of this here https://dev.azure.com/dnceng/internal/_build/results?buildId=847071&view=results. I suspect I will need to add something additional to publish the new packages, but that could be a subsequent PR.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

Looks like the internal build completed fine. Looks to me (a novice) like the osx-arm64 packages are properly publishing.

CI on this PR and the internal run are green. This is ready for a final review.

@mangod9mangod9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for enabling this.

@hoyosjs

Copy link
Copy Markdown
Member

The Runtime packs look good to me
image The assemblies are crossgen'd and all.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I'm going to merge this. I'll work on a follow up PR to add the crossBuild parameter.

If there is anything else which you would like resolved. Feel free to comment on this or that PR, and I will work to get it resolved.

@sdmaclea
sdmaclea merged commit 2a52f75 into dotnet:masterOct 9, 2020
@sdmaclea

sdmaclea commented Oct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

/cc @richlander This should enable nightly runtime osx-arm64 builds.

- Linux_arm64
- Linux_musl_arm64
- ${{ if eq(variables['includeOsxOuterloop'], true) }}:
- OSX_arm64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this... I think we don't have a helix queue setup for OSX_arm64, so I guess this broke the libraries outerloop build. I can fix this in a separate PR and we can enable them once we have a queue for it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we still dont have stable hardware for this we dont expect to enable test runs for osx_arm64 just yet. Can we leave it disabled till we have that?

@safernsafernOct 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I created a PR for that: #43242

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool.. thanks.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this...

@safern Thanks for being genereous, but this was clearly my fault. I forget there are three main pipelines all the time. I get the rolling outerloop and the official build pipeline confused and map them to a single pipeline...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we have so many ymls that it is sometimes hard to get it right and the easier way to add a new flavor is to try and look where we're already building osx_x64 and just add entries for the new platform there, so confusing for all I think 😄

@safernsafern left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't enable mono on that new platform. @akoeplinger@steveisok do we want to do that?

Overall looks good to me. Thanks for doing this @sdmaclea

@ghostghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osx-arm64 enable CoreCLR native component builds in CI.

6 participants

@sdmaclea@hoyosjs@safern@janvorli@wfurt@mangod9
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Enable building .NET Core osx-arm64 in CI - #43187

Merged
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS
Oct 9, 2020
Merged

Enable building .NET Core osx-arm64 in CI#43187
sdmaclea merged 6 commits into
dotnet:masterfrom
sdmaclea:dev/Arm64MacOS

Conversation

@sdmaclea

Copy link
Copy Markdown
Contributor

Enables building of .NET Core osx-arm64

Fixes#41131

/cc @mangod9

@ghost

ghost commented Oct 8, 2020

Copy link
Copy Markdown

Tagging subscribers to this area: @ViktorHofer
See info in area-owners.md if you want to be subscribed.

Comment threadeng/pipelines/common/platform-matrix.yml Outdated
Comment threadeng/pipelines/common/xplat-setup.yml
Comment threadeng/install-native-dependencies.sh Outdated
@@ -1,5 +1,10 @@
#!/usr/bin/env bash

if [ "$1" = "OSX" ] && [ "$2" = "arm64" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure this is a great place to begin doing this. This should probably go in the yaml as it's tied to AzDO pools, and in documentation we should specify it as a dependency for osx-arm64.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the rationale. Isn't this file just for installing dependencies for CI.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this just feels like hiding things in scripts. This is not an installation. At least it's not obvious to me that an installation script hardcodes a path to something that's only true in AzDO (for example, what if a customer runs this expecting to brew install the deps in their box).

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hoyosjs & I are in disagreement here... Any other opinions... Before we try to drive to consensus?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The build step that calls this is called Install Build Dependencies. It seems logical to configure dependencies here too. If we must I can add another configure dependencies step, but it seems overkill.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

azdo seems overly specific. Are you OK with setup-native-dependencies-4ci.sh & Setup Build Dependencies?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I chose azdo just because it's that: the xcode path for AzDO agents. It's not the same path as helix test agents necessarily or anything (my box doesn't have that path even though I use the same version of XCode now). However, I don't feel too strongly, mostly care about "setup" and "ci/not-local".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't be people that run this to help them set native dependencies locally?

@sdmacleasdmacleaOct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hadn't seen any instructions suggesting using this for developer setup.

Edit:
I guess since this is likely temporary I can just make it a separate build step.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added azDO to the CI install-native-dependencies.sh command line and made the xcode-select conditional on the azDO argument. I also added brief header comments.

I initial went down the path of renaming install -> setup, but I felt like it added confusion. setup in my mind doesn't imply install. However setup/configurations is a subset of install. So I reverted it all and went with this approach.

Hope this is a satisfactory compromise.

Comment threadeng/pipelines/common/xplat-setup.yml
@hoyosjs

Copy link
Copy Markdown
Member

Also unclear if related, somehow we ended with this job:

 - job: installer_coreclr__OSX_arm64_ReleasedisplayName: Installer Build and Test coreclr OSX_arm64 ReleasedependsOn:
- checkout
- coreclr__product_build_OSX_arm64_release
- libraries_build_OSX_arm64_Debug

@safern do release tests for installer depend on debug libraries bits?

@hoyosjs

Copy link
Copy Markdown
Member

The error remaining is just we are missing passing the architecture to the installer part of the build. The autodetection thinks it's x64 (given that's the host's architecture).

@safern

Copy link
Copy Markdown
Member

@safern do release tests for installer depend on debug libraries bits?

In PR some of them do, because we build some flavors of libraries in Debug on PR and Release for CI and official build.

There is a variable that controls this... debugOnPrReleaseOnRolling
.https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml#L617

Comment threadeng/pipelines/installer/jobs/base-job.yml Outdated
@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I am running an internal build of this here https://dev.azure.com/dnceng/internal/_build/results?buildId=847071&view=results. I suspect I will need to add something additional to publish the new packages, but that could be a subsequent PR.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

Looks like the internal build completed fine. Looks to me (a novice) like the osx-arm64 packages are properly publishing.

CI on this PR and the internal run are green. This is ready for a final review.

@mangod9mangod9 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for enabling this.

@hoyosjs

Copy link
Copy Markdown
Member

The Runtime packs look good to me
image The assemblies are crossgen'd and all.

@sdmaclea

Copy link
Copy Markdown
ContributorAuthor

I'm going to merge this. I'll work on a follow up PR to add the crossBuild parameter.

If there is anything else which you would like resolved. Feel free to comment on this or that PR, and I will work to get it resolved.

@sdmaclea
sdmaclea merged commit 2a52f75 into dotnet:masterOct 9, 2020
@sdmaclea

sdmaclea commented Oct 9, 2020

Copy link
Copy Markdown
ContributorAuthor

/cc @richlander This should enable nightly runtime osx-arm64 builds.

- Linux_arm64
- Linux_musl_arm64
- ${{ if eq(variables['includeOsxOuterloop'], true) }}:
- OSX_arm64

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this... I think we don't have a helix queue setup for OSX_arm64, so I guess this broke the libraries outerloop build. I can fix this in a separate PR and we can enable them once we have a queue for it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we still dont have stable hardware for this we dont expect to enable test runs for osx_arm64 just yet. Can we leave it disabled till we have that?

@safernsafernOct 9, 2020

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I created a PR for that: #43242

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool.. thanks.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should've caught this...

@safern Thanks for being genereous, but this was clearly my fault. I forget there are three main pipelines all the time. I get the rolling outerloop and the official build pipeline confused and map them to a single pipeline...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we have so many ymls that it is sometimes hard to get it right and the easier way to add a new flavor is to try and look where we're already building osx_x64 and just add entries for the new platform there, so confusing for all I think 😄

@safernsafern left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This didn't enable mono on that new platform. @akoeplinger@steveisok do we want to do that?

Overall looks good to me. Thanks for doing this @sdmaclea

@ghostghost locked as resolved and limited conversation to collaborators Dec 7, 2020
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

osx-arm64 enable CoreCLR native component builds in CI.

6 participants

@sdmaclea@hoyosjs@safern@janvorli@wfurt@mangod9