Build in Azure Pipelines with cached vcpkg artifacts by lukka · Pull Request #114 · microsoft/STL

Luca (lukka) · 2019-09-18T08:06:59Z

Please note that acceptance of community PRs will be delayed while we are
bringing our test and CI systems online. For more information, see the
README.md.

Description

I hope this PR is either useful and could be an inspiration on how to set up a CI system.
This PR is not meant to be merged as is, but it is created in order to show how it is possible to run a build using CMake and Ninja with cached vcpkg artifacts on Hosted Build Agents.

Hosted agents do not have the required MSVC 14.23, hence the build also install that toolset on the fly (considerably increasing build time).

The build time on Hosted agents with cached vcpkg artifacts is about:

restoring cache 30 seconds;
building 4 minutes.
building vcpkg artifacts 15 minutes;

See here for all build samples.

No tests are run.

Also, this PR shows how to explicitly define which vcpkg version is to be used by using a git submodule.

Checklist:

I understand README.md.
The STL builds and test harnesses have passed (must be manually verified
by an STL maintainer before CI is online, leave this unchecked for initial
submission).

Billy O'Neal (BillyONeal) · 2019-09-19T22:44:26Z

We are absolutely interested in setting up an Azure Pipelines (or maybe GitHub Actions) script for this repo, but we are looking into solutions for hosting the build agents. The default provided "hosted" agents are little dual core boxes, and once we get our tests in here you'd likely be waiting 8+ hours for an answer.

Unfortunately, with a self hosted build agent system, there don't appear to be good options to handle untrusted pull request submissions. We may just put Azure DevOps in "manually initiated" mode but we'd like to give people more immediate feedback (before review) if possible.

I've asked the Pipelines team if they have any ideas but haven't gotten reasonable answers back yet.

Thanks for your contribution!

Charles Milette (sylveon) · 2019-09-19T22:47:33Z

Just a thought, there could be a thing running on the hosted agents which only builds the tests to do basic validity checking and then use a GitHub bot to trigger the full test suite internally.

Luca (lukka) · 2019-09-19T22:57:25Z

Billy O'Neal (@BillyONeal) self hosted agents are the way to go indeed (the Microsoft hosted agents do not have the prerequisites to build anyway).
What about expendable VMs created off an existing image on Azure Compute? or the biggest size not powerful enough? If those are not and you need some extremely powerful physical machine, all they need is to not be part of corporate network and then you save any security concern.

Billy O'Neal (BillyONeal) · 2019-09-21T02:53:26Z

Just a thought, there could be a thing running on the hosted agents which only builds the tests to do basic validity checking and then use a GitHub bot to trigger the full test suite internally.

That wouldn't prevent a PR from sending malicious code to our internal test infrastructure that isn't prepared to deal with that.

What about expendable VMs created off an existing image on Azure Compute?

That would work fine but I don't think there's a great way to get Azure DevOps to recycle the machine upon completion (which is necessary if we want to build and run arbitrary internet submitted code).

If those are not and you need some extremely powerful physical machine, all they need is to not be part of corporate network and then you save any security concern.

Just because the machine isn't on the corporate network doesn't mean there aren't serious problems with letting an attacker gain persistence. For example, using Azure bandwidth to attack other Internet entities, or mining bitcoins on our hardware.

Azure DevOps does have an "OK to test check" thing ( https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/github?view=azure-devops&tabs=yaml#comment-triggers ) we'll probably end up using for now.

Charles Milette (sylveon) · 2019-09-21T03:15:04Z

I meant a manually triggered GitHub bot, basically like the comment triggers of DevOps.

Luca (lukka) · 2019-09-23T07:16:33Z

Running the build agent on a docker image stored in Azure Container Registry (and that contains all pre-reqs for building) could be a good fit, as long it is going to be cached locally on the host to start up quickly. Have not tried this scenario yet though.

Billy O'Neal (BillyONeal) · 2019-09-25T00:14:53Z

Stephan T. Lavavej (@StephanTLavavej) Casey Carter (@CaseyCarter) Charlie Barto (@barcharcraz) Are you OK with the submodule to get vcpkg here? Since we're using a self-hosted agent I can just put that in the agent setup instructions but given that we already have a contribution that does that setup for us which will work if/when we can use the hosted system, I'm reluctant to throw that work away.

Charlie Barto (barcharcraz) · 2019-09-25T00:16:23Z

@@ -0,0 +1,5 @@
+boost-build:x86-windows


why do we need boost build explicitly listed?

in my experience, when running 'vcpkg install boost-math:x86-windows', it failed asking me to install 'boost-build:x86-windows' as well. And doing this failed with same error: 'vcpkg install boost-build:x86-windows boost-math:x86-windows'. I had to run first the installation of boost-build:x86-windows, and then run the installation of boost-math:x86-windows separately to succeed.

this might be a issue worth reporting in vcpkg, but I'm not sure.

Charlie Barto (barcharcraz) · 2019-09-25T00:17:32Z

Stephan T. Lavavej (@StephanTLavavej) Stephan T. Lavavej FTE Casey Carter (@CaseyCarter) Casey Carter FTE Charlie Barto (@barcharcraz) Charlie Barto FTE Are you OK with the submodule to get vcpkg here? Since we're using a self-hosted agent I can just put that in the agent setup instructions but given that we already have a contribution that does that setup for us which will work if/when we can use the hosted system, I'm reluctant to throw that work away.

Also, I'm OK with it as long as we know it can work with the new caching feature.

VCPKG itself is pretty small, it's the built artifacts that take a long time.

Billy O'Neal (BillyONeal) · 2019-09-25T00:22:19Z

Hmmm this PR was created without the ability for us to edit it, so I can't try it out to see if it'll work on our self-hosted agent. Do you mind if I recreate this in my fork?

Thanks for your contribution!

Billy O'Neal (BillyONeal) · 2019-09-25T00:32:22Z

/AzurePipelines run STL

azure-pipelines · 2019-09-25T00:32:28Z

No pipelines are associated with this pull request.

Luca (lukka) · 2019-09-25T00:55:33Z

i can also do the requested changes here, just let me know what works better for you Billy O'Neal (@BillyONeal)

Casey Carter (CaseyCarter) · 2019-09-25T01:14:15Z

Stephan T. Lavavej (@StephanTLavavej) Stephan T. Lavavej FTE Casey Carter (@CaseyCarter) Casey Carter FTE Charlie Barto (@barcharcraz) Charlie Barto FTE Are you OK with the submodule to get vcpkg here? Since we're using a self-hosted agent I can just put that in the agent setup instructions but given that we already have a contribution that does that setup for us which will work if/when we can use the hosted system, I'm reluctant to throw that work away.

Also, I'm OK with it as long as we know it can work with the new caching feature.

VCPKG itself is pretty small, it's the built artifacts that take a long time.

Despite my earlier private comments that we should avoid adding a submodule, I'm now in favor of whatever gets us up and running the quickest. We can refine things later if we find improvements, but for now we shouldn't let the perfect be the enemy of the good.

Billy O'Neal (BillyONeal) · 2019-09-25T02:33:40Z

i can also do the requested changes here, just let me know what works better for you Billy O'Neal (@BillyONeal)

If you merge with master (please preserve the clang-format validation in at least one configuration) it should have the bits working. Note that it won't actually attempt to run the changes automatically without pre-review from us.

Billy O'Neal (BillyONeal) · 2019-09-25T02:35:06Z

i can also do the requested changes here, just let me know what works better for you Billy O'Neal (@BillyONeal)

If you merge with origin/master it should have the bits that indicate to run using the machine on my desk. It will require pre-review from us though... since it's a self-hosted agent and we don't have a mechanism to automatically recycle it we need pre-review to ensure folks aren't submitting malware or similar.

Billy O'Neal (BillyONeal) · 2019-09-25T02:51:53Z

/AzurePipelines run

azure-pipelines · 2019-09-25T02:52:03Z

Azure Pipelines failed to run 1 pipeline(s).

Billy O'Neal (BillyONeal) · 2019-09-25T02:53:46Z

That agent was set up like this: https://github.com/microsoft/STL/wiki/Setting-up-an-Azure-Pipelines-Agent

Luca (lukka) · 2019-09-25T02:53:48Z

Billy O'Neal (@BillyONeal) the tasks needs to be installed from the marketplace

Casey Carter (CaseyCarter) · 2019-09-25T21:16:41Z

@@ -0,0 +1,51 @@
+parameters:
+  targetPlatform: 'x64'


Why have a parameters block here when the parameters come from azure-pipelines.yml?

Here is declaration and default value of accepted params. The actual value is indeed in a-p.yml

Rephrasing: do we need to have this declaration here, or can we leave it out so this template will fail when it's not used correctly? It would be a shame if someone accidentally introduced a typo in the parameter names (e.g., targetP1atform) in azure-pipelines.yml and we consequently build x64 four times for every PR.

I suppose if it MUST be here we could give it an invalid value like potato to ensure it will fail very loudly.

Billy O'Neal (BillyONeal) · 2019-09-25T22:30:01Z

/azp run

azure-pipelines · 2019-09-25T22:30:13Z

Azure Pipelines successfully started running 1 pipeline(s).

Stephan T. Lavavej (StephanTLavavej) · 2019-09-25T22:46:22Z

+    [string]$bootstrapperExe = Join-Path ${env:Temp} ([System.IO.Path]::GetRandomFileName() + ".exe")
+    Invoke-WebRequest -Uri $VSBootstrapperURL -OutFile $bootstrapperExe
+
+    $Arguments = ('/c', $bootstrapperExe, $WorkLoads, '--quiet', '--norestart', '--wait', '--nocache' )


Nitpick: Why is there a space before the final )?

Stephan T. Lavavej (StephanTLavavej) · 2019-09-25T22:47:03Z

+    }
+    else
+    {
+      Write-Host -Object "Non zero exit code returned by the installation process : $exitCode."


Nitpick: Nonzero or Non-zero.

Stephan T. Lavavej (StephanTLavavej) · 2019-09-25T22:48:01Z

+}
+
+# Invalidate the standard installation of VS on the hosted agent.
+Move-Item "C:/Program Files (x86)/Microsoft Visual Studio/2019/Enterprise/" "C:/Program Files (x86)/Microsoft Visual Studio/2019/nouse/" -Verbose


This seems highly invasive. Is it desirable? Could we simply fail if we detect an existing installation of VS?

This script is only run on the "hosted" machines which are blown away after each build, so invasive changes are fine/normal.

then you would always fail on the hosted one.

Stephan T. Lavavej (StephanTLavavej) · 2019-09-25T22:49:23Z

+              '--add Microsoft.VisualStudio.Component.VC.Tools.x86.x64 ' + `
+              '--add Microsoft.VisualStudio.Component.VC.Tools.ARM64 ' + `
+              '--add Microsoft.VisualStudio.Component.VC.Tools.ARM ' + `
+              '--add Microsoft.VisualStudio.Component.Windows10SDK.18362 '


Is there any way to request the latest Win10 SDK, instead of encoding this version number here that might need to be updated?

Not as far as I am aware.

Stephan T. Lavavej (StephanTLavavej) · 2019-09-25T22:53:03Z

+      inputs:
+        cmakeListsTxtPath: 'CMakeSettings.json'
+        useVcpkgToolchainFile: true
+        configurationRegexFilter: '.*${{ parameters.targetPlatform }}.*'


Is this doing a whole-string or a substring match? If the latter, the .* at the front and back are unnecessary.

I'm not positive which it is but I also think it doesn't matter. (And we should merge this as-is)

Stephan T. Lavavej (StephanTLavavej) · 2019-09-25T22:53:40Z

+# Copyright (c) Microsoft Corporation.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+# Build STL targeting x64, arm64, x86, arm.


Our usual order is x86, x64, arm, arm64. Is this order significant in some way?

Stephan T. Lavavej (StephanTLavavej) · 2019-09-25T22:57:34Z

+boost-math:x64-windows
+boost-math:x86-windows
+boost-math:arm-windows
+boost-math:arm64-windows


Does this file have to be in the root of the repo? Would azure-devops be a better location? It's mentioned by azure-devops/run_build.yml.

Billy O'Neal (BillyONeal) · 2019-09-26T00:02:47Z

/azp run

azure-pipelines · 2019-09-26T00:02:58Z

Azure Pipelines successfully started running 1 pipeline(s).

Stephan T. Lavavej (StephanTLavavej) · 2019-09-26T02:04:49Z

+  parameters:
+    targetPlatform: x64
+
+


Is there a reason for there to be two newlines here?

Stephan T. Lavavej (StephanTLavavej) · 2019-09-26T02:05:12Z

@@ -0,0 +1,5 @@
+boost-build:x86-windows
+boost-math:x64-windows
+boost-math:x86-windows


I observe that this mentions x64 before x86.

Billy O'Neal (BillyONeal) · 2019-09-26T02:08:19Z

/azp run

azure-pipelines · 2019-09-26T02:08:29Z

Azure Pipelines successfully started running 1 pipeline(s).

This is provided by azure-pipelines.yml. See: microsoft#114 (comment)

Luca (lukka) requested review from Billy O'Neal (BillyONeal), Casey Carter (CaseyCarter), Stephan T. Lavavej (StephanTLavavej) and Charlie Barto (barcharcraz) as code owners September 18, 2019 08:06

Billy O'Neal (BillyONeal) self-assigned this Sep 19, 2019

Charlie Barto (barcharcraz) reviewed Sep 25, 2019

View reviewed changes

Comment thread .gitmodules

Charlie Barto (barcharcraz) reviewed Sep 25, 2019

View reviewed changes

Comment thread ado/run_build.yml Outdated

Charlie Barto (barcharcraz) reviewed Sep 25, 2019

View reviewed changes

add azure-pipeline for x64, arm64, x86, arm

24f4cc6

Luca (lukka) force-pushed the master branch from 584c744 to 24f4cc6 Compare September 25, 2019 02:49

Luca (lukka) requested a review from a team as a code owner September 25, 2019 02:49

Billy O'Neal (BillyONeal) mentioned this pull request Sep 25, 2019

P0439R0 enum class memory_order #124

Merged

5 tasks

Casey Carter (CaseyCarter) suggested changes Sep 25, 2019

View reviewed changes

Billy O'Neal (BillyONeal) added 2 commits September 25, 2019 15:28

Add newlines and license blocks, and remove Spectre libs.

44d4d31

Deindent install preview comment

ab4391f

Billy O'Neal (BillyONeal) requested a review from Casey Carter (CaseyCarter) September 25, 2019 22:29

Charlie Barto (barcharcraz) approved these changes Sep 25, 2019

View reviewed changes

Stephan T. Lavavej (StephanTLavavej) reviewed Sep 25, 2019

View reviewed changes

Casey Carter (CaseyCarter) approved these changes Sep 25, 2019

View reviewed changes

Luca (lukka) and others added 2 commits September 25, 2019 16:42

cosmetic fixes

3a5d74e

Reorder platforms and add license to .gitmodules.

87912e8

Billy O'Neal (BillyONeal) requested a review from Stephan T. Lavavej (StephanTLavavej) September 26, 2019 00:09

Stephan T. Lavavej (StephanTLavavej) approved these changes Sep 26, 2019

View reviewed changes

Stephan's nitpicks

f777520

Stephan T. Lavavej (StephanTLavavej) approved these changes Sep 26, 2019

View reviewed changes

Billy O'Neal (BillyONeal) merged commit d7c7691 into microsoft:master Sep 26, 2019

Billy O'Neal (BillyONeal) mentioned this pull request Sep 26, 2019

Move cgmanifest.json #138

Merged

7 tasks

Billy O'Neal (BillyONeal) added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

ad30215

Billy O'Neal (BillyONeal) added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

a1affaa

Billy O'Neal (BillyONeal) added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

5787f4a

Billy O'Neal (BillyONeal) added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

0e547bc

Billy O'Neal (BillyONeal) added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

ada27eb

Billy O'Neal (BillyONeal) added a commit to BillyONeal/cpprestsdk that referenced this pull request Sep 27, 2019

Attempt templates and other tricks I learned from microsoft/STL#114

8f0a93e

Stephan T. Lavavej (StephanTLavavej) added a commit to StephanTLavavej/STL that referenced this pull request Nov 5, 2019

Attempt to remove default targetPlatform from run_build.yml.

a178840

This is provided by azure-pipelines.yml. See: microsoft#114 (comment)

Stephan T. Lavavej (StephanTLavavej) mentioned this pull request Apr 29, 2020

README.md: Use the vcpkg submodule instead of separately cloning #761

Closed

Uh oh!

Conversation

Luca (lukka) commented Sep 18, 2019 • edited by BillyONeal Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist:

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 19, 2019

Uh oh!

Charles Milette (sylveon) commented Sep 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Luca (lukka) commented Sep 19, 2019

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 21, 2019

Uh oh!

Charles Milette (sylveon) commented Sep 21, 2019

Uh oh!

Luca (lukka) commented Sep 23, 2019

Uh oh!

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 25, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Charlie Barto (barcharcraz) commented Sep 25, 2019

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 25, 2019

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 25, 2019

Uh oh!

azure-pipelines Bot commented Sep 25, 2019

Uh oh!

Luca (lukka) commented Sep 25, 2019

Uh oh!

Casey Carter (CaseyCarter) commented Sep 25, 2019

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 25, 2019

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 25, 2019

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 25, 2019

Uh oh!

azure-pipelines Bot commented Sep 25, 2019

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 25, 2019

Uh oh!

Luca (lukka) commented Sep 25, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Casey Carter (CaseyCarter) Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Casey Carter (CaseyCarter) Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Billy O'Neal (BillyONeal) commented Sep 25, 2019

Uh oh!

azure-pipelines Bot commented Sep 25, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Luca (lukka) commented Sep 18, 2019 •

edited by BillyONeal

Loading

Charles Milette (sylveon) commented Sep 19, 2019 •

edited

Loading

Casey Carter (CaseyCarter) Sep 25, 2019 •

edited

Loading

Casey Carter (CaseyCarter) Sep 25, 2019 •

edited

Loading

Billy O'Neal (BillyONeal) Sep 26, 2019 •

edited

Loading