[tests] Stand up iOS arm runtime tests and enable tracing/eventpipe tests by mdh1418 · Pull Request #79169 · dotnet/runtime

mdh1418 · 2022-12-02T19:45:24Z

On iOS, runtime tests were previously only stood up on x64 (Simulator) in #43954.
This PR looks to extend the runtime testing coverage to iOS devices and arm simulators by doing the following:

Provide AOT'd assemblies to the AppleAppBuilder in the runtime build flow following the conventions of library tests (previously no assemblies were provided to the AppleAppBuilder)
Default RunAOTCompilation and MonoForceInterpreter properties in the runtime build flow for iOS
Enable diagnostics_tracing for iOS runtime tests
Reenable runtime tracing/eventpipe tests on iOS not involving subprocesses
Tweak ProcessInfo and ProcessInfo2 tests to pass on iOS
Skip IpcTraceTest.EnsureCleanEnvironment as ProcessManager.GetProcessInfos is not supported on iOS
Align AppleAppBuilder parameter order with the order of definition in the task

Fixes: #81233

ghost · 2022-12-02T19:45:31Z

Tagging subscribers to this area: @directhex
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #74891

On iOS, runtime tests were previously only stood up on x64 (Simulator) in #43954.
This PR looks to extend the runtime testing coverage to iOS devices and arm simulators by doing the following:

Provide AOT'd assemblies to the AppleAppBuilder in the runtime build flow following the conventions of library tests (previously no assemblies were provided to the AppleAppBuilder)
Default RunAOTCompilation and MonoForceInterpreter properties in the runtime build flow for iOS
Enable diagnostics_tracing for iOS runtime tests
Reenable runtime tracing/eventpipe tests on iOS not involving subprocesses
Tweak ProcessInfo and ProcessInfo2 tests to pass on iOS
Skip IpcTraceTest.EnsureCleanEnvironment as ProcessManager.GetProcessInfos is not supported on iOS
Align AppleAppBuilder parameter order with the order of definition in the task

Author:	mdh1418
Assignees:	-
Labels:	`area-Infrastructure-mono`, `runtime-mono`
Milestone:	-

naricc · 2022-12-16T21:21:15Z

I'm a little confused here; what happens if we are running iOSSimulator and both RunAOTCompilation and MonoForceInterpreter are true? Does this change how iOSSimulator tests are run?

For iOSlike devices and arm based simulators, AOT is needed for the tests to build. (At least that was the case for library tests, which motivated this PR #76551) When setting RunAOTCompilation and MonoForceInterpreter to true, what will happen is AOT will be ran, and if anything goes wrong, interp will be the fallback instead of JIT (which is important because JIT does not work on iOSlike platforms).
I believe the RunAOTCompilation is necessary here for arm architectures, but we can exclude x64 if thats preferred.

mdh1418 · 2022-12-16T23:30:33Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2022-12-16T23:30:44Z

Azure Pipelines successfully started running 2 pipeline(s).

mdh1418 · 2022-12-19T16:56:00Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2022-12-19T16:56:14Z

Azure Pipelines successfully started running 2 pipeline(s).

mdh1418 · 2022-12-19T18:05:31Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2022-12-19T18:05:44Z

Azure Pipelines successfully started running 2 pipeline(s).

naricc · 2022-12-19T22:00:41Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2022-12-19T22:02:16Z

Azure Pipelines successfully started running 2 pipeline(s).

naricc · 2022-12-20T15:45:28Z

/azp run runtime-ioslike

azure-pipelines · 2022-12-20T15:45:39Z

Azure Pipelines successfully started running 1 pipeline(s).

naricc · 2023-01-06T18:35:43Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2023-01-06T18:35:57Z

Azure Pipelines failed to run 2 pipeline(s).

kotlarmilos · 2023-04-13T09:49:59Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2023-04-13T09:50:23Z

Azure Pipelines successfully started running 2 pipeline(s).

kotlarmilos · 2023-04-13T12:53:15Z

The failure shouldn't be related. It has been tracked in dotnet/arcade#11683.

mdh1418

Just a bunch of nits and Jeremy's suggestion for the test grouping, but looks good overall to me!

Would approve, but the approve button is greyed out for me because I was the original PR author

steveisok

Looks good!

kotlarmilos · 2023-04-14T15:08:07Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2023-04-14T15:08:22Z

Azure Pipelines successfully started running 2 pipeline(s).

kotlarmilos · 2023-04-17T08:38:33Z

/azp run runtime-ioslikesimulator

azure-pipelines · 2023-04-17T08:38:44Z

Azure Pipelines successfully started running 1 pipeline(s).

mdh1418 · 2023-04-17T15:22:41Z

/azp run runtime-ioslikesimulator

azure-pipelines · 2023-04-17T15:22:56Z

Azure Pipelines successfully started running 1 pipeline(s).

kotlarmilos · 2023-04-18T03:14:12Z

/azp run runtime-ioslike,runtime-ioslikesimulator

azure-pipelines · 2023-04-18T03:14:27Z

Azure Pipelines successfully started running 2 pipeline(s).

kotlarmilos · 2023-04-18T14:06:12Z

Failures shouldn't be related. The TCP failure is known issue dotnet/arcade#11683.

kotlarmilos · 2023-04-18T17:46:03Z

The failures are known #84979 and #84995.

mdh1418 added area-Infrastructure-mono runtime-mono specific to the Mono runtime labels Dec 2, 2022

mdh1418 requested review from akoeplinger, fanyang-mono, lateralusX, naricc and steveisok December 2, 2022 19:45

mdh1418 requested a review from marek-safar as a code owner December 2, 2022 19:45

ghost assigned mdh1418 Dec 2, 2022

naricc reviewed Dec 16, 2022
View reviewed changes

This was referenced Dec 17, 2022
Tracking issue for CI build timeouts #76454
Closed
Precondition failure: File has not had execution verified #79439
Closed
[wasm] Library tests failing during linking for AOT - SIGKILL#79569
Closed

mdh1418 commented Dec 19, 2022
View reviewed changes

Comment threadeng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml Outdated

build-analysisBot mentioned this pull request Dec 20, 2022
GenerateHWIntrinsicTests_Arm/project.assets.json already exists #79843
Closed

build-analysisBot mentioned this pull request Jan 6, 2023
TestResourceManagerIsSafeForConcurrentAccessAndEnumeration fails intermittently in CI #80277
Closed

naricc requested a review from vargaz as a code owner January 6, 2023 20:53

Resolve PR comments
cbccac5

mdh1418 commented Apr 13, 2023
View reviewed changes

fanyang-mono reviewed Apr 13, 2023
View reviewed changes

Comment threadeng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml Outdated
Comment threadsrc/tests/Common/CLRTest.Execute.Bash.targets
Comment threadsrc/tests/Common/Coreclr.TestWrapper/MobileAppHandler.cs Outdated

steveisok approved these changes Apr 14, 2023
View reviewed changes

Resolve PR comments
3645fee

Test iossimulator_arm64 on rolling builds
f220b4e

build-analysisBot mentioned this pull request Apr 17, 2023
[6.0] Android test failure: Failed to install mobile app #83422
Open

Revert test iossimulator_arm64 on rolling builds
ec4988b

steveisok approved these changes Apr 18, 2023
View reviewed changes

Comment threadeng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml Outdated

Remove isRollingBuild condition
d6fd78a

kotlarmilos mentioned this pull request Apr 18, 2023
Could not load file or assembly 'Microsoft.CodeAnalysis.NetAnalyzers #84995
Closed

kotlarmilos merged commit 790b14b into dotnet:mainApr 18, 2023

kotlarmilos mentioned this pull request Apr 28, 2023
how can I run runtime tests with monovm on ios? #85160
Closed

ghost locked as resolved and limited conversation to collaborators May 18, 2023

mdh1418 deleted the ios_investigate_eventpipe_failures branch June 10, 2024 20:34

Uh oh!

Conversation

mdh1418 commented Dec 2, 2022• edited by naricc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ghost commented Dec 2, 2022

Uh oh!

nariccDec 16, 2022

Choose a reason for hiding this comment

Uh oh!

mdh1418Dec 19, 2022

Choose a reason for hiding this comment

Uh oh!

mdh1418 commented Dec 16, 2022

Uh oh!

azure-pipelinesBot commented Dec 16, 2022

Uh oh!

mdh1418 commented Dec 19, 2022

Uh oh!

azure-pipelinesBot commented Dec 19, 2022

Uh oh!

mdh1418 commented Dec 19, 2022

Uh oh!

azure-pipelinesBot commented Dec 19, 2022

Uh oh!

Uh oh!

naricc commented Dec 19, 2022

Uh oh!

azure-pipelinesBot commented Dec 19, 2022

Uh oh!

naricc commented Dec 20, 2022

Uh oh!

azure-pipelinesBot commented Dec 20, 2022

Uh oh!

naricc commented Jan 6, 2023

Uh oh!

azure-pipelinesBot commented Jan 6, 2023

Uh oh!

kotlarmilos commented Apr 13, 2023

Uh oh!

azure-pipelinesBot commented Apr 13, 2023

Uh oh!

kotlarmilos commented Apr 13, 2023

Uh oh!

mdh1418 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveisok left a comment

Choose a reason for hiding this comment

Uh oh!

kotlarmilos commented Apr 14, 2023

Uh oh!

azure-pipelinesBot commented Apr 14, 2023

Uh oh!

kotlarmilos commented Apr 17, 2023

Uh oh!

azure-pipelinesBot commented Apr 17, 2023

Uh oh!

mdh1418 commented Apr 17, 2023

Uh oh!

azure-pipelinesBot commented Apr 17, 2023

Uh oh!

kotlarmilos commented Apr 18, 2023

Uh oh!

azure-pipelinesBot commented Apr 18, 2023

Uh oh!

mdh1418 commented Dec 2, 2022•
edited by naricc
Loading

mdh1418 left a comment •
edited
Loading