[Android][libraries] Skip failing Android tests with ActiveIssues and proj level skips by mdh1418 · Pull Request #50800 · dotnet/runtime

mdh1418 · 2021-04-06T20:07:45Z

In effort to assess the state of the Android library tests, a number of tests will be skipped according to https://www.dropbox.com/scl/fi/hcv75228nu9b070x5sbso/State-of-Android-Library-Tests.paper?dl=0&rlkey=gck1s5hmz32jcom4fo1m7570k#:uid=471929463738166572469040&h2=All-Github-Android-issues-(Tha

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) across the different architectures of x64, x86, arm64 are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail, or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with Android had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236429

The ActiveIssues and tests.proj exclusions have been tested on #50095 as well.

ghost · 2021-04-06T20:07:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

…36848

….Tests ActiveIssues 37073

…7093

… proj skip 37094

…veIssues into 49868

…Issue 50566

…ts ActiveIssues 50567

…50570

…s 50572

…s 50574

…ctiveIssues 50575

…sues 50576

…0577

…Issues 50579

…e 50934

…eIssue 50935

…s ActiveIssue 50936

…s.Tests ActiveIssue 50937

…ssues 50942

… 50943

…ues 50944

…ests test proj skip

… test proj skip

…oj skip

…0568

… 50999

…edException related failures

…ests active issue

… issue

…e issue

… issue

mdh1418 · 2021-04-06T20:07:45Z

In effort to assess the state of the Android library tests, a number of tests will be skipped according to https://www.dropbox.com/scl/fi/hcv75228nu9b070x5sbso/State-of-Android-Library-Tests.paper?dl=0&rlkey=gck1s5hmz32jcom4fo1m7570k#:uid=471929463738166572469040&h2=All-Github-Android-issues-(Tha

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) across the different architectures of x64, x86, arm64 are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail, or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with Android had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236429

The ActiveIssues and tests.proj exclusions have been tested on #50095 as well.

ghost · 2021-04-06T20:07:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

…36848

….Tests ActiveIssues 37073

…7093

… proj skip 37094

…veIssues into 49868

…Issue 50566

…ts ActiveIssues 50567

…50570

…s 50572

…s 50574

…ctiveIssues 50575

…sues 50576

…0577

…Issues 50579

…e 50934

…eIssue 50935

…s ActiveIssue 50936

…s.Tests ActiveIssue 50937

…ssues 50942

… 50943

…ues 50944

…ests test proj skip

… test proj skip

…oj skip

…0568

… 50999

…edException related failures

…ests active issue

… issue

…e issue

… issue

mdh1418 · 2021-04-06T20:07:45Z

In effort to assess the state of the Android library tests, a number of tests will be skipped according to https://www.dropbox.com/scl/fi/hcv75228nu9b070x5sbso/State-of-Android-Library-Tests.paper?dl=0&rlkey=gck1s5hmz32jcom4fo1m7570k#:uid=471929463738166572469040&h2=All-Github-Android-issues-(Tha

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) across the different architectures of x64, x86, arm64 are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail, or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with Android had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236429

The ActiveIssues and tests.proj exclusions have been tested on #50095 as well.

ghost · 2021-04-06T20:07:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

…36848

….Tests ActiveIssues 37073

…7093

… proj skip 37094

…veIssues into 49868

…Issue 50566

…ts ActiveIssues 50567

…50570

…s 50572

…s 50574

…ctiveIssues 50575

…sues 50576

…0577

…Issues 50579

…e 50934

…eIssue 50935

…s ActiveIssue 50936

…s.Tests ActiveIssue 50937

…ssues 50942

… 50943

…ues 50944

…ests test proj skip

… test proj skip

…oj skip

…0568

… 50999

…edException related failures

…ests active issue

… issue

…e issue

… issue

mdh1418 · 2021-04-06T20:07:45Z

In effort to assess the state of the Android library tests, a number of tests will be skipped according to https://www.dropbox.com/scl/fi/hcv75228nu9b070x5sbso/State-of-Android-Library-Tests.paper?dl=0&rlkey=gck1s5hmz32jcom4fo1m7570k#:uid=471929463738166572469040&h2=All-Github-Android-issues-(Tha

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) across the different architectures of x64, x86, arm64 are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail, or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with Android had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236429

The ActiveIssues and tests.proj exclusions have been tested on #50095 as well.

ghost · 2021-04-06T20:07:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

…36848

….Tests ActiveIssues 37073

…7093

… proj skip 37094

…veIssues into 49868

…Issue 50566

…ts ActiveIssues 50567

…50570

…s 50572

…s 50574

…ctiveIssues 50575

…sues 50576

…0577

…Issues 50579

…e 50934

…eIssue 50935

…s ActiveIssue 50936

…s.Tests ActiveIssue 50937

…ssues 50942

… 50943

…ues 50944

…ests test proj skip

… test proj skip

…oj skip

…0568

… 50999

…edException related failures

…ests active issue

… issue

…e issue

… issue

mdh1418 · 2021-04-06T20:07:45Z

In effort to assess the state of the Android library tests, a number of tests will be skipped according to https://www.dropbox.com/scl/fi/hcv75228nu9b070x5sbso/State-of-Android-Library-Tests.paper?dl=0&rlkey=gck1s5hmz32jcom4fo1m7570k#:uid=471929463738166572469040&h2=All-Github-Android-issues-(Tha

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) across the different architectures of x64, x86, arm64 are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail, or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with Android had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236429

The ActiveIssues and tests.proj exclusions have been tested on #50095 as well.

ghost · 2021-04-06T20:07:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

…36848

….Tests ActiveIssues 37073

…7093

… proj skip 37094

…veIssues into 49868

…Issue 50566

…ts ActiveIssues 50567

…50570

…s 50572

…s 50574

…ctiveIssues 50575

…sues 50576

…0577

…Issues 50579

…e 50934

…eIssue 50935

…s ActiveIssue 50936

…s.Tests ActiveIssue 50937

…ssues 50942

… 50943

…ues 50944

…ests test proj skip

… test proj skip

…oj skip

…0568

… 50999

…edException related failures

…ests active issue

… issue

…e issue

… issue

mdh1418 · 2021-04-06T20:07:45Z

In effort to assess the state of the Android library tests, a number of tests will be skipped according to https://www.dropbox.com/scl/fi/hcv75228nu9b070x5sbso/State-of-Android-Library-Tests.paper?dl=0&rlkey=gck1s5hmz32jcom4fo1m7570k#:uid=471929463738166572469040&h2=All-Github-Android-issues-(Tha

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) across the different architectures of x64, x86, arm64 are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail, or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with Android had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236429

The ActiveIssues and tests.proj exclusions have been tested on #50095 as well.

ghost · 2021-04-06T20:07:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

…36848

….Tests ActiveIssues 37073

…7093

… proj skip 37094

…veIssues into 49868

…Issue 50566

…ts ActiveIssues 50567

…50570

…s 50572

…s 50574

…ctiveIssues 50575

…sues 50576

…0577

…Issues 50579

…e 50934

…eIssue 50935

…s ActiveIssue 50936

…s.Tests ActiveIssue 50937

…ssues 50942

… 50943

…ues 50944

…ests test proj skip

… test proj skip

…oj skip

…0568

… 50999

…edException related failures

…ests active issue

… issue

…e issue

… issue

mdh1418 · 2021-04-06T20:07:45Z

In effort to assess the state of the Android library tests, a number of tests will be skipped according to https://www.dropbox.com/scl/fi/hcv75228nu9b070x5sbso/State-of-Android-Library-Tests.paper?dl=0&rlkey=gck1s5hmz32jcom4fo1m7570k#:uid=471929463738166572469040&h2=All-Github-Android-issues-(Tha

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) across the different architectures of x64, x86, arm64 are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail, or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with Android had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236429

The ActiveIssues and tests.proj exclusions have been tested on #50095 as well.

ghost · 2021-04-06T20:07:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

…36848

….Tests ActiveIssues 37073

…7093

… proj skip 37094

…veIssues into 49868

…Issue 50566

…ts ActiveIssues 50567

…50570

…s 50572

…s 50574

…ctiveIssues 50575

…sues 50576

…0577

…Issues 50579

…e 50934

…eIssue 50935

…s ActiveIssue 50936

…s.Tests ActiveIssue 50937

…ssues 50942

… 50943

…ues 50944

…ests test proj skip

… test proj skip

…oj skip

…0568

… 50999

…edException related failures

…ests active issue

… issue

…e issue

… issue

mdh1418 · 2021-04-06T20:07:45Z

In effort to assess the state of the Android library tests, a number of tests will be skipped according to https://www.dropbox.com/scl/fi/hcv75228nu9b070x5sbso/State-of-Android-Library-Tests.paper?dl=0&rlkey=gck1s5hmz32jcom4fo1m7570k#:uid=471929463738166572469040&h2=All-Github-Android-issues-(Tha

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) across the different architectures of x64, x86, arm64 are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail, or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with Android had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236429

The ActiveIssues and tests.proj exclusions have been tested on #50095 as well.

ghost · 2021-04-06T20:07:49Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

…36848

….Tests ActiveIssues 37073

…7093

… proj skip 37094

…veIssues into 49868

…Issue 50566

…ts ActiveIssues 50567

…50570

…s 50572

…s 50574

…ctiveIssues 50575

…sues 50576

…0577

…Issues 50579

…e 50934

…eIssue 50935

…s ActiveIssue 50936

…s.Tests ActiveIssue 50937

…ssues 50942

… 50943

…ues 50944

…ests test proj skip

… test proj skip

…oj skip

…0568

… 50999

…edException related failures

…ests active issue

… issue

…e issue

… issue

mdh1418 requested a review from steveisok April 6, 2021 20:07

mdh1418 force-pushed the skip_android_library_tests branch 2 times, most recently from 33f91c8 to a53c6ceCompare April 7, 2021 14:49

This was referenced Apr 7, 2021
Runtime jobs being abandoned due to infra #50746
Closed
Runtime jobs being abandoned due to infra #50747
Closed

mdh1418 force-pushed the skip_android_library_tests branch from 3a38d25 to 6c95420Compare April 9, 2021 14:36

mdh1418 force-pushed the skip_android_library_tests branch from 121b6b3 to 11426bdCompare May 5, 2021 15:30

Mitchell Hwang added 22 commits May 5, 2021 11:32

[Android][libraries] Update test suites skipped
b721687

[Android][libraries] Add System.Globalization.Tests ActiveIssues 36672
3c7aab9

[Android][libraries] Add System.IO.Compression.Tests ActiveIssues 36845
96470df

[Android][libraries] Add System.Text.RegularExpressions ActiveIssues …
8d440d3
…36848

[Android][libraries] Add System.Diagnostics.DiagnosticSource.Switches…
e809a3d
….Tests ActiveIssues 37073

[Android][libraries] Add System.Net.Requests.Tests ActiveIssues 37087
443536e

[Android][libraries] Add System.Runtime.Tests test proj skip 37088
2f19c15

[Android][libraries] Add System.Runtime.Numerics.Tests ActiveIssues 3…
71b958f
…7093

[Android][libraries] Add System.Security.Cryptography.Pkcs.Tests test…
63e8d60
… proj skip 37094

[Android][libraries] Add System.Console.Tests Active Issues 37465
7865227

[Android][libraries] Consolidate System.Runtime.Extensions.Tests Acti…
19dacd6
…veIssues into 49868

[Android][libraries] Add System.Threading.Tasks.Parallel.Tests Active…
9c99c47
…Issue 50566

[Android][libraries] Add System.Net.NetworkInformation.Functional.Tes…
2f2ccfd
…ts ActiveIssues 50567

[Android][libraries] Add System.Net.Sockets.Tests ActiveIssues 50568
d424a07

[Android][libraries] Add System.Diagnostics.Debug.Tests ActiveIssues …
f8e2c4a
…50570

[Android][libraries] Add Microsoft.VisualBasic.Core.Tests ActiveIssue…
ca34370
…s 50572

[Android][libraries] Add System.Text.Encoding.Tests ActiveIssues 50573
ee0c515

[Android][libraries] Add System.Net.Ping.Functional.Tests ActiveIssue…
394886e
…s 50574

[Android][libraries] Add Microsoft.Extensions.Logging.Console.Tests A…
8cd6099
…ctiveIssues 50575

[Android][libraries] Add System.Collections.Concurrent.Tests ActiveIs…
6792e54
…sues 50576

[Android][libraries] Add System.Threading.Thread.Tests ActiveIssues 5…
43d731a
…0577

[Android][libraries] Rename System.Collections.Immutable.Tests Active…
f802ae0
…Issues 50579

Mitchell Hwang added 24 commits May 5, 2021 11:32

[Android][libraries] Add System.Resources.Extensions.Tests ActiveIssu…
7fc8006
…e 50934

[Android][libraries] Add System.Resources.ResourceManager.Tests Activ…
391c643
…eIssue 50935

[Android][libraries] Add System.Runtime.Serialization.Primitives.Test…
18546d1
…s ActiveIssue 50936

[Android][libraries] Add System.Security.Cryptography.X509Certificate…
a323b8b
…s.Tests ActiveIssue 50937

[Android][libraries] Add System.Text.Json.Tests test proj skip
7245f13

[Android][libraries] Add System.Text.RegularExpressions.Tests ActiveI…
cf41ca9
…ssues 50942

[Android][libraries] Add System.Threading.Channels.Tests ActiveIssues…
06dc65e
… 50943

[Android][libraries] Add System.Xml.Linq.xNodeBuilder.Tests ActiveIss…
b959759
…ues 50944

[Android][libraries] Add System.Xml.XmlSchema.XmlSchemaValidatorApi.T…
9a14825
…ests test proj skip

[Android][libraries] Add System.Xml.XmlSchemaSet.Tests test proj skip
96e9914

[Android][libraries] Add System.Xml.Xsl.XslCompiledTransformApi.Tests…
9a99182
… test proj skip

[Android][libraries] Add System.Xml.Xsl.XslTransformApi.Tests test pr…
3f79697
…oj skip

[Android][libraries] Add System.Net.Sockets ActiveIssue EventSource 5…
94cbe92
…0568

[Android][libraries] Add System.Net.Mail.Functional.Tests ActiveIssue…
ebd5fe4
… 50999

[Android][libraries] Clean up Android test architecture specific skips
5dec8b4

[Android][libraries] Add more test suite skips
f3c2c31

[Android][libraries] Skip test suites failing with PlatformNotSupport…
96f8df7
…edException related failures

[Android][libraries] Add more suites that have execution compromised
11426bd

[Android] Add new Microsoft.Extensions.Configuration.FileExtensions.T…
ac76ccd
…ests active issue

[Android] Add new Microsoft.Extensions.Configuration.Ini.Tests active…
a991609
… issue

[Android] Add new Microsoft.Extensions.Configuration.Json.Tests activ…
144547a
…e issue

[Android] Add new Microsoft.Extensions.Configuration.Xml.Tests active…
b31d1f1
… issue

[Android] Add new Microsoft.Extensions.Http.Tests active issue
5679e4b

[Android] Fix System.Text.Json.Tests project path skip
3474770

akoeplinger approved these changes May 10, 2021
View reviewed changes

mdh1418 merged commit 913fdbe into dotnet:mainMay 10, 2021

mdh1418 deleted the skip_android_library_tests branch May 10, 2021 15:16

mdh1418 mentioned this pull request May 19, 2021
Skip flakey mobile library test crashes #52931
Merged

karelz added this to the 6.0.0 milestone May 20, 2021

ghost locked as resolved and limited conversation to collaborators Jun 19, 2021