[iOS][libraries] Skip failing iOS tests with ActiveIssues and proj level skips by mdh1418 · Pull Request #51491 · dotnet/runtime

mdh1418 · 2021-04-19T13:58:25Z

In effort to assess the state of the iOS library tests, a number of tests will be skipped

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with iOS had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException. [SkipOnPlatform] attributes are attached to the failing test methods.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436

The ActiveIssues and tests.proj exclusions have been tested on #49917 as well.

This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj - Removed $(NetCoreAppCurrent)-iOS from target frameworks as it is stated to currently only supported on Windows.

src/libraries/tests.proj - Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.

The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]

ghost · 2021-04-19T13:58:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

marek-safar · 2021-04-30T13:33:41Z

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

Which are they?

mdh1418 · 2021-04-30T14:06:51Z

@marek-safar So far the PNSE failure test suites being completely skipped are System.Console.Tests.csproj, System.IO.FileSystem.Tests.csproj, System.Security.Cryptography.OpenSsl.Tests.csproj, System.Threading.Thread.Tests.csproj. There is some more consolidation and checking to be done, especially since tests cannot run to completion related to https://github.com/dotnet/core-eng/issues/12969

There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE.

marek-safar · 2021-04-30T16:53:24Z

Out of these 4 I think only System.Security.Cryptography.OpenSsl.Tests.csproj should be skipped completely.

tqiu8 · 2021-04-30T21:10:04Z

On my local machine System.Threading.Thread is passing.

mdh1418 · 2021-05-03T14:34:21Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

mdh1418 · 2021-05-03T14:36:56Z

@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?


System.Console.Tests.dll - 46
System.Diagnostics.Process.Tests.dll - 12
System.IO.FileSystem.Tests.dll - 3 System.Net.NetworkInformation.Functional.Tests.dll - 1 System.Net.Ping.Functional.Tests.dll - 25 System.Security.Cryptography.OpenSsl.Tests.dll - 726
System.Security.Cryptography.X509Certificates.Tests.dll - 163
System.Security.Cryptography.Xml.Tests.dll - 3 System.Runtime.Extensions.Tests.dll - 1
System.Threading.Thread.Tests.dll - 1
System.Xml.RW.XmlReader.Tests.dll - 3

all_ios_tvos_PNSE_us.txt

tqiu8 · 2021-05-03T14:39:49Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

Yep, that's probably what happened.

marek-safar · 2021-05-03T15:36:20Z

Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?

Preferably not

…m.Net.NameResolution.Unit test suite skips

… fix typo

This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

…Issue

…2614

…ng on CI but passes locally

…on CI but passes locally

…sing

…Algorithms test exclusions

…tiveIssues

akoeplinger · 2021-05-12T15:36:36Z

Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)

mdh1418 · 2021-04-19T13:58:25Z

In effort to assess the state of the iOS library tests, a number of tests will be skipped

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with iOS had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException. [SkipOnPlatform] attributes are attached to the failing test methods.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436

The ActiveIssues and tests.proj exclusions have been tested on #49917 as well.

This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj - Removed $(NetCoreAppCurrent)-iOS from target frameworks as it is stated to currently only supported on Windows.

src/libraries/tests.proj - Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.

The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]

ghost · 2021-04-19T13:58:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

marek-safar · 2021-04-30T13:33:41Z

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

Which are they?

mdh1418 · 2021-04-30T14:06:51Z

@marek-safar So far the PNSE failure test suites being completely skipped are System.Console.Tests.csproj, System.IO.FileSystem.Tests.csproj, System.Security.Cryptography.OpenSsl.Tests.csproj, System.Threading.Thread.Tests.csproj. There is some more consolidation and checking to be done, especially since tests cannot run to completion related to https://github.com/dotnet/core-eng/issues/12969

There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE.

marek-safar · 2021-04-30T16:53:24Z

Out of these 4 I think only System.Security.Cryptography.OpenSsl.Tests.csproj should be skipped completely.

tqiu8 · 2021-04-30T21:10:04Z

On my local machine System.Threading.Thread is passing.

mdh1418 · 2021-05-03T14:34:21Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

mdh1418 · 2021-05-03T14:36:56Z

@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?


System.Console.Tests.dll - 46
System.Diagnostics.Process.Tests.dll - 12
System.IO.FileSystem.Tests.dll - 3 System.Net.NetworkInformation.Functional.Tests.dll - 1 System.Net.Ping.Functional.Tests.dll - 25 System.Security.Cryptography.OpenSsl.Tests.dll - 726
System.Security.Cryptography.X509Certificates.Tests.dll - 163
System.Security.Cryptography.Xml.Tests.dll - 3 System.Runtime.Extensions.Tests.dll - 1
System.Threading.Thread.Tests.dll - 1
System.Xml.RW.XmlReader.Tests.dll - 3

all_ios_tvos_PNSE_us.txt

tqiu8 · 2021-05-03T14:39:49Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

Yep, that's probably what happened.

marek-safar · 2021-05-03T15:36:20Z

Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?

Preferably not

…m.Net.NameResolution.Unit test suite skips

… fix typo

This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

…Issue

…2614

…ng on CI but passes locally

…on CI but passes locally

…sing

…Algorithms test exclusions

…tiveIssues

akoeplinger · 2021-05-12T15:36:36Z

Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)

mdh1418 · 2021-04-19T13:58:25Z

In effort to assess the state of the iOS library tests, a number of tests will be skipped

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with iOS had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException. [SkipOnPlatform] attributes are attached to the failing test methods.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436

The ActiveIssues and tests.proj exclusions have been tested on #49917 as well.

This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj - Removed $(NetCoreAppCurrent)-iOS from target frameworks as it is stated to currently only supported on Windows.

src/libraries/tests.proj - Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.

The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]

ghost · 2021-04-19T13:58:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

marek-safar · 2021-04-30T13:33:41Z

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

Which are they?

mdh1418 · 2021-04-30T14:06:51Z

@marek-safar So far the PNSE failure test suites being completely skipped are System.Console.Tests.csproj, System.IO.FileSystem.Tests.csproj, System.Security.Cryptography.OpenSsl.Tests.csproj, System.Threading.Thread.Tests.csproj. There is some more consolidation and checking to be done, especially since tests cannot run to completion related to https://github.com/dotnet/core-eng/issues/12969

There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE.

marek-safar · 2021-04-30T16:53:24Z

Out of these 4 I think only System.Security.Cryptography.OpenSsl.Tests.csproj should be skipped completely.

tqiu8 · 2021-04-30T21:10:04Z

On my local machine System.Threading.Thread is passing.

mdh1418 · 2021-05-03T14:34:21Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

mdh1418 · 2021-05-03T14:36:56Z

@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?


System.Console.Tests.dll - 46
System.Diagnostics.Process.Tests.dll - 12
System.IO.FileSystem.Tests.dll - 3 System.Net.NetworkInformation.Functional.Tests.dll - 1 System.Net.Ping.Functional.Tests.dll - 25 System.Security.Cryptography.OpenSsl.Tests.dll - 726
System.Security.Cryptography.X509Certificates.Tests.dll - 163
System.Security.Cryptography.Xml.Tests.dll - 3 System.Runtime.Extensions.Tests.dll - 1
System.Threading.Thread.Tests.dll - 1
System.Xml.RW.XmlReader.Tests.dll - 3

all_ios_tvos_PNSE_us.txt

tqiu8 · 2021-05-03T14:39:49Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

Yep, that's probably what happened.

marek-safar · 2021-05-03T15:36:20Z

Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?

Preferably not

…m.Net.NameResolution.Unit test suite skips

… fix typo

This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

…Issue

…2614

…ng on CI but passes locally

…on CI but passes locally

…sing

…Algorithms test exclusions

…tiveIssues

akoeplinger · 2021-05-12T15:36:36Z

Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)

mdh1418 · 2021-04-19T13:58:25Z

In effort to assess the state of the iOS library tests, a number of tests will be skipped

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with iOS had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException. [SkipOnPlatform] attributes are attached to the failing test methods.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436

The ActiveIssues and tests.proj exclusions have been tested on #49917 as well.

This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj - Removed $(NetCoreAppCurrent)-iOS from target frameworks as it is stated to currently only supported on Windows.

src/libraries/tests.proj - Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.

The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]

ghost · 2021-04-19T13:58:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

marek-safar · 2021-04-30T13:33:41Z

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

Which are they?

mdh1418 · 2021-04-30T14:06:51Z

@marek-safar So far the PNSE failure test suites being completely skipped are System.Console.Tests.csproj, System.IO.FileSystem.Tests.csproj, System.Security.Cryptography.OpenSsl.Tests.csproj, System.Threading.Thread.Tests.csproj. There is some more consolidation and checking to be done, especially since tests cannot run to completion related to https://github.com/dotnet/core-eng/issues/12969

There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE.

marek-safar · 2021-04-30T16:53:24Z

Out of these 4 I think only System.Security.Cryptography.OpenSsl.Tests.csproj should be skipped completely.

tqiu8 · 2021-04-30T21:10:04Z

On my local machine System.Threading.Thread is passing.

mdh1418 · 2021-05-03T14:34:21Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

mdh1418 · 2021-05-03T14:36:56Z

@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?


System.Console.Tests.dll - 46
System.Diagnostics.Process.Tests.dll - 12
System.IO.FileSystem.Tests.dll - 3 System.Net.NetworkInformation.Functional.Tests.dll - 1 System.Net.Ping.Functional.Tests.dll - 25 System.Security.Cryptography.OpenSsl.Tests.dll - 726
System.Security.Cryptography.X509Certificates.Tests.dll - 163
System.Security.Cryptography.Xml.Tests.dll - 3 System.Runtime.Extensions.Tests.dll - 1
System.Threading.Thread.Tests.dll - 1
System.Xml.RW.XmlReader.Tests.dll - 3

all_ios_tvos_PNSE_us.txt

tqiu8 · 2021-05-03T14:39:49Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

Yep, that's probably what happened.

marek-safar · 2021-05-03T15:36:20Z

Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?

Preferably not

…m.Net.NameResolution.Unit test suite skips

… fix typo

This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

…Issue

…2614

…ng on CI but passes locally

…on CI but passes locally

…sing

…Algorithms test exclusions

…tiveIssues

akoeplinger · 2021-05-12T15:36:36Z

Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)

mdh1418 · 2021-04-19T13:58:25Z

In effort to assess the state of the iOS library tests, a number of tests will be skipped

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with iOS had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException. [SkipOnPlatform] attributes are attached to the failing test methods.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436

The ActiveIssues and tests.proj exclusions have been tested on #49917 as well.

This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj - Removed $(NetCoreAppCurrent)-iOS from target frameworks as it is stated to currently only supported on Windows.

src/libraries/tests.proj - Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.

The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]

ghost · 2021-04-19T13:58:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

marek-safar · 2021-04-30T13:33:41Z

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

Which are they?

mdh1418 · 2021-04-30T14:06:51Z

@marek-safar So far the PNSE failure test suites being completely skipped are System.Console.Tests.csproj, System.IO.FileSystem.Tests.csproj, System.Security.Cryptography.OpenSsl.Tests.csproj, System.Threading.Thread.Tests.csproj. There is some more consolidation and checking to be done, especially since tests cannot run to completion related to https://github.com/dotnet/core-eng/issues/12969

There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE.

marek-safar · 2021-04-30T16:53:24Z

Out of these 4 I think only System.Security.Cryptography.OpenSsl.Tests.csproj should be skipped completely.

tqiu8 · 2021-04-30T21:10:04Z

On my local machine System.Threading.Thread is passing.

mdh1418 · 2021-05-03T14:34:21Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

mdh1418 · 2021-05-03T14:36:56Z

@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?


System.Console.Tests.dll - 46
System.Diagnostics.Process.Tests.dll - 12
System.IO.FileSystem.Tests.dll - 3 System.Net.NetworkInformation.Functional.Tests.dll - 1 System.Net.Ping.Functional.Tests.dll - 25 System.Security.Cryptography.OpenSsl.Tests.dll - 726
System.Security.Cryptography.X509Certificates.Tests.dll - 163
System.Security.Cryptography.Xml.Tests.dll - 3 System.Runtime.Extensions.Tests.dll - 1
System.Threading.Thread.Tests.dll - 1
System.Xml.RW.XmlReader.Tests.dll - 3

all_ios_tvos_PNSE_us.txt

tqiu8 · 2021-05-03T14:39:49Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

Yep, that's probably what happened.

marek-safar · 2021-05-03T15:36:20Z

Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?

Preferably not

…m.Net.NameResolution.Unit test suite skips

… fix typo

This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

…Issue

…2614

…ng on CI but passes locally

…on CI but passes locally

…sing

…Algorithms test exclusions

…tiveIssues

akoeplinger · 2021-05-12T15:36:36Z

Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)

mdh1418 · 2021-04-19T13:58:25Z

In effort to assess the state of the iOS library tests, a number of tests will be skipped

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with iOS had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException. [SkipOnPlatform] attributes are attached to the failing test methods.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436

The ActiveIssues and tests.proj exclusions have been tested on #49917 as well.

This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj - Removed $(NetCoreAppCurrent)-iOS from target frameworks as it is stated to currently only supported on Windows.

src/libraries/tests.proj - Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.

The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]

ghost · 2021-04-19T13:58:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

marek-safar · 2021-04-30T13:33:41Z

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

Which are they?

mdh1418 · 2021-04-30T14:06:51Z

@marek-safar So far the PNSE failure test suites being completely skipped are System.Console.Tests.csproj, System.IO.FileSystem.Tests.csproj, System.Security.Cryptography.OpenSsl.Tests.csproj, System.Threading.Thread.Tests.csproj. There is some more consolidation and checking to be done, especially since tests cannot run to completion related to https://github.com/dotnet/core-eng/issues/12969

There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE.

marek-safar · 2021-04-30T16:53:24Z

Out of these 4 I think only System.Security.Cryptography.OpenSsl.Tests.csproj should be skipped completely.

tqiu8 · 2021-04-30T21:10:04Z

On my local machine System.Threading.Thread is passing.

mdh1418 · 2021-05-03T14:34:21Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

mdh1418 · 2021-05-03T14:36:56Z

@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?


System.Console.Tests.dll - 46
System.Diagnostics.Process.Tests.dll - 12
System.IO.FileSystem.Tests.dll - 3 System.Net.NetworkInformation.Functional.Tests.dll - 1 System.Net.Ping.Functional.Tests.dll - 25 System.Security.Cryptography.OpenSsl.Tests.dll - 726
System.Security.Cryptography.X509Certificates.Tests.dll - 163
System.Security.Cryptography.Xml.Tests.dll - 3 System.Runtime.Extensions.Tests.dll - 1
System.Threading.Thread.Tests.dll - 1
System.Xml.RW.XmlReader.Tests.dll - 3

all_ios_tvos_PNSE_us.txt

tqiu8 · 2021-05-03T14:39:49Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

Yep, that's probably what happened.

marek-safar · 2021-05-03T15:36:20Z

Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?

Preferably not

…m.Net.NameResolution.Unit test suite skips

… fix typo

This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

…Issue

…2614

…ng on CI but passes locally

…on CI but passes locally

…sing

…Algorithms test exclusions

…tiveIssues

akoeplinger · 2021-05-12T15:36:36Z

Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)

mdh1418 · 2021-04-19T13:58:25Z

In effort to assess the state of the iOS library tests, a number of tests will be skipped

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with iOS had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException. [SkipOnPlatform] attributes are attached to the failing test methods.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436

The ActiveIssues and tests.proj exclusions have been tested on #49917 as well.

This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj - Removed $(NetCoreAppCurrent)-iOS from target frameworks as it is stated to currently only supported on Windows.

src/libraries/tests.proj - Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.

The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]

ghost · 2021-04-19T13:58:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

marek-safar · 2021-04-30T13:33:41Z

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

Which are they?

mdh1418 · 2021-04-30T14:06:51Z

@marek-safar So far the PNSE failure test suites being completely skipped are System.Console.Tests.csproj, System.IO.FileSystem.Tests.csproj, System.Security.Cryptography.OpenSsl.Tests.csproj, System.Threading.Thread.Tests.csproj. There is some more consolidation and checking to be done, especially since tests cannot run to completion related to https://github.com/dotnet/core-eng/issues/12969

There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE.

marek-safar · 2021-04-30T16:53:24Z

Out of these 4 I think only System.Security.Cryptography.OpenSsl.Tests.csproj should be skipped completely.

tqiu8 · 2021-04-30T21:10:04Z

On my local machine System.Threading.Thread is passing.

mdh1418 · 2021-05-03T14:34:21Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

mdh1418 · 2021-05-03T14:36:56Z

@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?


System.Console.Tests.dll - 46
System.Diagnostics.Process.Tests.dll - 12
System.IO.FileSystem.Tests.dll - 3 System.Net.NetworkInformation.Functional.Tests.dll - 1 System.Net.Ping.Functional.Tests.dll - 25 System.Security.Cryptography.OpenSsl.Tests.dll - 726
System.Security.Cryptography.X509Certificates.Tests.dll - 163
System.Security.Cryptography.Xml.Tests.dll - 3 System.Runtime.Extensions.Tests.dll - 1
System.Threading.Thread.Tests.dll - 1
System.Xml.RW.XmlReader.Tests.dll - 3

all_ios_tvos_PNSE_us.txt

tqiu8 · 2021-05-03T14:39:49Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

Yep, that's probably what happened.

marek-safar · 2021-05-03T15:36:20Z

Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?

Preferably not

…m.Net.NameResolution.Unit test suite skips

… fix typo

This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

…Issue

…2614

…ng on CI but passes locally

…on CI but passes locally

…sing

…Algorithms test exclusions

…tiveIssues

akoeplinger · 2021-05-12T15:36:36Z

Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)

mdh1418 · 2021-04-19T13:58:25Z

In effort to assess the state of the iOS library tests, a number of tests will be skipped

Test suites that crash, hang, or have a significant number of failures (mostly 20+ failing unique test methods) are skipped on at the tests.proj level. Otherwise, [ActiveIssue] attributes have been attached to either the individual test methods that fail or the entire class if the whole test class fails. Any pre-existing ActiveIssue attribute associated with iOS had been reassessed to either remain or be closed.

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException. [SkipOnPlatform] attributes are attached to the failing test methods.

All of the Issues can be tracked at https://github.com/dotnet/runtime/projects/48#column-9236436

The ActiveIssues and tests.proj exclusions have been tested on #49917 as well.

This PR makes the following changes in the following files:
src/libraries/System.IO.Ports/tests/System.IO.Ports.Tests.csproj - Removed $(NetCoreAppCurrent)-iOS from target frameworks as it is stated to currently only supported on Windows.

src/libraries/tests.proj - Update the test suites that need to be skipped at the project level, reducing the number and categorizing as a skip due to PNSE, a skip due to numerous failures, and a skip due to the suite crashing.

The remaining files changed are either:
[ActiveIssue("<issue-url>", TestPlatforms.iOS | TestPlatforms.tvOS | TestPlatforms.MacCatalyst)]
[SkipOnPlatform(TestPlatforms.iOS | TestPlatforms.tvOS, "Not supported on iOS or tvOS.")]

ghost · 2021-04-19T13:58:30Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

marek-safar · 2021-04-30T13:33:41Z

A few test suites have been skipped because they fail with some form of PlatformNotSupportedException.

Which are they?

mdh1418 · 2021-04-30T14:06:51Z

@marek-safar So far the PNSE failure test suites being completely skipped are System.Console.Tests.csproj, System.IO.FileSystem.Tests.csproj, System.Security.Cryptography.OpenSsl.Tests.csproj, System.Threading.Thread.Tests.csproj. There is some more consolidation and checking to be done, especially since tests cannot run to completion related to https://github.com/dotnet/core-eng/issues/12969

There are some test facts that fail with PNSE that are skipped individually, I'll need to ensure that they're being tracked/marked with some note of PNSE.

marek-safar · 2021-04-30T16:53:24Z

Out of these 4 I think only System.Security.Cryptography.OpenSsl.Tests.csproj should be skipped completely.

tqiu8 · 2021-04-30T21:10:04Z

On my local machine System.Threading.Thread is passing.

mdh1418 · 2021-05-03T14:34:21Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

mdh1418 · 2021-05-03T14:36:56Z

@marek-safar Of all the previous CI runs of iOS and tvOS, the following suites have the following number of methods that fail with PNSE. Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?


System.Console.Tests.dll - 46
System.Diagnostics.Process.Tests.dll - 12
System.IO.FileSystem.Tests.dll - 3 System.Net.NetworkInformation.Functional.Tests.dll - 1 System.Net.Ping.Functional.Tests.dll - 25 System.Security.Cryptography.OpenSsl.Tests.dll - 726
System.Security.Cryptography.X509Certificates.Tests.dll - 163
System.Security.Cryptography.Xml.Tests.dll - 3 System.Runtime.Extensions.Tests.dll - 1
System.Threading.Thread.Tests.dll - 1
System.Xml.RW.XmlReader.Tests.dll - 3

all_ios_tvos_PNSE_us.txt

tqiu8 · 2021-05-03T14:39:49Z

@tqiu8 when you ran System.Threading.Thread.Tests, was that including this commit? ca51de9 (Which skips the test facts)

Yep, that's probably what happened.

marek-safar · 2021-05-03T15:36:20Z

Should System.Security.Cryptography.X509Certificates.Tests also be skipped completely?

Preferably not

…m.Net.NameResolution.Unit test suite skips

… fix typo

This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

…Issue

…2614

…ng on CI but passes locally

…on CI but passes locally

…sing

…Algorithms test exclusions

…tiveIssues

akoeplinger · 2021-05-12T15:36:36Z

Yup sounds good, but we'll still go ahead and merge this PR for now to get to a green state in runtime-staging :)

mdh1418 changed the title ~~[iOS][libraries] Skip failing Android tests with ActiveIssues and proj level skips~~[iOS][libraries] Skip failing iOS tests with ActiveIssues and proj level skipsApr 19, 2021

mdh1418 requested a review from tqiu8 April 19, 2021 17:44

mdh1418 force-pushed the skip_ios_library_tests branch 6 times, most recently from 521b003 to bfa8f8aCompare April 29, 2021 21:13

mdh1418 force-pushed the skip_ios_library_tests branch from c144b51 to 05a9946Compare April 30, 2021 19:15

mdh1418 requested review from BrzVlad, CoffeeFlux, SamMonoRT, eiriktsarpalis, imhameed, jozkee, lambdageek, layomia, steveharter, thaystg and vargaz as code owners May 3, 2021 18:36

mdh1418 force-pushed the skip_ios_library_tests branch from 2387efb to 0da7e61Compare May 3, 2021 18:38

Mitchell Hwang added 23 commits May 12, 2021 09:22

[iOS][tvOS] Update test suites skipped due to PNSE
580308d

[iOS][tvOS] Remove Microsoft.Extensions.HostFactoryResolver and Syste…
e7a4d5d
…m.Net.NameResolution.Unit test suite skips

[iOS][tvOS] Reenable System.Diagnostics.Process.Tests ActiveIssue and…
5f65e18
… fix typo

[tvOS] Skip tvOS.Simulator.Aot.Test due to linking issue
5c48370

[iOS][tvOS] Skip ipv6 System.Net.Sockets.Tests PNSE
3f044b9

[iOS][tvOS] Update System.Threading.Thread.Tests PNSE SkipOnPlatform
9a3155a
This reverts commit 163ff12da16e602e52830c79f267e14c420284c0.

[iOS][tvOS] Add System.Runtime.Extensions.Tests UserName_Valid Active…
662e4cd
…Issue

[iOS][tvOS][Catalyst] Update System.IO.FileSystem SkipOnPlatform
093db84

[iOS][tvOS] Remove System.Xml.XmlSchemaSet.Tests test proj skip
80fcf69

[iOS][tvOS] Remove System.IO.Compression.Tests test proj skip
acac640

[MacCatalyst] Amend all iOS tvOS ActiveIssues to include MacCatalyst
efe09f4

[iOS][tvOS] Remove System.Xml.RW.XmlReader.Tests ActiveIssue 49187
b0b1a8c

[libraries] Undo incorrect SkipOnPlatform change
e5d9708

Temporarily skip iOS Simulator AOT functional test
287b6aa

[iOS][tvOS][libraries] Add System.Threading.Tasks.Tests ActiveIssue 5…
755f1f9
…2614

[iOS][tvOS] Skip suites crashing on CI that pass locally
0dbecd9

[iOS][tvOS] Add System.IO.Compression.ZipFile.Tests ActiveIssue faili…
ccd9398
…ng on CI but passes locally

[iOS][tvOS] Add System.Net.Ping.Functional.Tests ActiveIssue failing …
e28bbde
…on CI but passes locally

Fix typo
2aa52ed

[iOS][tvOS][macCatalyst] Temporary test suite skips to get checks pas…
f01f3cf
…sing

[iOS][tvOS] Add System.IO.Packaging and System.Security.Cryptography.…
31ac47c
…Algorithms test exclusions

Fix typo in SkipOnPlatform in System.Runtime.Extensions
488a6f7

[iOS][tvOS] Add GitHub issue to System.Linq.Expressions.Tests tvOS Ac…
fe031a4
…tiveIssues

akoeplinger approved these changes May 12, 2021
View reviewed changes

akoeplinger merged commit 5297337 into dotnet:mainMay 12, 2021

mdh1418 deleted the skip_ios_library_tests branch May 12, 2021 15:41

This was referenced May 17, 2021
[MacCatalyst][libraries] Add specific SkipOnPlatform and ActiveIssues instead of test level skip #52859
Merged
Skip flakey mobile library test crashes #52931
Merged

karelz added this to the 6.0.0 milestone May 20, 2021

ghost locked as resolved and limited conversation to collaborators Jun 19, 2021