Delete default port tests

[jkotalik]

Fixeshttps://github.com/aspnet/AspNetCore-Internal/issues/2854

These tests are never reliable and fail frequently. We can't rely on binding to a static port, even with the extra double-check we do before running the test. I'd rather just delete them or make them manually ran tests. These scenarios are covered by CTI too.

[Tratcher]

@anurse you had one alternate proposal to containerize these?

[halter73]

My only question is if we should come up with something to replace these tests before removing them. Maybe CTI testing is enough, but part of the reason we haven't removed these yet is because doing so would likely decrease our motivation to add less/non-flaky replacement tests.

[jkotalik]

I think the cost-benefit of keeping/replacing this test is fairly high. Let's say we want to spin up containers to verify default port selection. Then we would need to set up infrastructure to run those either manually or automatically. My guess is it would take a week to get that work done. And the benefit is low: CTI covers default port selection well. We also still test IPV4 and IPV6 selection.

[Tratcher]

Perhaps, but such infrastructure could be used for a lot more than just these tests.

[analogrelay]

Here's a spicy question: Have these tests ever caught a regression that we wouldn't expect CTI to catch?

[analogrelay]

Containerizing is a solution but has a non-zero cost (from past experience, docker-related tests can themselves be flaky, plus they wouldn't work on macOS). I don't know that I see a big enough benefit to make it worth the cost of having these tests (assuming we have adequate CTI coverage).

[Tratcher]

Can we configure them to skip on the CI but to run locally? E.g. [SkipOnCI]. These tests are primarily useful during development to check for refactor regressions.

[jkotalik]

Alright between:

• Delete these tests
• Keeping tests and always skipping them on CI (which requires a new attribute)

What do people prefer?

[halter73]

I prefer just skipping on the CI, but I don't feel too strongly either way.

[Tratcher]

Agreed, skipping on the CI gets rid of the flakyness and still lets us have some refactor coverage.

[jkotalik]

I disagree but 2 against 1, I won't fight it.

[jkotalik]

I'm going to make a change to extensions to do a few things:

• Move the SkipOnHelix attribute to Extensions
• Create a new attribute SkipOnCI
• Remove a bunch of copied contents in all tests. Today, all test projects have like 10 extra files that only a few tests need:
  
  This includes the SkipOnHelixAttribute.

[jkotalik]

Waiting on dotnet/extensions#2186.

[jkotalik]

Can someone hit me with approval for this?

[jkotalik]

Fixeshttps://github.com/aspnet/AspNetCore-Internal/issues/2854

These tests are never reliable and fail frequently. We can't rely on binding to a static port, even with the extra double-check we do before running the test. I'd rather just delete them or make them manually ran tests. These scenarios are covered by CTI too.

[Tratcher]

@anurse you had one alternate proposal to containerize these?

[halter73]

My only question is if we should come up with something to replace these tests before removing them. Maybe CTI testing is enough, but part of the reason we haven't removed these yet is because doing so would likely decrease our motivation to add less/non-flaky replacement tests.

[jkotalik]

I think the cost-benefit of keeping/replacing this test is fairly high. Let's say we want to spin up containers to verify default port selection. Then we would need to set up infrastructure to run those either manually or automatically. My guess is it would take a week to get that work done. And the benefit is low: CTI covers default port selection well. We also still test IPV4 and IPV6 selection.

[Tratcher]

Perhaps, but such infrastructure could be used for a lot more than just these tests.

[analogrelay]

Here's a spicy question: Have these tests ever caught a regression that we wouldn't expect CTI to catch?

[analogrelay]

Containerizing is a solution but has a non-zero cost (from past experience, docker-related tests can themselves be flaky, plus they wouldn't work on macOS). I don't know that I see a big enough benefit to make it worth the cost of having these tests (assuming we have adequate CTI coverage).

[Tratcher]

Can we configure them to skip on the CI but to run locally? E.g. [SkipOnCI]. These tests are primarily useful during development to check for refactor regressions.

[jkotalik]

Alright between:

• Delete these tests
• Keeping tests and always skipping them on CI (which requires a new attribute)

What do people prefer?

[halter73]

I prefer just skipping on the CI, but I don't feel too strongly either way.

[Tratcher]

Agreed, skipping on the CI gets rid of the flakyness and still lets us have some refactor coverage.

[jkotalik]

I disagree but 2 against 1, I won't fight it.

[jkotalik]

I'm going to make a change to extensions to do a few things:

• Move the SkipOnHelix attribute to Extensions
• Create a new attribute SkipOnCI
• Remove a bunch of copied contents in all tests. Today, all test projects have like 10 extra files that only a few tests need:
  
  This includes the SkipOnHelixAttribute.

[jkotalik]

Waiting on dotnet/extensions#2186.

[jkotalik]

Can someone hit me with approval for this?

[jkotalik]

Fixeshttps://github.com/aspnet/AspNetCore-Internal/issues/2854

These tests are never reliable and fail frequently. We can't rely on binding to a static port, even with the extra double-check we do before running the test. I'd rather just delete them or make them manually ran tests. These scenarios are covered by CTI too.

[Tratcher]

@anurse you had one alternate proposal to containerize these?

[halter73]

My only question is if we should come up with something to replace these tests before removing them. Maybe CTI testing is enough, but part of the reason we haven't removed these yet is because doing so would likely decrease our motivation to add less/non-flaky replacement tests.

[jkotalik]

I think the cost-benefit of keeping/replacing this test is fairly high. Let's say we want to spin up containers to verify default port selection. Then we would need to set up infrastructure to run those either manually or automatically. My guess is it would take a week to get that work done. And the benefit is low: CTI covers default port selection well. We also still test IPV4 and IPV6 selection.

[Tratcher]

Perhaps, but such infrastructure could be used for a lot more than just these tests.

[analogrelay]

Here's a spicy question: Have these tests ever caught a regression that we wouldn't expect CTI to catch?

[analogrelay]

Containerizing is a solution but has a non-zero cost (from past experience, docker-related tests can themselves be flaky, plus they wouldn't work on macOS). I don't know that I see a big enough benefit to make it worth the cost of having these tests (assuming we have adequate CTI coverage).

[Tratcher]

Can we configure them to skip on the CI but to run locally? E.g. [SkipOnCI]. These tests are primarily useful during development to check for refactor regressions.

[jkotalik]

Alright between:

• Delete these tests
• Keeping tests and always skipping them on CI (which requires a new attribute)

What do people prefer?

[halter73]

I prefer just skipping on the CI, but I don't feel too strongly either way.

[Tratcher]

Agreed, skipping on the CI gets rid of the flakyness and still lets us have some refactor coverage.

[jkotalik]

I disagree but 2 against 1, I won't fight it.

[jkotalik]

I'm going to make a change to extensions to do a few things:

• Move the SkipOnHelix attribute to Extensions
• Create a new attribute SkipOnCI
• Remove a bunch of copied contents in all tests. Today, all test projects have like 10 extra files that only a few tests need:
  
  This includes the SkipOnHelixAttribute.

[jkotalik]

Waiting on dotnet/extensions#2186.

[jkotalik]

Can someone hit me with approval for this?

[jkotalik]

Fixeshttps://github.com/aspnet/AspNetCore-Internal/issues/2854

These tests are never reliable and fail frequently. We can't rely on binding to a static port, even with the extra double-check we do before running the test. I'd rather just delete them or make them manually ran tests. These scenarios are covered by CTI too.

[Tratcher]

@anurse you had one alternate proposal to containerize these?

[halter73]

My only question is if we should come up with something to replace these tests before removing them. Maybe CTI testing is enough, but part of the reason we haven't removed these yet is because doing so would likely decrease our motivation to add less/non-flaky replacement tests.

[jkotalik]

I think the cost-benefit of keeping/replacing this test is fairly high. Let's say we want to spin up containers to verify default port selection. Then we would need to set up infrastructure to run those either manually or automatically. My guess is it would take a week to get that work done. And the benefit is low: CTI covers default port selection well. We also still test IPV4 and IPV6 selection.

[Tratcher]

Perhaps, but such infrastructure could be used for a lot more than just these tests.

[analogrelay]

Here's a spicy question: Have these tests ever caught a regression that we wouldn't expect CTI to catch?

[analogrelay]

Containerizing is a solution but has a non-zero cost (from past experience, docker-related tests can themselves be flaky, plus they wouldn't work on macOS). I don't know that I see a big enough benefit to make it worth the cost of having these tests (assuming we have adequate CTI coverage).

[Tratcher]

Can we configure them to skip on the CI but to run locally? E.g. [SkipOnCI]. These tests are primarily useful during development to check for refactor regressions.

[jkotalik]

Alright between:

• Delete these tests
• Keeping tests and always skipping them on CI (which requires a new attribute)

What do people prefer?

[halter73]

I prefer just skipping on the CI, but I don't feel too strongly either way.

[Tratcher]

Agreed, skipping on the CI gets rid of the flakyness and still lets us have some refactor coverage.

[jkotalik]

I disagree but 2 against 1, I won't fight it.

[jkotalik]

I'm going to make a change to extensions to do a few things:

• Move the SkipOnHelix attribute to Extensions
• Create a new attribute SkipOnCI
• Remove a bunch of copied contents in all tests. Today, all test projects have like 10 extra files that only a few tests need:
  
  This includes the SkipOnHelixAttribute.

[jkotalik]

Waiting on dotnet/extensions#2186.

[jkotalik]

Can someone hit me with approval for this?

[jkotalik]

Fixeshttps://github.com/aspnet/AspNetCore-Internal/issues/2854

These tests are never reliable and fail frequently. We can't rely on binding to a static port, even with the extra double-check we do before running the test. I'd rather just delete them or make them manually ran tests. These scenarios are covered by CTI too.

[Tratcher]

@anurse you had one alternate proposal to containerize these?

[halter73]

My only question is if we should come up with something to replace these tests before removing them. Maybe CTI testing is enough, but part of the reason we haven't removed these yet is because doing so would likely decrease our motivation to add less/non-flaky replacement tests.

[jkotalik]

I think the cost-benefit of keeping/replacing this test is fairly high. Let's say we want to spin up containers to verify default port selection. Then we would need to set up infrastructure to run those either manually or automatically. My guess is it would take a week to get that work done. And the benefit is low: CTI covers default port selection well. We also still test IPV4 and IPV6 selection.

[Tratcher]

Perhaps, but such infrastructure could be used for a lot more than just these tests.

[analogrelay]

Here's a spicy question: Have these tests ever caught a regression that we wouldn't expect CTI to catch?

[analogrelay]

Containerizing is a solution but has a non-zero cost (from past experience, docker-related tests can themselves be flaky, plus they wouldn't work on macOS). I don't know that I see a big enough benefit to make it worth the cost of having these tests (assuming we have adequate CTI coverage).

[Tratcher]

Can we configure them to skip on the CI but to run locally? E.g. [SkipOnCI]. These tests are primarily useful during development to check for refactor regressions.

[jkotalik]

Alright between:

• Delete these tests
• Keeping tests and always skipping them on CI (which requires a new attribute)

What do people prefer?

[halter73]

I prefer just skipping on the CI, but I don't feel too strongly either way.

[Tratcher]

Agreed, skipping on the CI gets rid of the flakyness and still lets us have some refactor coverage.

[jkotalik]

I disagree but 2 against 1, I won't fight it.

[jkotalik]

I'm going to make a change to extensions to do a few things:

• Move the SkipOnHelix attribute to Extensions
• Create a new attribute SkipOnCI
• Remove a bunch of copied contents in all tests. Today, all test projects have like 10 extra files that only a few tests need:
  
  This includes the SkipOnHelixAttribute.

[jkotalik]

Waiting on dotnet/extensions#2186.

[jkotalik]

Can someone hit me with approval for this?

[jkotalik]

Fixeshttps://github.com/aspnet/AspNetCore-Internal/issues/2854

These tests are never reliable and fail frequently. We can't rely on binding to a static port, even with the extra double-check we do before running the test. I'd rather just delete them or make them manually ran tests. These scenarios are covered by CTI too.

[Tratcher]

@anurse you had one alternate proposal to containerize these?

[halter73]

My only question is if we should come up with something to replace these tests before removing them. Maybe CTI testing is enough, but part of the reason we haven't removed these yet is because doing so would likely decrease our motivation to add less/non-flaky replacement tests.

[jkotalik]

I think the cost-benefit of keeping/replacing this test is fairly high. Let's say we want to spin up containers to verify default port selection. Then we would need to set up infrastructure to run those either manually or automatically. My guess is it would take a week to get that work done. And the benefit is low: CTI covers default port selection well. We also still test IPV4 and IPV6 selection.

[Tratcher]

Perhaps, but such infrastructure could be used for a lot more than just these tests.

[analogrelay]

Here's a spicy question: Have these tests ever caught a regression that we wouldn't expect CTI to catch?

[analogrelay]

Containerizing is a solution but has a non-zero cost (from past experience, docker-related tests can themselves be flaky, plus they wouldn't work on macOS). I don't know that I see a big enough benefit to make it worth the cost of having these tests (assuming we have adequate CTI coverage).

[Tratcher]

Can we configure them to skip on the CI but to run locally? E.g. [SkipOnCI]. These tests are primarily useful during development to check for refactor regressions.

[jkotalik]

Alright between:

• Delete these tests
• Keeping tests and always skipping them on CI (which requires a new attribute)

What do people prefer?

[halter73]

I prefer just skipping on the CI, but I don't feel too strongly either way.

[Tratcher]

Agreed, skipping on the CI gets rid of the flakyness and still lets us have some refactor coverage.

[jkotalik]

I disagree but 2 against 1, I won't fight it.

[jkotalik]

I'm going to make a change to extensions to do a few things:

• Move the SkipOnHelix attribute to Extensions
• Create a new attribute SkipOnCI
• Remove a bunch of copied contents in all tests. Today, all test projects have like 10 extra files that only a few tests need:
  
  This includes the SkipOnHelixAttribute.

[jkotalik]

Waiting on dotnet/extensions#2186.

[jkotalik]

Can someone hit me with approval for this?

[jkotalik]

Fixeshttps://github.com/aspnet/AspNetCore-Internal/issues/2854

These tests are never reliable and fail frequently. We can't rely on binding to a static port, even with the extra double-check we do before running the test. I'd rather just delete them or make them manually ran tests. These scenarios are covered by CTI too.

[Tratcher]

@anurse you had one alternate proposal to containerize these?

[halter73]

My only question is if we should come up with something to replace these tests before removing them. Maybe CTI testing is enough, but part of the reason we haven't removed these yet is because doing so would likely decrease our motivation to add less/non-flaky replacement tests.

[jkotalik]

I think the cost-benefit of keeping/replacing this test is fairly high. Let's say we want to spin up containers to verify default port selection. Then we would need to set up infrastructure to run those either manually or automatically. My guess is it would take a week to get that work done. And the benefit is low: CTI covers default port selection well. We also still test IPV4 and IPV6 selection.

[Tratcher]

Perhaps, but such infrastructure could be used for a lot more than just these tests.

[analogrelay]

Here's a spicy question: Have these tests ever caught a regression that we wouldn't expect CTI to catch?

[analogrelay]

Containerizing is a solution but has a non-zero cost (from past experience, docker-related tests can themselves be flaky, plus they wouldn't work on macOS). I don't know that I see a big enough benefit to make it worth the cost of having these tests (assuming we have adequate CTI coverage).

[Tratcher]

Can we configure them to skip on the CI but to run locally? E.g. [SkipOnCI]. These tests are primarily useful during development to check for refactor regressions.

[jkotalik]

Alright between:

• Delete these tests
• Keeping tests and always skipping them on CI (which requires a new attribute)

What do people prefer?

[halter73]

I prefer just skipping on the CI, but I don't feel too strongly either way.

[Tratcher]

Agreed, skipping on the CI gets rid of the flakyness and still lets us have some refactor coverage.

[jkotalik]

I disagree but 2 against 1, I won't fight it.

[jkotalik]

I'm going to make a change to extensions to do a few things:

• Move the SkipOnHelix attribute to Extensions
• Create a new attribute SkipOnCI
• Remove a bunch of copied contents in all tests. Today, all test projects have like 10 extra files that only a few tests need:
  
  This includes the SkipOnHelixAttribute.

[jkotalik]

Waiting on dotnet/extensions#2186.

[jkotalik]

Can someone hit me with approval for this?

[jkotalik]

Fixeshttps://github.com/aspnet/AspNetCore-Internal/issues/2854

These tests are never reliable and fail frequently. We can't rely on binding to a static port, even with the extra double-check we do before running the test. I'd rather just delete them or make them manually ran tests. These scenarios are covered by CTI too.

[Tratcher]

@anurse you had one alternate proposal to containerize these?

[halter73]

My only question is if we should come up with something to replace these tests before removing them. Maybe CTI testing is enough, but part of the reason we haven't removed these yet is because doing so would likely decrease our motivation to add less/non-flaky replacement tests.

[jkotalik]

I think the cost-benefit of keeping/replacing this test is fairly high. Let's say we want to spin up containers to verify default port selection. Then we would need to set up infrastructure to run those either manually or automatically. My guess is it would take a week to get that work done. And the benefit is low: CTI covers default port selection well. We also still test IPV4 and IPV6 selection.

[Tratcher]

Perhaps, but such infrastructure could be used for a lot more than just these tests.

[analogrelay]

Here's a spicy question: Have these tests ever caught a regression that we wouldn't expect CTI to catch?

[analogrelay]

Containerizing is a solution but has a non-zero cost (from past experience, docker-related tests can themselves be flaky, plus they wouldn't work on macOS). I don't know that I see a big enough benefit to make it worth the cost of having these tests (assuming we have adequate CTI coverage).

[Tratcher]

Can we configure them to skip on the CI but to run locally? E.g. [SkipOnCI]. These tests are primarily useful during development to check for refactor regressions.

[jkotalik]

Alright between:

• Delete these tests
• Keeping tests and always skipping them on CI (which requires a new attribute)

What do people prefer?

[halter73]

I prefer just skipping on the CI, but I don't feel too strongly either way.

[Tratcher]

Agreed, skipping on the CI gets rid of the flakyness and still lets us have some refactor coverage.

[jkotalik]

I disagree but 2 against 1, I won't fight it.

[jkotalik]

I'm going to make a change to extensions to do a few things:

• Move the SkipOnHelix attribute to Extensions
• Create a new attribute SkipOnCI
• Remove a bunch of copied contents in all tests. Today, all test projects have like 10 extra files that only a few tests need:
  
  This includes the SkipOnHelixAttribute.

[jkotalik]

Waiting on dotnet/extensions#2186.

[jkotalik]

Can someone hit me with approval for this?