Switch remaining x64 queues to AL3

[richlander]

• Switch remaining x64 helix queues to Azure Linux 3.
• Reordered the long list of configs to be VM first, then container, all alpha ordered.
• We do not have any Arm64 Azure Linux 3 VMs and should change that.
• Not changing/adding any raw VM usage at this time.
• Need to run outerloop tests for this PR (the changes are in outerloop config)
• Related: Revert "Switch libraries testing to Ubuntu 22 temporarily" #115312

[Copilot]

Pull Request Overview

This PR updates the remaining x64 Helix queues to use Azure Linux 3 and reorders the configuration list.

• Replaces Ubuntu-based queue references with azurelinux.3 in multiple configurations.
• Updates image references for CentOS, Fedora, Ubuntu, and Debian queues accordingly.

Comments suppressed due to low confidence (1)

eng/pipelines/libraries/helix-queues-setup.yml:61

• [nitpick] There is an inconsistency in the casing between the original 'AzureLinux.3.0.Amd64.Open' and the updated 'azurelinux.3.Amd64.open'. Consider using consistent capitalization throughout for clarity.

- (AzureLinux.3.0.Amd64.Open)azurelinux.3.Amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

[richlander]

/azp run runtime-libraries-coreclr outerloop-linux

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[richlander]

Failures like the following seem like a machine configuration issue. There are many of them.

 System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileStreamLargeCount_Throws [FAIL]
System.IO.IOException : No file descriptors available :'/tmp/ropucomr.mjl_.ctor_39'
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>
System.Net.Sockets.Tests.SendPacketsAsync.Disposed_Throw [FAIL]
System.OutOfMemoryException : Exception of type'System.OutOfMemoryException' was thrown.
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>

@ilyas1974 -- can you run ulimit -a on the Ubuntu 22.04 and Azure Linux 3 VMs? We need to see what the difference is. I doubt the test code is at issue. Or how do I do this myself?

[ilyas1974]

Will see what I can do about getting you this information later today. Here are the instructions for self-service: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[ilyas1974]

Azure Linux 3

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31700
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31700
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Ubuntu 22.04

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31733
max locked memory (kbytes, -l) 1016148
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31733
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[richlander]

Thanks! Looks like -n is the problem. It is the difference I was expecting. We need to set that to the same value as Ubuntu. Does the Ubuntu VM just come that way or do we set it some way?

[ilyas1974]

This is the way the system comes to us. While we install software on systems - based on our customer's needs, we don't make any changes to the OS.

[akoeplinger]

@ilyas1974 ulimit for open files is explicitly changed here: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/definitions/shared/linux.yaml&version=GBmain&_a=contents&line=133&lineStyle=plain&lineEnd=136&lineStartColumn=1&lineEndColumn=1

[richlander]

Nice find. We need that same change for Azure Linux (or any other VM host we use).

Can that happen for the next roll-out? If not, we're going to need an off-cycle deployment.

[richlander]

• Switch remaining x64 helix queues to Azure Linux 3.
• Reordered the long list of configs to be VM first, then container, all alpha ordered.
• We do not have any Arm64 Azure Linux 3 VMs and should change that.
• Not changing/adding any raw VM usage at this time.
• Need to run outerloop tests for this PR (the changes are in outerloop config)
• Related: Revert "Switch libraries testing to Ubuntu 22 temporarily" #115312

[Copilot]

Pull Request Overview

This PR updates the remaining x64 Helix queues to use Azure Linux 3 and reorders the configuration list.

• Replaces Ubuntu-based queue references with azurelinux.3 in multiple configurations.
• Updates image references for CentOS, Fedora, Ubuntu, and Debian queues accordingly.

Comments suppressed due to low confidence (1)

eng/pipelines/libraries/helix-queues-setup.yml:61

• [nitpick] There is an inconsistency in the casing between the original 'AzureLinux.3.0.Amd64.Open' and the updated 'azurelinux.3.Amd64.open'. Consider using consistent capitalization throughout for clarity.

- (AzureLinux.3.0.Amd64.Open)azurelinux.3.Amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

[richlander]

/azp run runtime-libraries-coreclr outerloop-linux

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[richlander]

Failures like the following seem like a machine configuration issue. There are many of them.

 System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileStreamLargeCount_Throws [FAIL]
System.IO.IOException : No file descriptors available :'/tmp/ropucomr.mjl_.ctor_39'
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>
System.Net.Sockets.Tests.SendPacketsAsync.Disposed_Throw [FAIL]
System.OutOfMemoryException : Exception of type'System.OutOfMemoryException' was thrown.
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>

@ilyas1974 -- can you run ulimit -a on the Ubuntu 22.04 and Azure Linux 3 VMs? We need to see what the difference is. I doubt the test code is at issue. Or how do I do this myself?

[ilyas1974]

Will see what I can do about getting you this information later today. Here are the instructions for self-service: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[ilyas1974]

Azure Linux 3

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31700
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31700
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Ubuntu 22.04

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31733
max locked memory (kbytes, -l) 1016148
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31733
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[richlander]

Thanks! Looks like -n is the problem. It is the difference I was expecting. We need to set that to the same value as Ubuntu. Does the Ubuntu VM just come that way or do we set it some way?

[ilyas1974]

This is the way the system comes to us. While we install software on systems - based on our customer's needs, we don't make any changes to the OS.

[akoeplinger]

@ilyas1974 ulimit for open files is explicitly changed here: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/definitions/shared/linux.yaml&version=GBmain&_a=contents&line=133&lineStyle=plain&lineEnd=136&lineStartColumn=1&lineEndColumn=1

[richlander]

Nice find. We need that same change for Azure Linux (or any other VM host we use).

Can that happen for the next roll-out? If not, we're going to need an off-cycle deployment.

[richlander]

• Switch remaining x64 helix queues to Azure Linux 3.
• Reordered the long list of configs to be VM first, then container, all alpha ordered.
• We do not have any Arm64 Azure Linux 3 VMs and should change that.
• Not changing/adding any raw VM usage at this time.
• Need to run outerloop tests for this PR (the changes are in outerloop config)
• Related: Revert "Switch libraries testing to Ubuntu 22 temporarily" #115312

[Copilot]

Pull Request Overview

This PR updates the remaining x64 Helix queues to use Azure Linux 3 and reorders the configuration list.

• Replaces Ubuntu-based queue references with azurelinux.3 in multiple configurations.
• Updates image references for CentOS, Fedora, Ubuntu, and Debian queues accordingly.

Comments suppressed due to low confidence (1)

eng/pipelines/libraries/helix-queues-setup.yml:61

• [nitpick] There is an inconsistency in the casing between the original 'AzureLinux.3.0.Amd64.Open' and the updated 'azurelinux.3.Amd64.open'. Consider using consistent capitalization throughout for clarity.

- (AzureLinux.3.0.Amd64.Open)azurelinux.3.Amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

[richlander]

/azp run runtime-libraries-coreclr outerloop-linux

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[richlander]

Failures like the following seem like a machine configuration issue. There are many of them.

 System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileStreamLargeCount_Throws [FAIL]
System.IO.IOException : No file descriptors available :'/tmp/ropucomr.mjl_.ctor_39'
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>
System.Net.Sockets.Tests.SendPacketsAsync.Disposed_Throw [FAIL]
System.OutOfMemoryException : Exception of type'System.OutOfMemoryException' was thrown.
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>

@ilyas1974 -- can you run ulimit -a on the Ubuntu 22.04 and Azure Linux 3 VMs? We need to see what the difference is. I doubt the test code is at issue. Or how do I do this myself?

[ilyas1974]

Will see what I can do about getting you this information later today. Here are the instructions for self-service: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[ilyas1974]

Azure Linux 3

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31700
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31700
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Ubuntu 22.04

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31733
max locked memory (kbytes, -l) 1016148
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31733
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[richlander]

Thanks! Looks like -n is the problem. It is the difference I was expecting. We need to set that to the same value as Ubuntu. Does the Ubuntu VM just come that way or do we set it some way?

[ilyas1974]

This is the way the system comes to us. While we install software on systems - based on our customer's needs, we don't make any changes to the OS.

[akoeplinger]

@ilyas1974 ulimit for open files is explicitly changed here: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/definitions/shared/linux.yaml&version=GBmain&_a=contents&line=133&lineStyle=plain&lineEnd=136&lineStartColumn=1&lineEndColumn=1

[richlander]

Nice find. We need that same change for Azure Linux (or any other VM host we use).

Can that happen for the next roll-out? If not, we're going to need an off-cycle deployment.

[richlander]

• Switch remaining x64 helix queues to Azure Linux 3.
• Reordered the long list of configs to be VM first, then container, all alpha ordered.
• We do not have any Arm64 Azure Linux 3 VMs and should change that.
• Not changing/adding any raw VM usage at this time.
• Need to run outerloop tests for this PR (the changes are in outerloop config)
• Related: Revert "Switch libraries testing to Ubuntu 22 temporarily" #115312

[Copilot]

Pull Request Overview

This PR updates the remaining x64 Helix queues to use Azure Linux 3 and reorders the configuration list.

• Replaces Ubuntu-based queue references with azurelinux.3 in multiple configurations.
• Updates image references for CentOS, Fedora, Ubuntu, and Debian queues accordingly.

Comments suppressed due to low confidence (1)

eng/pipelines/libraries/helix-queues-setup.yml:61

• [nitpick] There is an inconsistency in the casing between the original 'AzureLinux.3.0.Amd64.Open' and the updated 'azurelinux.3.Amd64.open'. Consider using consistent capitalization throughout for clarity.

- (AzureLinux.3.0.Amd64.Open)azurelinux.3.Amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

[richlander]

/azp run runtime-libraries-coreclr outerloop-linux

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[richlander]

Failures like the following seem like a machine configuration issue. There are many of them.

 System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileStreamLargeCount_Throws [FAIL]
System.IO.IOException : No file descriptors available :'/tmp/ropucomr.mjl_.ctor_39'
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>
System.Net.Sockets.Tests.SendPacketsAsync.Disposed_Throw [FAIL]
System.OutOfMemoryException : Exception of type'System.OutOfMemoryException' was thrown.
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>

@ilyas1974 -- can you run ulimit -a on the Ubuntu 22.04 and Azure Linux 3 VMs? We need to see what the difference is. I doubt the test code is at issue. Or how do I do this myself?

[ilyas1974]

Will see what I can do about getting you this information later today. Here are the instructions for self-service: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[ilyas1974]

Azure Linux 3

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31700
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31700
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Ubuntu 22.04

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31733
max locked memory (kbytes, -l) 1016148
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31733
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[richlander]

Thanks! Looks like -n is the problem. It is the difference I was expecting. We need to set that to the same value as Ubuntu. Does the Ubuntu VM just come that way or do we set it some way?

[ilyas1974]

This is the way the system comes to us. While we install software on systems - based on our customer's needs, we don't make any changes to the OS.

[akoeplinger]

@ilyas1974 ulimit for open files is explicitly changed here: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/definitions/shared/linux.yaml&version=GBmain&_a=contents&line=133&lineStyle=plain&lineEnd=136&lineStartColumn=1&lineEndColumn=1

[richlander]

Nice find. We need that same change for Azure Linux (or any other VM host we use).

Can that happen for the next roll-out? If not, we're going to need an off-cycle deployment.

[richlander]

• Switch remaining x64 helix queues to Azure Linux 3.
• Reordered the long list of configs to be VM first, then container, all alpha ordered.
• We do not have any Arm64 Azure Linux 3 VMs and should change that.
• Not changing/adding any raw VM usage at this time.
• Need to run outerloop tests for this PR (the changes are in outerloop config)
• Related: Revert "Switch libraries testing to Ubuntu 22 temporarily" #115312

[Copilot]

Pull Request Overview

This PR updates the remaining x64 Helix queues to use Azure Linux 3 and reorders the configuration list.

• Replaces Ubuntu-based queue references with azurelinux.3 in multiple configurations.
• Updates image references for CentOS, Fedora, Ubuntu, and Debian queues accordingly.

Comments suppressed due to low confidence (1)

eng/pipelines/libraries/helix-queues-setup.yml:61

• [nitpick] There is an inconsistency in the casing between the original 'AzureLinux.3.0.Amd64.Open' and the updated 'azurelinux.3.Amd64.open'. Consider using consistent capitalization throughout for clarity.

- (AzureLinux.3.0.Amd64.Open)azurelinux.3.Amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

[richlander]

/azp run runtime-libraries-coreclr outerloop-linux

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[richlander]

Failures like the following seem like a machine configuration issue. There are many of them.

 System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileStreamLargeCount_Throws [FAIL]
System.IO.IOException : No file descriptors available :'/tmp/ropucomr.mjl_.ctor_39'
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>
System.Net.Sockets.Tests.SendPacketsAsync.Disposed_Throw [FAIL]
System.OutOfMemoryException : Exception of type'System.OutOfMemoryException' was thrown.
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>

@ilyas1974 -- can you run ulimit -a on the Ubuntu 22.04 and Azure Linux 3 VMs? We need to see what the difference is. I doubt the test code is at issue. Or how do I do this myself?

[ilyas1974]

Will see what I can do about getting you this information later today. Here are the instructions for self-service: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[ilyas1974]

Azure Linux 3

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31700
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31700
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Ubuntu 22.04

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31733
max locked memory (kbytes, -l) 1016148
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31733
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[richlander]

Thanks! Looks like -n is the problem. It is the difference I was expecting. We need to set that to the same value as Ubuntu. Does the Ubuntu VM just come that way or do we set it some way?

[ilyas1974]

This is the way the system comes to us. While we install software on systems - based on our customer's needs, we don't make any changes to the OS.

[akoeplinger]

@ilyas1974 ulimit for open files is explicitly changed here: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/definitions/shared/linux.yaml&version=GBmain&_a=contents&line=133&lineStyle=plain&lineEnd=136&lineStartColumn=1&lineEndColumn=1

[richlander]

Nice find. We need that same change for Azure Linux (or any other VM host we use).

Can that happen for the next roll-out? If not, we're going to need an off-cycle deployment.

[richlander]

• Switch remaining x64 helix queues to Azure Linux 3.
• Reordered the long list of configs to be VM first, then container, all alpha ordered.
• We do not have any Arm64 Azure Linux 3 VMs and should change that.
• Not changing/adding any raw VM usage at this time.
• Need to run outerloop tests for this PR (the changes are in outerloop config)
• Related: Revert "Switch libraries testing to Ubuntu 22 temporarily" #115312

[Copilot]

Pull Request Overview

This PR updates the remaining x64 Helix queues to use Azure Linux 3 and reorders the configuration list.

• Replaces Ubuntu-based queue references with azurelinux.3 in multiple configurations.
• Updates image references for CentOS, Fedora, Ubuntu, and Debian queues accordingly.

Comments suppressed due to low confidence (1)

eng/pipelines/libraries/helix-queues-setup.yml:61

• [nitpick] There is an inconsistency in the casing between the original 'AzureLinux.3.0.Amd64.Open' and the updated 'azurelinux.3.Amd64.open'. Consider using consistent capitalization throughout for clarity.

- (AzureLinux.3.0.Amd64.Open)azurelinux.3.Amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

[richlander]

/azp run runtime-libraries-coreclr outerloop-linux

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[richlander]

Failures like the following seem like a machine configuration issue. There are many of them.

 System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileStreamLargeCount_Throws [FAIL]
System.IO.IOException : No file descriptors available :'/tmp/ropucomr.mjl_.ctor_39'
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>
System.Net.Sockets.Tests.SendPacketsAsync.Disposed_Throw [FAIL]
System.OutOfMemoryException : Exception of type'System.OutOfMemoryException' was thrown.
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>

@ilyas1974 -- can you run ulimit -a on the Ubuntu 22.04 and Azure Linux 3 VMs? We need to see what the difference is. I doubt the test code is at issue. Or how do I do this myself?

[ilyas1974]

Will see what I can do about getting you this information later today. Here are the instructions for self-service: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[ilyas1974]

Azure Linux 3

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31700
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31700
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Ubuntu 22.04

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31733
max locked memory (kbytes, -l) 1016148
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31733
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[richlander]

Thanks! Looks like -n is the problem. It is the difference I was expecting. We need to set that to the same value as Ubuntu. Does the Ubuntu VM just come that way or do we set it some way?

[ilyas1974]

This is the way the system comes to us. While we install software on systems - based on our customer's needs, we don't make any changes to the OS.

[akoeplinger]

@ilyas1974 ulimit for open files is explicitly changed here: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/definitions/shared/linux.yaml&version=GBmain&_a=contents&line=133&lineStyle=plain&lineEnd=136&lineStartColumn=1&lineEndColumn=1

[richlander]

Nice find. We need that same change for Azure Linux (or any other VM host we use).

Can that happen for the next roll-out? If not, we're going to need an off-cycle deployment.

[richlander]

• Switch remaining x64 helix queues to Azure Linux 3.
• Reordered the long list of configs to be VM first, then container, all alpha ordered.
• We do not have any Arm64 Azure Linux 3 VMs and should change that.
• Not changing/adding any raw VM usage at this time.
• Need to run outerloop tests for this PR (the changes are in outerloop config)
• Related: Revert "Switch libraries testing to Ubuntu 22 temporarily" #115312

[Copilot]

Pull Request Overview

This PR updates the remaining x64 Helix queues to use Azure Linux 3 and reorders the configuration list.

• Replaces Ubuntu-based queue references with azurelinux.3 in multiple configurations.
• Updates image references for CentOS, Fedora, Ubuntu, and Debian queues accordingly.

Comments suppressed due to low confidence (1)

eng/pipelines/libraries/helix-queues-setup.yml:61

• [nitpick] There is an inconsistency in the casing between the original 'AzureLinux.3.0.Amd64.Open' and the updated 'azurelinux.3.Amd64.open'. Consider using consistent capitalization throughout for clarity.

- (AzureLinux.3.0.Amd64.Open)azurelinux.3.Amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

[richlander]

/azp run runtime-libraries-coreclr outerloop-linux

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[richlander]

Failures like the following seem like a machine configuration issue. There are many of them.

 System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileStreamLargeCount_Throws [FAIL]
System.IO.IOException : No file descriptors available :'/tmp/ropucomr.mjl_.ctor_39'
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>
System.Net.Sockets.Tests.SendPacketsAsync.Disposed_Throw [FAIL]
System.OutOfMemoryException : Exception of type'System.OutOfMemoryException' was thrown.
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>

@ilyas1974 -- can you run ulimit -a on the Ubuntu 22.04 and Azure Linux 3 VMs? We need to see what the difference is. I doubt the test code is at issue. Or how do I do this myself?

[ilyas1974]

Will see what I can do about getting you this information later today. Here are the instructions for self-service: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[ilyas1974]

Azure Linux 3

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31700
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31700
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Ubuntu 22.04

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31733
max locked memory (kbytes, -l) 1016148
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31733
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[richlander]

Thanks! Looks like -n is the problem. It is the difference I was expecting. We need to set that to the same value as Ubuntu. Does the Ubuntu VM just come that way or do we set it some way?

[ilyas1974]

This is the way the system comes to us. While we install software on systems - based on our customer's needs, we don't make any changes to the OS.

[akoeplinger]

@ilyas1974 ulimit for open files is explicitly changed here: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/definitions/shared/linux.yaml&version=GBmain&_a=contents&line=133&lineStyle=plain&lineEnd=136&lineStartColumn=1&lineEndColumn=1

[richlander]

Nice find. We need that same change for Azure Linux (or any other VM host we use).

Can that happen for the next roll-out? If not, we're going to need an off-cycle deployment.

[richlander]

• Switch remaining x64 helix queues to Azure Linux 3.
• Reordered the long list of configs to be VM first, then container, all alpha ordered.
• We do not have any Arm64 Azure Linux 3 VMs and should change that.
• Not changing/adding any raw VM usage at this time.
• Need to run outerloop tests for this PR (the changes are in outerloop config)
• Related: Revert "Switch libraries testing to Ubuntu 22 temporarily" #115312

[Copilot]

Pull Request Overview

This PR updates the remaining x64 Helix queues to use Azure Linux 3 and reorders the configuration list.

• Replaces Ubuntu-based queue references with azurelinux.3 in multiple configurations.
• Updates image references for CentOS, Fedora, Ubuntu, and Debian queues accordingly.

Comments suppressed due to low confidence (1)

eng/pipelines/libraries/helix-queues-setup.yml:61

• [nitpick] There is an inconsistency in the casing between the original 'AzureLinux.3.0.Amd64.Open' and the updated 'azurelinux.3.Amd64.open'. Consider using consistent capitalization throughout for clarity.

- (AzureLinux.3.0.Amd64.Open)azurelinux.3.Amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:azurelinux-3.0-helix-amd64

[richlander]

/azp run runtime-libraries-coreclr outerloop-linux

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[richlander]

Failures like the following seem like a machine configuration issue. There are many of them.

 System.Net.Sockets.Tests.SendPacketsAsync.SendPacketsElement_FileStreamLargeCount_Throws [FAIL]
System.IO.IOException : No file descriptors available :'/tmp/ropucomr.mjl_.ctor_39'
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>
System.Net.Sockets.Tests.SendPacketsAsync.Disposed_Throw [FAIL]
System.OutOfMemoryException : Exception of type'System.OutOfMemoryException' was thrown.
Stack Trace:
<exception thrown while retrieving exception stack trace: Exception of type'System.OutOfMemoryException' was thrown.>

@ilyas1974 -- can you run ulimit -a on the Ubuntu 22.04 and Azure Linux 3 VMs? We need to see what the difference is. I doubt the test code is at issue. Or how do I do this myself?

[ilyas1974]

Will see what I can do about getting you this information later today. Here are the instructions for self-service: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/915/Investigating-Helix-VM-images

[ilyas1974]

Azure Linux 3

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31700
max locked memory (kbytes, -l) 8192
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31700
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

Ubuntu 22.04

real-time non-blocking time (microseconds, -R) unlimited
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31733
max locked memory (kbytes, -l) 1016148
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31733
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

[richlander]

Thanks! Looks like -n is the problem. It is the difference I was expecting. We need to set that to the same value as Ubuntu. Does the Ubuntu VM just come that way or do we set it some way?

[ilyas1974]

This is the way the system comes to us. While we install software on systems - based on our customer's needs, we don't make any changes to the OS.

[akoeplinger]

@ilyas1974 ulimit for open files is explicitly changed here: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-machines?path=/definitions/shared/linux.yaml&version=GBmain&_a=contents&line=133&lineStyle=plain&lineEnd=136&lineStartColumn=1&lineEndColumn=1

[richlander]

Nice find. We need that same change for Azure Linux (or any other VM host we use).

Can that happen for the next roll-out? If not, we're going to need an off-cycle deployment.