Privately expose MsQuic performance counters

[rzikm]

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

Since this is going to be internal use only, I went with capping the refresh rate of backgorund counters at once every 50ms, current monitoring tools usually query every X seconds so this should be low enough not to confuse any monitoring tool and high enough to finish querying all counters before the next refresh.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic.MsQuic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic.MsQuic]
msquic.app.recv_bytes (By / 1 sec) 594,652
msquic.app.send_bytes (By / 1 sec) 1,056,986
msquic.connection.allocated ({connection}) 2
msquic.connection.app_rejected ({connection} / 1 sec) 0
msquic.connection.connected ({connection}) 2
msquic.connection.created ({connection} / 1 sec) 48
msquic.connection.handshake_failures ({connection} / 1 sec) 0 msquic.connection.no_alpn ({connection} / 1 sec) 0
msquic.connection.protocol_errors ({connection} / 1 sec) 0
msquic.connection.resumed ({connection} / 1 sec) 0
msquic.datapath.path_failure ({challenge} / 1 sec) 0
msquic.datapath.path_validated ({challenge} / 1 sec) 0
msquic.datapath.send_stateless_reset ({packet} / 1 sec) 0
msquic.datapath.send_stateless_retry ({packet} / 1 sec) 0
msquic.packet.decryption_failures ({packet} / 1 sec) 0
msquic.packet.dropped ({packet} / 1 sec) 0
msquic.packet.suspected_lost ({packet} / 1 sec) 0
msquic.stream.allocated ({stream}) 1
msquic.threadpool.conn_oper_completed ({operation} / 1 sec) 1,540
msquic.threadpool.conn_oper_queue_depth ({operation}) 0
msquic.threadpool.conn_oper_queued ({operation} / 1 sec) 1,540
msquic.threadpool.conn_queue_depth ({connection}) 0
msquic.threadpool.work_oper_completed ({operation} / 1 sec) 0
msquic.threadpool.work_oper_queue_depth ({operation}) 0
msquic.threadpool.work_oper_queued ({operation} / 1 sec) 0
msquic.udp.recv_bytes (By / 1 sec) 984,227
msquic.udp.recv_datagrams ({datagram} / 1 sec) 1,006
msquic.udp.recv_events ({event} / 1 sec) 1,006
msquic.udp.send_bytes (By / 1 sec) 984,227
msquic.udp.send_calls ({call} / 1 sec) 596
msquic.udp.send_datagrams ({datagram} / 1 sec) 1,117

This PR also adds dump of selected metrics after running QUIC functional tests.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

I think using the new Metrics API is the best middle-ground solution. We can use one ObservableGauge and one ObservableCounter and report multiple counters by tagging each measurement with perf counter name. While we can't specify individual descriptions and units for each individual counter, I think this is acceptable for internal diagnostics.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic]
MsQuic
Name=APP_RECV_BYTES 26,368,237
Name=APP_SEND_BYTES 26,368,237
Name=CONN_ACTIVE 1
Name=CONN_APP_REJECT 0
Name=CONN_CONNECTED 0
Name=CONN_CREATED 22
Name=CONN_HANDSHAKE_FAIL 2
Name=CONN_LOAD_REJECT 0
Name=CONN_NO_ALPN 0
Name=CONN_OPER_COMPLETED 3,047
Name=CONN_OPER_QUEUE_DEPTH 0
Name=CONN_OPER_QUEUED 3,046
Name=CONN_PROTOCOL_ERRORS 0
Name=CONN_QUEUE_DEPTH 0
Name=CONN_RESUMED 0
Name=PATH_FAILURE 0
Name=PATH_VALIDATED 0
Name=PKTS_DECRYPTION_FAIL 0
Name=PKTS_DROPPED 0
Name=PKTS_SUSPECTED_LOST 644
Name=SEND_STATELESS_RESET 0
Name=SEND_STATELESS_RETRY 0
Name=STRM_ACTIVE 0
Name=UDP_RECV 19,000
Name=UDP_RECV_BYTES 27,227,902
Name=UDP_RECV_EVENTS 1,151
Name=UDP_SEND 19,690
Name=UDP_SEND_BYTES 28,174,485
Name=UDP_SEND_CALLS 1,168
Name=WORK_OPER_COMPLETED 0
Name=WORK_OPER_QUEUE_DEPTH 0

Author:	rzikm
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[rzikm]

This is related to microsoft/msquic#4120, we should wait until that one merges.

We don't need to wait with MsQuic update before merging, the counter will be 0 until a version which supports it is loaded.

[ManickaP]

In general LGTM, thanks.

[danmoseley]

Cc @JamesNK in case relevant

[ManickaP]

Thank you!

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rzikm]

CI failures are unrelated.

[rzikm]

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

Since this is going to be internal use only, I went with capping the refresh rate of backgorund counters at once every 50ms, current monitoring tools usually query every X seconds so this should be low enough not to confuse any monitoring tool and high enough to finish querying all counters before the next refresh.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic.MsQuic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic.MsQuic]
msquic.app.recv_bytes (By / 1 sec) 594,652
msquic.app.send_bytes (By / 1 sec) 1,056,986
msquic.connection.allocated ({connection}) 2
msquic.connection.app_rejected ({connection} / 1 sec) 0
msquic.connection.connected ({connection}) 2
msquic.connection.created ({connection} / 1 sec) 48
msquic.connection.handshake_failures ({connection} / 1 sec) 0 msquic.connection.no_alpn ({connection} / 1 sec) 0
msquic.connection.protocol_errors ({connection} / 1 sec) 0
msquic.connection.resumed ({connection} / 1 sec) 0
msquic.datapath.path_failure ({challenge} / 1 sec) 0
msquic.datapath.path_validated ({challenge} / 1 sec) 0
msquic.datapath.send_stateless_reset ({packet} / 1 sec) 0
msquic.datapath.send_stateless_retry ({packet} / 1 sec) 0
msquic.packet.decryption_failures ({packet} / 1 sec) 0
msquic.packet.dropped ({packet} / 1 sec) 0
msquic.packet.suspected_lost ({packet} / 1 sec) 0
msquic.stream.allocated ({stream}) 1
msquic.threadpool.conn_oper_completed ({operation} / 1 sec) 1,540
msquic.threadpool.conn_oper_queue_depth ({operation}) 0
msquic.threadpool.conn_oper_queued ({operation} / 1 sec) 1,540
msquic.threadpool.conn_queue_depth ({connection}) 0
msquic.threadpool.work_oper_completed ({operation} / 1 sec) 0
msquic.threadpool.work_oper_queue_depth ({operation}) 0
msquic.threadpool.work_oper_queued ({operation} / 1 sec) 0
msquic.udp.recv_bytes (By / 1 sec) 984,227
msquic.udp.recv_datagrams ({datagram} / 1 sec) 1,006
msquic.udp.recv_events ({event} / 1 sec) 1,006
msquic.udp.send_bytes (By / 1 sec) 984,227
msquic.udp.send_calls ({call} / 1 sec) 596
msquic.udp.send_datagrams ({datagram} / 1 sec) 1,117

This PR also adds dump of selected metrics after running QUIC functional tests.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

I think using the new Metrics API is the best middle-ground solution. We can use one ObservableGauge and one ObservableCounter and report multiple counters by tagging each measurement with perf counter name. While we can't specify individual descriptions and units for each individual counter, I think this is acceptable for internal diagnostics.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic]
MsQuic
Name=APP_RECV_BYTES 26,368,237
Name=APP_SEND_BYTES 26,368,237
Name=CONN_ACTIVE 1
Name=CONN_APP_REJECT 0
Name=CONN_CONNECTED 0
Name=CONN_CREATED 22
Name=CONN_HANDSHAKE_FAIL 2
Name=CONN_LOAD_REJECT 0
Name=CONN_NO_ALPN 0
Name=CONN_OPER_COMPLETED 3,047
Name=CONN_OPER_QUEUE_DEPTH 0
Name=CONN_OPER_QUEUED 3,046
Name=CONN_PROTOCOL_ERRORS 0
Name=CONN_QUEUE_DEPTH 0
Name=CONN_RESUMED 0
Name=PATH_FAILURE 0
Name=PATH_VALIDATED 0
Name=PKTS_DECRYPTION_FAIL 0
Name=PKTS_DROPPED 0
Name=PKTS_SUSPECTED_LOST 644
Name=SEND_STATELESS_RESET 0
Name=SEND_STATELESS_RETRY 0
Name=STRM_ACTIVE 0
Name=UDP_RECV 19,000
Name=UDP_RECV_BYTES 27,227,902
Name=UDP_RECV_EVENTS 1,151
Name=UDP_SEND 19,690
Name=UDP_SEND_BYTES 28,174,485
Name=UDP_SEND_CALLS 1,168
Name=WORK_OPER_COMPLETED 0
Name=WORK_OPER_QUEUE_DEPTH 0

Author:	rzikm
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[rzikm]

This is related to microsoft/msquic#4120, we should wait until that one merges.

We don't need to wait with MsQuic update before merging, the counter will be 0 until a version which supports it is loaded.

[ManickaP]

In general LGTM, thanks.

[danmoseley]

Cc @JamesNK in case relevant

[ManickaP]

Thank you!

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rzikm]

CI failures are unrelated.

[rzikm]

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

Since this is going to be internal use only, I went with capping the refresh rate of backgorund counters at once every 50ms, current monitoring tools usually query every X seconds so this should be low enough not to confuse any monitoring tool and high enough to finish querying all counters before the next refresh.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic.MsQuic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic.MsQuic]
msquic.app.recv_bytes (By / 1 sec) 594,652
msquic.app.send_bytes (By / 1 sec) 1,056,986
msquic.connection.allocated ({connection}) 2
msquic.connection.app_rejected ({connection} / 1 sec) 0
msquic.connection.connected ({connection}) 2
msquic.connection.created ({connection} / 1 sec) 48
msquic.connection.handshake_failures ({connection} / 1 sec) 0 msquic.connection.no_alpn ({connection} / 1 sec) 0
msquic.connection.protocol_errors ({connection} / 1 sec) 0
msquic.connection.resumed ({connection} / 1 sec) 0
msquic.datapath.path_failure ({challenge} / 1 sec) 0
msquic.datapath.path_validated ({challenge} / 1 sec) 0
msquic.datapath.send_stateless_reset ({packet} / 1 sec) 0
msquic.datapath.send_stateless_retry ({packet} / 1 sec) 0
msquic.packet.decryption_failures ({packet} / 1 sec) 0
msquic.packet.dropped ({packet} / 1 sec) 0
msquic.packet.suspected_lost ({packet} / 1 sec) 0
msquic.stream.allocated ({stream}) 1
msquic.threadpool.conn_oper_completed ({operation} / 1 sec) 1,540
msquic.threadpool.conn_oper_queue_depth ({operation}) 0
msquic.threadpool.conn_oper_queued ({operation} / 1 sec) 1,540
msquic.threadpool.conn_queue_depth ({connection}) 0
msquic.threadpool.work_oper_completed ({operation} / 1 sec) 0
msquic.threadpool.work_oper_queue_depth ({operation}) 0
msquic.threadpool.work_oper_queued ({operation} / 1 sec) 0
msquic.udp.recv_bytes (By / 1 sec) 984,227
msquic.udp.recv_datagrams ({datagram} / 1 sec) 1,006
msquic.udp.recv_events ({event} / 1 sec) 1,006
msquic.udp.send_bytes (By / 1 sec) 984,227
msquic.udp.send_calls ({call} / 1 sec) 596
msquic.udp.send_datagrams ({datagram} / 1 sec) 1,117

This PR also adds dump of selected metrics after running QUIC functional tests.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

I think using the new Metrics API is the best middle-ground solution. We can use one ObservableGauge and one ObservableCounter and report multiple counters by tagging each measurement with perf counter name. While we can't specify individual descriptions and units for each individual counter, I think this is acceptable for internal diagnostics.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic]
MsQuic
Name=APP_RECV_BYTES 26,368,237
Name=APP_SEND_BYTES 26,368,237
Name=CONN_ACTIVE 1
Name=CONN_APP_REJECT 0
Name=CONN_CONNECTED 0
Name=CONN_CREATED 22
Name=CONN_HANDSHAKE_FAIL 2
Name=CONN_LOAD_REJECT 0
Name=CONN_NO_ALPN 0
Name=CONN_OPER_COMPLETED 3,047
Name=CONN_OPER_QUEUE_DEPTH 0
Name=CONN_OPER_QUEUED 3,046
Name=CONN_PROTOCOL_ERRORS 0
Name=CONN_QUEUE_DEPTH 0
Name=CONN_RESUMED 0
Name=PATH_FAILURE 0
Name=PATH_VALIDATED 0
Name=PKTS_DECRYPTION_FAIL 0
Name=PKTS_DROPPED 0
Name=PKTS_SUSPECTED_LOST 644
Name=SEND_STATELESS_RESET 0
Name=SEND_STATELESS_RETRY 0
Name=STRM_ACTIVE 0
Name=UDP_RECV 19,000
Name=UDP_RECV_BYTES 27,227,902
Name=UDP_RECV_EVENTS 1,151
Name=UDP_SEND 19,690
Name=UDP_SEND_BYTES 28,174,485
Name=UDP_SEND_CALLS 1,168
Name=WORK_OPER_COMPLETED 0
Name=WORK_OPER_QUEUE_DEPTH 0

Author:	rzikm
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[rzikm]

This is related to microsoft/msquic#4120, we should wait until that one merges.

We don't need to wait with MsQuic update before merging, the counter will be 0 until a version which supports it is loaded.

[ManickaP]

In general LGTM, thanks.

[danmoseley]

Cc @JamesNK in case relevant

[ManickaP]

Thank you!

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rzikm]

CI failures are unrelated.

[rzikm]

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

Since this is going to be internal use only, I went with capping the refresh rate of backgorund counters at once every 50ms, current monitoring tools usually query every X seconds so this should be low enough not to confuse any monitoring tool and high enough to finish querying all counters before the next refresh.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic.MsQuic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic.MsQuic]
msquic.app.recv_bytes (By / 1 sec) 594,652
msquic.app.send_bytes (By / 1 sec) 1,056,986
msquic.connection.allocated ({connection}) 2
msquic.connection.app_rejected ({connection} / 1 sec) 0
msquic.connection.connected ({connection}) 2
msquic.connection.created ({connection} / 1 sec) 48
msquic.connection.handshake_failures ({connection} / 1 sec) 0 msquic.connection.no_alpn ({connection} / 1 sec) 0
msquic.connection.protocol_errors ({connection} / 1 sec) 0
msquic.connection.resumed ({connection} / 1 sec) 0
msquic.datapath.path_failure ({challenge} / 1 sec) 0
msquic.datapath.path_validated ({challenge} / 1 sec) 0
msquic.datapath.send_stateless_reset ({packet} / 1 sec) 0
msquic.datapath.send_stateless_retry ({packet} / 1 sec) 0
msquic.packet.decryption_failures ({packet} / 1 sec) 0
msquic.packet.dropped ({packet} / 1 sec) 0
msquic.packet.suspected_lost ({packet} / 1 sec) 0
msquic.stream.allocated ({stream}) 1
msquic.threadpool.conn_oper_completed ({operation} / 1 sec) 1,540
msquic.threadpool.conn_oper_queue_depth ({operation}) 0
msquic.threadpool.conn_oper_queued ({operation} / 1 sec) 1,540
msquic.threadpool.conn_queue_depth ({connection}) 0
msquic.threadpool.work_oper_completed ({operation} / 1 sec) 0
msquic.threadpool.work_oper_queue_depth ({operation}) 0
msquic.threadpool.work_oper_queued ({operation} / 1 sec) 0
msquic.udp.recv_bytes (By / 1 sec) 984,227
msquic.udp.recv_datagrams ({datagram} / 1 sec) 1,006
msquic.udp.recv_events ({event} / 1 sec) 1,006
msquic.udp.send_bytes (By / 1 sec) 984,227
msquic.udp.send_calls ({call} / 1 sec) 596
msquic.udp.send_datagrams ({datagram} / 1 sec) 1,117

This PR also adds dump of selected metrics after running QUIC functional tests.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

I think using the new Metrics API is the best middle-ground solution. We can use one ObservableGauge and one ObservableCounter and report multiple counters by tagging each measurement with perf counter name. While we can't specify individual descriptions and units for each individual counter, I think this is acceptable for internal diagnostics.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic]
MsQuic
Name=APP_RECV_BYTES 26,368,237
Name=APP_SEND_BYTES 26,368,237
Name=CONN_ACTIVE 1
Name=CONN_APP_REJECT 0
Name=CONN_CONNECTED 0
Name=CONN_CREATED 22
Name=CONN_HANDSHAKE_FAIL 2
Name=CONN_LOAD_REJECT 0
Name=CONN_NO_ALPN 0
Name=CONN_OPER_COMPLETED 3,047
Name=CONN_OPER_QUEUE_DEPTH 0
Name=CONN_OPER_QUEUED 3,046
Name=CONN_PROTOCOL_ERRORS 0
Name=CONN_QUEUE_DEPTH 0
Name=CONN_RESUMED 0
Name=PATH_FAILURE 0
Name=PATH_VALIDATED 0
Name=PKTS_DECRYPTION_FAIL 0
Name=PKTS_DROPPED 0
Name=PKTS_SUSPECTED_LOST 644
Name=SEND_STATELESS_RESET 0
Name=SEND_STATELESS_RETRY 0
Name=STRM_ACTIVE 0
Name=UDP_RECV 19,000
Name=UDP_RECV_BYTES 27,227,902
Name=UDP_RECV_EVENTS 1,151
Name=UDP_SEND 19,690
Name=UDP_SEND_BYTES 28,174,485
Name=UDP_SEND_CALLS 1,168
Name=WORK_OPER_COMPLETED 0
Name=WORK_OPER_QUEUE_DEPTH 0

Author:	rzikm
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[rzikm]

This is related to microsoft/msquic#4120, we should wait until that one merges.

We don't need to wait with MsQuic update before merging, the counter will be 0 until a version which supports it is loaded.

[ManickaP]

In general LGTM, thanks.

[danmoseley]

Cc @JamesNK in case relevant

[ManickaP]

Thank you!

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rzikm]

CI failures are unrelated.

[rzikm]

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

Since this is going to be internal use only, I went with capping the refresh rate of backgorund counters at once every 50ms, current monitoring tools usually query every X seconds so this should be low enough not to confuse any monitoring tool and high enough to finish querying all counters before the next refresh.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic.MsQuic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic.MsQuic]
msquic.app.recv_bytes (By / 1 sec) 594,652
msquic.app.send_bytes (By / 1 sec) 1,056,986
msquic.connection.allocated ({connection}) 2
msquic.connection.app_rejected ({connection} / 1 sec) 0
msquic.connection.connected ({connection}) 2
msquic.connection.created ({connection} / 1 sec) 48
msquic.connection.handshake_failures ({connection} / 1 sec) 0 msquic.connection.no_alpn ({connection} / 1 sec) 0
msquic.connection.protocol_errors ({connection} / 1 sec) 0
msquic.connection.resumed ({connection} / 1 sec) 0
msquic.datapath.path_failure ({challenge} / 1 sec) 0
msquic.datapath.path_validated ({challenge} / 1 sec) 0
msquic.datapath.send_stateless_reset ({packet} / 1 sec) 0
msquic.datapath.send_stateless_retry ({packet} / 1 sec) 0
msquic.packet.decryption_failures ({packet} / 1 sec) 0
msquic.packet.dropped ({packet} / 1 sec) 0
msquic.packet.suspected_lost ({packet} / 1 sec) 0
msquic.stream.allocated ({stream}) 1
msquic.threadpool.conn_oper_completed ({operation} / 1 sec) 1,540
msquic.threadpool.conn_oper_queue_depth ({operation}) 0
msquic.threadpool.conn_oper_queued ({operation} / 1 sec) 1,540
msquic.threadpool.conn_queue_depth ({connection}) 0
msquic.threadpool.work_oper_completed ({operation} / 1 sec) 0
msquic.threadpool.work_oper_queue_depth ({operation}) 0
msquic.threadpool.work_oper_queued ({operation} / 1 sec) 0
msquic.udp.recv_bytes (By / 1 sec) 984,227
msquic.udp.recv_datagrams ({datagram} / 1 sec) 1,006
msquic.udp.recv_events ({event} / 1 sec) 1,006
msquic.udp.send_bytes (By / 1 sec) 984,227
msquic.udp.send_calls ({call} / 1 sec) 596
msquic.udp.send_datagrams ({datagram} / 1 sec) 1,117

This PR also adds dump of selected metrics after running QUIC functional tests.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

I think using the new Metrics API is the best middle-ground solution. We can use one ObservableGauge and one ObservableCounter and report multiple counters by tagging each measurement with perf counter name. While we can't specify individual descriptions and units for each individual counter, I think this is acceptable for internal diagnostics.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic]
MsQuic
Name=APP_RECV_BYTES 26,368,237
Name=APP_SEND_BYTES 26,368,237
Name=CONN_ACTIVE 1
Name=CONN_APP_REJECT 0
Name=CONN_CONNECTED 0
Name=CONN_CREATED 22
Name=CONN_HANDSHAKE_FAIL 2
Name=CONN_LOAD_REJECT 0
Name=CONN_NO_ALPN 0
Name=CONN_OPER_COMPLETED 3,047
Name=CONN_OPER_QUEUE_DEPTH 0
Name=CONN_OPER_QUEUED 3,046
Name=CONN_PROTOCOL_ERRORS 0
Name=CONN_QUEUE_DEPTH 0
Name=CONN_RESUMED 0
Name=PATH_FAILURE 0
Name=PATH_VALIDATED 0
Name=PKTS_DECRYPTION_FAIL 0
Name=PKTS_DROPPED 0
Name=PKTS_SUSPECTED_LOST 644
Name=SEND_STATELESS_RESET 0
Name=SEND_STATELESS_RETRY 0
Name=STRM_ACTIVE 0
Name=UDP_RECV 19,000
Name=UDP_RECV_BYTES 27,227,902
Name=UDP_RECV_EVENTS 1,151
Name=UDP_SEND 19,690
Name=UDP_SEND_BYTES 28,174,485
Name=UDP_SEND_CALLS 1,168
Name=WORK_OPER_COMPLETED 0
Name=WORK_OPER_QUEUE_DEPTH 0

Author:	rzikm
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[rzikm]

This is related to microsoft/msquic#4120, we should wait until that one merges.

We don't need to wait with MsQuic update before merging, the counter will be 0 until a version which supports it is loaded.

[ManickaP]

In general LGTM, thanks.

[danmoseley]

Cc @JamesNK in case relevant

[ManickaP]

Thank you!

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rzikm]

CI failures are unrelated.

[rzikm]

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

Since this is going to be internal use only, I went with capping the refresh rate of backgorund counters at once every 50ms, current monitoring tools usually query every X seconds so this should be low enough not to confuse any monitoring tool and high enough to finish querying all counters before the next refresh.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic.MsQuic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic.MsQuic]
msquic.app.recv_bytes (By / 1 sec) 594,652
msquic.app.send_bytes (By / 1 sec) 1,056,986
msquic.connection.allocated ({connection}) 2
msquic.connection.app_rejected ({connection} / 1 sec) 0
msquic.connection.connected ({connection}) 2
msquic.connection.created ({connection} / 1 sec) 48
msquic.connection.handshake_failures ({connection} / 1 sec) 0 msquic.connection.no_alpn ({connection} / 1 sec) 0
msquic.connection.protocol_errors ({connection} / 1 sec) 0
msquic.connection.resumed ({connection} / 1 sec) 0
msquic.datapath.path_failure ({challenge} / 1 sec) 0
msquic.datapath.path_validated ({challenge} / 1 sec) 0
msquic.datapath.send_stateless_reset ({packet} / 1 sec) 0
msquic.datapath.send_stateless_retry ({packet} / 1 sec) 0
msquic.packet.decryption_failures ({packet} / 1 sec) 0
msquic.packet.dropped ({packet} / 1 sec) 0
msquic.packet.suspected_lost ({packet} / 1 sec) 0
msquic.stream.allocated ({stream}) 1
msquic.threadpool.conn_oper_completed ({operation} / 1 sec) 1,540
msquic.threadpool.conn_oper_queue_depth ({operation}) 0
msquic.threadpool.conn_oper_queued ({operation} / 1 sec) 1,540
msquic.threadpool.conn_queue_depth ({connection}) 0
msquic.threadpool.work_oper_completed ({operation} / 1 sec) 0
msquic.threadpool.work_oper_queue_depth ({operation}) 0
msquic.threadpool.work_oper_queued ({operation} / 1 sec) 0
msquic.udp.recv_bytes (By / 1 sec) 984,227
msquic.udp.recv_datagrams ({datagram} / 1 sec) 1,006
msquic.udp.recv_events ({event} / 1 sec) 1,006
msquic.udp.send_bytes (By / 1 sec) 984,227
msquic.udp.send_calls ({call} / 1 sec) 596
msquic.udp.send_datagrams ({datagram} / 1 sec) 1,117

This PR also adds dump of selected metrics after running QUIC functional tests.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

I think using the new Metrics API is the best middle-ground solution. We can use one ObservableGauge and one ObservableCounter and report multiple counters by tagging each measurement with perf counter name. While we can't specify individual descriptions and units for each individual counter, I think this is acceptable for internal diagnostics.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic]
MsQuic
Name=APP_RECV_BYTES 26,368,237
Name=APP_SEND_BYTES 26,368,237
Name=CONN_ACTIVE 1
Name=CONN_APP_REJECT 0
Name=CONN_CONNECTED 0
Name=CONN_CREATED 22
Name=CONN_HANDSHAKE_FAIL 2
Name=CONN_LOAD_REJECT 0
Name=CONN_NO_ALPN 0
Name=CONN_OPER_COMPLETED 3,047
Name=CONN_OPER_QUEUE_DEPTH 0
Name=CONN_OPER_QUEUED 3,046
Name=CONN_PROTOCOL_ERRORS 0
Name=CONN_QUEUE_DEPTH 0
Name=CONN_RESUMED 0
Name=PATH_FAILURE 0
Name=PATH_VALIDATED 0
Name=PKTS_DECRYPTION_FAIL 0
Name=PKTS_DROPPED 0
Name=PKTS_SUSPECTED_LOST 644
Name=SEND_STATELESS_RESET 0
Name=SEND_STATELESS_RETRY 0
Name=STRM_ACTIVE 0
Name=UDP_RECV 19,000
Name=UDP_RECV_BYTES 27,227,902
Name=UDP_RECV_EVENTS 1,151
Name=UDP_SEND 19,690
Name=UDP_SEND_BYTES 28,174,485
Name=UDP_SEND_CALLS 1,168
Name=WORK_OPER_COMPLETED 0
Name=WORK_OPER_QUEUE_DEPTH 0

Author:	rzikm
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[rzikm]

This is related to microsoft/msquic#4120, we should wait until that one merges.

We don't need to wait with MsQuic update before merging, the counter will be 0 until a version which supports it is loaded.

[ManickaP]

In general LGTM, thanks.

[danmoseley]

Cc @JamesNK in case relevant

[ManickaP]

Thank you!

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rzikm]

CI failures are unrelated.

[rzikm]

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

Since this is going to be internal use only, I went with capping the refresh rate of backgorund counters at once every 50ms, current monitoring tools usually query every X seconds so this should be low enough not to confuse any monitoring tool and high enough to finish querying all counters before the next refresh.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic.MsQuic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic.MsQuic]
msquic.app.recv_bytes (By / 1 sec) 594,652
msquic.app.send_bytes (By / 1 sec) 1,056,986
msquic.connection.allocated ({connection}) 2
msquic.connection.app_rejected ({connection} / 1 sec) 0
msquic.connection.connected ({connection}) 2
msquic.connection.created ({connection} / 1 sec) 48
msquic.connection.handshake_failures ({connection} / 1 sec) 0 msquic.connection.no_alpn ({connection} / 1 sec) 0
msquic.connection.protocol_errors ({connection} / 1 sec) 0
msquic.connection.resumed ({connection} / 1 sec) 0
msquic.datapath.path_failure ({challenge} / 1 sec) 0
msquic.datapath.path_validated ({challenge} / 1 sec) 0
msquic.datapath.send_stateless_reset ({packet} / 1 sec) 0
msquic.datapath.send_stateless_retry ({packet} / 1 sec) 0
msquic.packet.decryption_failures ({packet} / 1 sec) 0
msquic.packet.dropped ({packet} / 1 sec) 0
msquic.packet.suspected_lost ({packet} / 1 sec) 0
msquic.stream.allocated ({stream}) 1
msquic.threadpool.conn_oper_completed ({operation} / 1 sec) 1,540
msquic.threadpool.conn_oper_queue_depth ({operation}) 0
msquic.threadpool.conn_oper_queued ({operation} / 1 sec) 1,540
msquic.threadpool.conn_queue_depth ({connection}) 0
msquic.threadpool.work_oper_completed ({operation} / 1 sec) 0
msquic.threadpool.work_oper_queue_depth ({operation}) 0
msquic.threadpool.work_oper_queued ({operation} / 1 sec) 0
msquic.udp.recv_bytes (By / 1 sec) 984,227
msquic.udp.recv_datagrams ({datagram} / 1 sec) 1,006
msquic.udp.recv_events ({event} / 1 sec) 1,006
msquic.udp.send_bytes (By / 1 sec) 984,227
msquic.udp.send_calls ({call} / 1 sec) 596
msquic.udp.send_datagrams ({datagram} / 1 sec) 1,117

This PR also adds dump of selected metrics after running QUIC functional tests.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

I think using the new Metrics API is the best middle-ground solution. We can use one ObservableGauge and one ObservableCounter and report multiple counters by tagging each measurement with perf counter name. While we can't specify individual descriptions and units for each individual counter, I think this is acceptable for internal diagnostics.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic]
MsQuic
Name=APP_RECV_BYTES 26,368,237
Name=APP_SEND_BYTES 26,368,237
Name=CONN_ACTIVE 1
Name=CONN_APP_REJECT 0
Name=CONN_CONNECTED 0
Name=CONN_CREATED 22
Name=CONN_HANDSHAKE_FAIL 2
Name=CONN_LOAD_REJECT 0
Name=CONN_NO_ALPN 0
Name=CONN_OPER_COMPLETED 3,047
Name=CONN_OPER_QUEUE_DEPTH 0
Name=CONN_OPER_QUEUED 3,046
Name=CONN_PROTOCOL_ERRORS 0
Name=CONN_QUEUE_DEPTH 0
Name=CONN_RESUMED 0
Name=PATH_FAILURE 0
Name=PATH_VALIDATED 0
Name=PKTS_DECRYPTION_FAIL 0
Name=PKTS_DROPPED 0
Name=PKTS_SUSPECTED_LOST 644
Name=SEND_STATELESS_RESET 0
Name=SEND_STATELESS_RETRY 0
Name=STRM_ACTIVE 0
Name=UDP_RECV 19,000
Name=UDP_RECV_BYTES 27,227,902
Name=UDP_RECV_EVENTS 1,151
Name=UDP_SEND 19,690
Name=UDP_SEND_BYTES 28,174,485
Name=UDP_SEND_CALLS 1,168
Name=WORK_OPER_COMPLETED 0
Name=WORK_OPER_QUEUE_DEPTH 0

Author:	rzikm
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[rzikm]

This is related to microsoft/msquic#4120, we should wait until that one merges.

We don't need to wait with MsQuic update before merging, the counter will be 0 until a version which supports it is loaded.

[ManickaP]

In general LGTM, thanks.

[danmoseley]

Cc @JamesNK in case relevant

[ManickaP]

Thank you!

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rzikm]

CI failures are unrelated.

[rzikm]

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

Since this is going to be internal use only, I went with capping the refresh rate of backgorund counters at once every 50ms, current monitoring tools usually query every X seconds so this should be low enough not to confuse any monitoring tool and high enough to finish querying all counters before the next refresh.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic.MsQuic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic.MsQuic]
msquic.app.recv_bytes (By / 1 sec) 594,652
msquic.app.send_bytes (By / 1 sec) 1,056,986
msquic.connection.allocated ({connection}) 2
msquic.connection.app_rejected ({connection} / 1 sec) 0
msquic.connection.connected ({connection}) 2
msquic.connection.created ({connection} / 1 sec) 48
msquic.connection.handshake_failures ({connection} / 1 sec) 0 msquic.connection.no_alpn ({connection} / 1 sec) 0
msquic.connection.protocol_errors ({connection} / 1 sec) 0
msquic.connection.resumed ({connection} / 1 sec) 0
msquic.datapath.path_failure ({challenge} / 1 sec) 0
msquic.datapath.path_validated ({challenge} / 1 sec) 0
msquic.datapath.send_stateless_reset ({packet} / 1 sec) 0
msquic.datapath.send_stateless_retry ({packet} / 1 sec) 0
msquic.packet.decryption_failures ({packet} / 1 sec) 0
msquic.packet.dropped ({packet} / 1 sec) 0
msquic.packet.suspected_lost ({packet} / 1 sec) 0
msquic.stream.allocated ({stream}) 1
msquic.threadpool.conn_oper_completed ({operation} / 1 sec) 1,540
msquic.threadpool.conn_oper_queue_depth ({operation}) 0
msquic.threadpool.conn_oper_queued ({operation} / 1 sec) 1,540
msquic.threadpool.conn_queue_depth ({connection}) 0
msquic.threadpool.work_oper_completed ({operation} / 1 sec) 0
msquic.threadpool.work_oper_queue_depth ({operation}) 0
msquic.threadpool.work_oper_queued ({operation} / 1 sec) 0
msquic.udp.recv_bytes (By / 1 sec) 984,227
msquic.udp.recv_datagrams ({datagram} / 1 sec) 1,006
msquic.udp.recv_events ({event} / 1 sec) 1,006
msquic.udp.send_bytes (By / 1 sec) 984,227
msquic.udp.send_calls ({call} / 1 sec) 596
msquic.udp.send_datagrams ({datagram} / 1 sec) 1,117

This PR also adds dump of selected metrics after running QUIC functional tests.

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Contributes to #55979.

The main problem is that we can only get the MsQuic perf counters in bulk, so any observable/polling metrics/coutners API would require fetching entire array (32 items) for each separate counter and there is no reasonable way to extend the existing API without needing public API changes.

I think using the new Metrics API is the best middle-ground solution. We can use one ObservableGauge and one ObservableCounter and report multiple counters by tagging each measurement with perf counter name. While we can't specify individual descriptions and units for each individual counter, I think this is acceptable for internal diagnostics.

example when monitoring functional tests run:

dotnet counters monitor Private.InternalDiagnostics.System.Net.Quic -p (gps dotnet | ? { $_.Path -like '*testhost*' }).Id

Press p to pause, r to resume, q to quit.
Status: Running
[Private.InternalDiagnostics.System.Net.Quic]
MsQuic
Name=APP_RECV_BYTES 26,368,237
Name=APP_SEND_BYTES 26,368,237
Name=CONN_ACTIVE 1
Name=CONN_APP_REJECT 0
Name=CONN_CONNECTED 0
Name=CONN_CREATED 22
Name=CONN_HANDSHAKE_FAIL 2
Name=CONN_LOAD_REJECT 0
Name=CONN_NO_ALPN 0
Name=CONN_OPER_COMPLETED 3,047
Name=CONN_OPER_QUEUE_DEPTH 0
Name=CONN_OPER_QUEUED 3,046
Name=CONN_PROTOCOL_ERRORS 0
Name=CONN_QUEUE_DEPTH 0
Name=CONN_RESUMED 0
Name=PATH_FAILURE 0
Name=PATH_VALIDATED 0
Name=PKTS_DECRYPTION_FAIL 0
Name=PKTS_DROPPED 0
Name=PKTS_SUSPECTED_LOST 644
Name=SEND_STATELESS_RESET 0
Name=SEND_STATELESS_RETRY 0
Name=STRM_ACTIVE 0
Name=UDP_RECV 19,000
Name=UDP_RECV_BYTES 27,227,902
Name=UDP_RECV_EVENTS 1,151
Name=UDP_SEND 19,690
Name=UDP_SEND_BYTES 28,174,485
Name=UDP_SEND_CALLS 1,168
Name=WORK_OPER_COMPLETED 0
Name=WORK_OPER_QUEUE_DEPTH 0

Author:	rzikm
Assignees:	-
Labels:	area-System.Net.Quic
Milestone:	-

[rzikm]

This is related to microsoft/msquic#4120, we should wait until that one merges.

We don't need to wait with MsQuic update before merging, the counter will be 0 until a version which supports it is loaded.

[ManickaP]

In general LGTM, thanks.

[danmoseley]

Cc @JamesNK in case relevant

[ManickaP]

Thank you!

[rzikm]

/azp run runtime-libraries-coreclr outerloop

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[rzikm]

CI failures are unrelated.