Fix HTTP3 header decoder buffer allocation by BrunoBlanes · Pull Request #78862 · dotnet/runtime

BrunoBlanes · 2022-11-26T01:44:57Z

ghost · 2022-11-26T01:45:07Z

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

Author:	BrunoBlanes
Assignees:	-
Labels:	`area-System.Net.Http`
Milestone:	-

dnfadmin · 2022-11-26T01:45:11Z

All CLA requirements met.

BrunoBlanes · 2022-11-26T06:05:33Z

Although this should fix the issue, I believe I've identified at least one other point of failure in the original code, so I am willing to run some more tests.

BrunoBlanes · 2022-11-27T01:15:06Z

I added a few more tests and they all pass fine, so I did not find any other bug, however they run with a fixed sized header name and value, which in turn doesn't test how InternalDecoder behaves when the value length is over 127 bytes. Also, these tests only cover the Literal Field Without Name Reference case.

I'm aware of another test file of the same name located under the Common.Tests solution, however I wasn't able to get that running and also don't think these tests I've written are worth adding into the code base since they only cover a niche case, I've added them here mostly to shed a light into the investigation that's been done.

MihaZupan · 2022-12-07T20:46:48Z

Thank you for the investigation work here and for even providing a fix.
Sorry about the review delay, I'll get to this change this week.

MihaZupan

While looking more closely at this change, I realized that the same issue is present with the HPackDecoder (HTTP/2), it's just unlikely to be hit in practice as the initial size of the name buffer is 4 KB. As these files are quite similar, would you be willing to match the changes you made in QPackDecoder to HPackDecoder as well?

I'm aware of another test file of the same name located under the Common.Tests solution, however I wasn't able to get that running and also don't think these tests I've written are worth adding into the code base since they only cover a niche case, I've added them here mostly to shed a light into the investigation that's been done.

I think these tests are very valuable - they would have caught the bug if we had the before.

We shouldn't introduce a new QPackDecoderTest.cs file as the shared one already exists though. The HPackDecoder and QPackDecoder logic and their tests are shared between the runtime (HttpClient) and ASP.NET Core (Kestrel).

If you look at the System.Net.Http.Unit.Tests.csproj file, you can see how the product code and test code are linked into the project

<CompileInclude="$(CommonPath)System\Net\Http\aspnetcore\Http2\Hpack\HPackDecoder.cs"Link="Common\System\Net\Http\aspnetcore\Http2\Hpack\HPackDecoder.cs" />

<CompileInclude="$(CommonPath)..\tests\Tests\System\Net\aspnetcore\Http2\HPackDecoderTest.cs"Link="HPack\HPackDecoderTest.cs" />

We should be able to do the same for the QPackDecoder and its tests. Let me know if I can help with that.

BrunoBlanes · 2022-12-14T18:48:13Z

While looking more closely at this change, I realized that the same issue is present with the HPackDecoder (HTTP/2), it's just unlikely to be hit in practice as the initial size of the name buffer is 4 KB. As these files are quite similar, would you be willing to match the changes you made in QPackDecoder to HPackDecoder as well?

Yes, I'd love to help with that.

We should be able to do the same for the QPackDecoder and its tests. Let me know if I can help with that.

I'll see if I can manage that, else I'll let you know to help me out.

BrunoBlanes · 2022-12-15T22:02:12Z

I've been trying to do some debugging but am unable to attach:

System.Runtime.InteropServices.COMException (0x89710080): Unable to attach. A debug component is not installed.
at Microsoft.VisualStudio.Shell.Interop.IVsDebuggerLaunchAsync.LaunchDebugTargetsAsync(UInt32 debugTargetCount, VsDebugTargetInfo4[] debugTargetArray, IVsDebuggerLaunchCompletionCallback completionCallback)
at Microsoft.VisualStudio.TestWindow.ShellServices.DebuggerService.<>c__DisplayClass24_0.<<LaunchDebuggerAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.TestWindow.Logging.ILoggerExtensions.<CallWithCatchAsync>d__11`1.MoveNext()

Is there a known solution to this? I am running VS 17.5 Preview 2, I have updated my code base and rebuilt it but still cannot debug.

MihaZupan · 2022-12-16T15:34:18Z

How did you go about starting the debug session?
Personally I've had the most luck adding Debugger.Launch directly into the source I'm interested in and then running the test suite via CLI. Does that work?

BrunoBlanes · 2022-12-16T22:53:36Z

How did you go about starting the debug session?

I was just right clicking and selecting "Debug" in VS, but your method also gave me the same error:

The tests do indeed run, I just can't debug them. :/ Am I supposed to be using a specific version of .NET or VS?

PSC:\Users\bruno\runtime\src\libraries\System.Net.Http\tests\UnitTests> dotnetbuild/t:test/p:XunitMethodName=System.Net.Http.Unit.Tests.QPack.QPackDecoderTests.LiteralFieldWithoutNameReferece_ValueBrokenIntoSeparateBuffersMSBuildversion17.4.0+18d5aef85for.NETDeterminingprojectstorestore...Allprojectsareup-to-dateforrestore.Microsoft.Interop.SourceGeneration-> C:\Users\bruno\runtime\artifacts\bin\Microsoft.Interop.SourceGeneration\Debug\netstandard2.0\Microsoft.Interop.SourceGeneration.dllLibraryImportGenerator-> C:\Users\bruno\runtime\artifacts\bin\LibraryImportGenerator\Debug\netstandard2.0\Microsoft.Interop.LibraryImportGenerator.dllTestUtilities-> C:\Users\bruno\runtime\artifacts\bin\TestUtilities\Debug\net6.0\TestUtilities.dllSystem.Net.Http.Unit.Tests-> C:\Users\bruno\runtime\artifacts\bin\System.Net.Http.Unit.Tests\Debug\net8.0-windows\System.Net.Http.Unit.Tests.dll-----startFri12/16/202219:49:45.42===============Toreprodirectly:=====================================================pushdC:\Users\bruno\runtime\artifacts\bin\System.Net.Http.Unit.Tests\Debug\net8.0-windows\"C:\Users\bruno\runtime\artifacts\bin\testhost\net8.0-windows-Debug-x64\dotnet.exe"exec--runtimeconfigSystem.Net.Http.Unit.Tests.runtimeconfig.json--depsfileSystem.Net.Http.Unit.Tests.deps.jsonxunit.console.dllSystem.Net.Http.Unit.Tests.dll-xmltestResults.xml-nologo-methodSystem.Net.Http.Unit.Tests.QPack.QPackDecoderTests.LiteralFieldWithoutNameReferece_ValueBrokenIntoSeparateBuffers-notraitcategory=OuterLoop-notraitcategory=failingpopd===========================================================================================================Discovering:System.Net.Http.Unit.Tests(method display = ClassAndMethod, method display options = None)Discovered:System.Net.Http.Unit.Tests(found 1 of 915 test case)Starting:System.Net.Http.Unit.Tests(parallel test collections = on, max threads = 4)Finished:System.Net.Http.Unit.Tests===TESTEXECUTIONSUMMARY===System.Net.Http.Unit.TestsTotal:1,Errors:0,Failed:0,Skipped:0,Time:29.397s-----endFri12/16/202219:50:25.45-----exitcode0----------------------------------------------------------Buildsucceeded.0Warning(s)0Error(s)TimeElapsed00:01:35.99

pedrobsaila · 2022-12-26T19:18:35Z

@MihaZupan @BrunoBlanes I have the same issue (using VS 17.5.0 Preview 2.0). I tried also Debugger.Launch and got the same error.

System.Runtime.InteropServices.COMException (0x89710080): Unable to attach. A debug component is not installed.
at Microsoft.VisualStudio.Shell.Interop.IVsDebuggerLaunchAsync.LaunchDebugTargetsAsync(UInt32 debugTargetCount, VsDebugTargetInfo4[] debugTargetArray, IVsDebuggerLaunchCompletionCallback completionCallback)
at Microsoft.VisualStudio.TestWindow.ShellServices.DebuggerService.<>c__DisplayClass24_0.<<LaunchDebuggerAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.VisualStudio.TestWindow.Logging.ILoggerExtensions.<CallWithCatchAsync>d__11`1.MoveNext()

Rollbacking to VS 17.5.0 Preview 1.0 made debugging works again.

MihaZupan · 2022-12-27T10:37:41Z

Am I supposed to be using a specific version of .NET or VS?

You shouldn't need a preview version of VS. It seems to be working with 17.4.3 for example, but I too am running into issues with the current preview version.

Can you try using VS 17.5.0 Preview 1.0 as pedrobsaila suggested?

@dotnet/dnceng who would be the right contact to help here (the current preview of VS is failing to attach the debugger to a local build of dotnet/runtime)?

dkurepa · 2022-12-27T10:42:00Z

Hello, I'm not sure who'd be the best person to contact about this. @ilyas1974, could you recommend somebody?

ilyas1974 · 2022-12-27T14:53:21Z

Larry Bynum would be the person to ask about issues with VS.

danmoseley · 2022-12-27T15:32:18Z

@dotnet/dotnet-diag for that.

tommcdon · 2022-12-27T19:52:16Z

Please try the workaround noted here. Although the error message does not match it may be that signature validation is the root cause:

Starting with Visual Studio 2022 version 17.5, Visual Studio will validate that the debugging libraries that shipped with the .NET Runtime are correctly signed before loading them. If they are unsigned, Visual Studio will show an error like:

Unable to attach to CoreCLR. Signature validation failed for a .NET Runtime Debugger library because the file is unsigned.
This error is expected if you are working with non-official releases of .NET (example: daily builds from https://github.com/dotnet/installer). See https://aka.ms/vs/unsigned-dotnet-debugger-lib for more information.

If the target process is using a .NET Runtime that is either from a daily build, or one that you built on your own computer, this error will show up. NOTE: This error should never happen for official builds of the .NET Runtime from Microsoft. So don’t disable the validation if you expect to be using a .NET Runtime supported by Microsoft.

There are three ways to configure Visual Studio to disable signature validation:

The DOTNET_ROOT environment variable: if Visual Studio is started from a command prompt where DOTNET_ROOT is set, it will ignore unsigned .NET runtime debugger libraries which are under the DOTNET_ROOT directory.
The VSDebugger_ValidateDotnetDebugLibSignatures environment variable: If you want to temporarily disable signature validation, run set VSDebugger_ValidateDotnetDebugLibSignatures=0 in a command prompt, and start Visual Studio (devenv.exe) from this command prompt.
Set the ValidateDotnetDebugLibSignatures registry key: To disable signature validation on a more permanent bases, you can set the VS registry key to turn it off. To do so, open a Developer Command Prompt, and run Common7\IDE\VsRegEdit.exe set local HKCU Debugger\EngineSwitches ValidateDotnetDebugLibSignatures dword 0

karelz · 2023-03-07T17:25:12Z

We plan to look at it next week - @ManickaP

ghost · 2023-04-06T23:01:58Z

Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it.

ManickaP · 2023-04-13T13:28:27Z

@MihaZupan I think this is ready for re-review.

cc @Tratcher @JamesNK

MihaZupan

Looks good. I admit I forgot some of the details around the decoder implementation, but your change in HPack looks similar enough to Qpack.

ManickaP · 2023-04-17T07:48:13Z

@BrunoBlanes could you please resolve the CLA from #78862 (comment)? I cannot merge this without it.
The other option for me, would be to port the changes into a new PR of mine. I'd prefer not to as you're the original author here.

BrunoBlanes · 2023-04-18T17:12:09Z

@dotnet-policy-service agree

* Add test for literal field without name reference * Fix header name buffer allocation * Add more tests * Unified QPackDecoderTest test files * Fix variable name * Fixed HPackDecoder and ported QPack tests * Feedback --------- Co-authored-by: ManickaP <mapichov@microsoft.com>

* Add test for literal field without name reference * Fix header name buffer allocation * Add more tests * Unified QPackDecoderTest test files * Fix variable name * Fixed HPackDecoder and ported QPack tests * Feedback --------- Co-authored-by: Bruno Blanes <bruno.blanes@outlook.com>

BrunoBlanes added 2 commits November 25, 2022 22:39

Add test for literal field without name reference
9f728e2

Fix header name buffer allocation
6b847cb

ghost added area-System.Net.Http community-contribution Indicates that the PR has been added by a community member labels Nov 26, 2022

build-analysisBot mentioned this pull request Nov 26, 2022
CI build failure: Build MacCatalyst x64 Release AllSubsets_Mono - XcodeBuildApp task failed unexpectedly - Could not find System.Runtime.Tests.app #78778
Closed

Add more tests
c456ff5

BrunoBlanes commented Nov 27, 2022
View reviewed changes

Comment threadsrc/libraries/Common/src/System/Net/Http/aspnetcore/Http3/QPack/QPackDecoder.cs

karelz requested a review from MihaZupan November 29, 2022 17:22

karelz assigned MihaZupan Nov 29, 2022

MihaZupan reviewed Dec 14, 2022
View reviewed changes

Comment threadsrc/libraries/Common/src/System/Net/Http/aspnetcore/Http3/QPack/QPackDecoder.cs

Unified QPackDecoderTest test files
2838c30

This was referenced Dec 15, 2022
Tracking issue for CI build timeouts #76454
Closed
[wasm] Library tests failing during linking for AOT - SIGKILL#79569
Closed

BrunoBlanes added 2 commits December 15, 2022 01:02

Fix variable name
7f0239a

Merge branch 'dotnet:main' into main
555d08f

ManickaP marked this pull request as draft March 7, 2023 17:28

ghost closed this Apr 6, 2023

rzikm reopened this Apr 11, 2023

Fixed HPackDecoder and ported QPack tests
4c4413c

ManickaP marked this pull request as ready for review April 13, 2023 13:27

MihaZupan approved these changes Apr 13, 2023
View reviewed changes

Comment threadsrc/libraries/Common/tests/Tests/System/Net/aspnetcore/Http2/HPackDecoderTest.cs Outdated

Feedback
8fc8fdc

This was referenced Apr 13, 2023
[wasm] interpreter timeouts when WebSocket closes unexpectedly #84101
Closed
WasmTestOnBrowser-System.Text.Json.Tests.WorkItemExecution timing out #84434
Closed

ManickaP merged commit 0f0eba4 into dotnet:mainApr 19, 2023

ManickaP mentioned this pull request Apr 25, 2023
[release/7.0] Fix HTTP/3 and HTTTP/2 header decoder buffer allocation #85337
Closed

This was referenced Apr 25, 2023
[release/6.0] Fix HTTTP/2 header decoder buffer allocation #85341
Closed
[release/7.0] Fix HTTP/3 and HTTP/2 header decoder buffer allocation dotnet/aspnetcore#47949
Merged
[release/6.0] Fix HTTP/2 header decoder buffer allocation dotnet/aspnetcore#47950
Merged

ManickaP mentioned this pull request May 9, 2023
[release/6.0] Fix HTTTP/2 header decoder buffer allocation #85976
Merged

ManickaP mentioned this pull request May 9, 2023
[release/7.0] Fix HTTP/3 and HTTTP/2 header decoder buffer allocation #85977
Merged

ghost locked as resolved and limited conversation to collaborators May 19, 2023

Uh oh!

Conversation

BrunoBlanes commented Nov 26, 2022

Uh oh!

ghost commented Nov 26, 2022

Uh oh!

dnfadmin commented Nov 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BrunoBlanes commented Nov 26, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BrunoBlanes commented Nov 27, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

MihaZupan commented Dec 7, 2022

Uh oh!

MihaZupan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BrunoBlanes commented Dec 14, 2022

Uh oh!

BrunoBlanes commented Dec 15, 2022

Uh oh!

MihaZupan commented Dec 16, 2022

Uh oh!

BrunoBlanes commented Dec 16, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pedrobsaila commented Dec 26, 2022• edited by karelz Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MihaZupan commented Dec 27, 2022

Uh oh!

dkurepa commented Dec 27, 2022

Uh oh!

ilyas1974 commented Dec 27, 2022

Uh oh!

danmoseley commented Dec 27, 2022

Uh oh!

tommcdon commented Dec 27, 2022

Uh oh!

karelz commented Mar 7, 2023

Uh oh!

ghost commented Apr 6, 2023

Uh oh!

ManickaP commented Apr 13, 2023

Uh oh!

MihaZupan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ManickaP commented Apr 17, 2023

Uh oh!

BrunoBlanes commented Apr 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

dnfadmin commented Nov 26, 2022•
edited
Loading

BrunoBlanes commented Nov 26, 2022•
edited
Loading

BrunoBlanes commented Nov 27, 2022•
edited
Loading

BrunoBlanes commented Dec 16, 2022•
edited
Loading

pedrobsaila commented Dec 26, 2022•
edited by karelz
Loading