Optimize HttpUtility.UrlPathEncode allocations. - #102823

Closed
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main
Closed

Optimize HttpUtility.UrlPathEncode allocations.#102823
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main

Conversation

@TrayanZapryanov

Copy link
Copy Markdown
Contributor
  1. Omit intermediate string allocations by using Ranges.
  2. Use stackalloc arrays when input is small enough for encoding and ASCII conversions.
  3. Remove string.Contains(' ') loop.

@ghostghost added the area-System.Net label May 29, 2024
@dotnet-policy-servicedotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label May 29, 2024
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

Benchmark:

[MemoryDiagnoser]publicclassHttpUtilityBenchmarks{[Benchmark(Baseline=true)]publicstringUrlTryCreate()=>Uri.TryCreate("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds",UriKind.Absolute,outvaruri)?uri.Authority:"error";[Benchmark]publicstringUrlPathEncode()=>HttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");[Benchmark]publicstringUrlPathEncode_PR()=>MyHttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");}

Results:

MethodMeanErrorStdDevRatioRatioSDGen0AllocatedAlloc Ratio
UrlTryCreate260.7 ns2.41 ns2.01 ns1.000.000.0277232 B1.00
UrlPathEncode338.5 ns6.03 ns5.64 ns1.300.020.0839704 B3.03
UrlPathEncode_PR325.4 ns4.59 ns4.30 ns1.250.020.0658552 B2.38

}

private static byte[] UrlEncodeNonAscii(byte[] bytes, int offset, int count)
private static string UrlEncodeNonAscii(Span<byte> bytes)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods are still allocating quite a bit (the original string even if it's already Ascii, substring and combined string in UrlPathEncodeImpl ...).

The 1 caller of this helper is using Encoding.UTF8 and wants to stop after you see a ?.
Assuming there's sufficient test coverage, I would change this to instead look something like

privatestaticstringUrlPathEncodeImpl(stringvalue){inti=value.AsSpan().IndexOfAnyExceptInRange((char)0x21,(char)0x7F);if(i<0){returnvalue;}intindexOfQuery=value.IndexOf('?');if(indexOfQuery>=0&&indexOfQuery<i){// Everything before the Query is valid ASCIIreturnvalue;}ReadOnlySpan<char>toEncode=indexOfQuery>=0?value.AsSpan(i,indexOfQuery-i):value.AsSpan(i);byte[]bytes=ArrayPool<char>.Shared.Rent(Encoding.UTF8.GetMaxByteCount(toEncode.Length));char[]chars=ArrayPool<char>.Shared.Rent(bytes.Length*3);intutf8Length=Encoding.UTF8.GetBytes(toEncode,bytes);intcharCount=0;foreach(bytebinbytes.AsSpan(0,utf8Length)){if(IsNonAsciiOrSpaceByte(b)){chars[charCount++]='%';chars[charCount++]=HexConverter.ToCharLower(b>>4);chars[charCount++]=HexConverter.ToCharLower(b);}else{chars[charCount++]=(char)b;}}ArrayPool<byte>.Shared.Return(bytes);stringresult=string.Concat(value.AsSpan(0,i),chars.AsSpan(0,charCount),indexOfQuery>=0?value.AsSpan(indexOfQuery):ReadOnlySpan<char>.Empty);ArrayPool<char>.Shared.Return(chars);returnresult;}

This would avoid a bunch of intermediate allocations and transcoding.

Comment on lines +595 to +597
schemeAndAuthority = default(Range);
path = Range.All;
queryAndFragment = default(Range);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
schemeAndAuthority=default(Range);
path=Range.All;
queryAndFragment=default(Range);
returnUrlPathEncodeImpl(value);

Otherwise you would end up allocating a substring for value.AsSpan(path).ToString() + a substring for the string.Concat.

}

return schemeAndAuthority + UrlPathEncodeImpl(path) + queryAndFragment;
return string.Concat(value.AsSpan(schemeAndAuthority), UrlPathEncodeImpl(value.AsSpan(path).ToString()), value.AsSpan(queryAndFragment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be combined into the whole UrlPathEncodeImpl to avoid the substring allocations

// Even if this method returns true, one or more of the out parameters might contain a null or empty string, e.g. if there is no query / fragment.
// Concatenating the pieces back together will form the original input string.
internal static bool TrySplitUriForPathEncode(string input, [NotNullWhen(true)] out string? schemeAndAuthority, [NotNullWhen(true)] out string? path, out string? queryAndFragment)
internal static bool TrySplitUriForPathEncode(string input, out Range schemeAndAuthority, out Range path, out Range queryAndFragment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By forcing everything into spans and ranges, you are now also allocating substrings where you might not have needed to previously. E.g. if pathRange is Range.All, you're reallocating the original string.

You could inline ExtractQueryAndFragment into this method, and always use Substring for the path portion.
I would also suggest returning Span instead of Range as it's easier to use given that the callers only need the range to make the span anyway.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

@MihaZupan Sorry I messed git while trying to rebase after merging previous PRs.
Sorry for the stupid question, but can you tell me how you are updating older branches to latest version without closing existing PR.
Meanwhile I will try to open another PR with your comments integrated.

@MihaZupan

Copy link
Copy Markdown
Member

Assuming upstream is this repo and origin is your fork, my workflow looks something like

git checkout main
git pull upstream
git checkout your-branch
git rebase main // And now fix any conflicts
git push origin -f

@karelzkarelz added this to the 9.0.0 milestone Jun 24, 2024
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Jul 25, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Netcommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TrayanZapryanov@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Optimize HttpUtility.UrlPathEncode allocations. - #102823

Closed
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main
Closed

Optimize HttpUtility.UrlPathEncode allocations.#102823
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main

Conversation

@TrayanZapryanov

Copy link
Copy Markdown
Contributor
  1. Omit intermediate string allocations by using Ranges.
  2. Use stackalloc arrays when input is small enough for encoding and ASCII conversions.
  3. Remove string.Contains(' ') loop.

@ghostghost added the area-System.Net label May 29, 2024
@dotnet-policy-servicedotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label May 29, 2024
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

Benchmark:

[MemoryDiagnoser]publicclassHttpUtilityBenchmarks{[Benchmark(Baseline=true)]publicstringUrlTryCreate()=>Uri.TryCreate("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds",UriKind.Absolute,outvaruri)?uri.Authority:"error";[Benchmark]publicstringUrlPathEncode()=>HttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");[Benchmark]publicstringUrlPathEncode_PR()=>MyHttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");}

Results:

MethodMeanErrorStdDevRatioRatioSDGen0AllocatedAlloc Ratio
UrlTryCreate260.7 ns2.41 ns2.01 ns1.000.000.0277232 B1.00
UrlPathEncode338.5 ns6.03 ns5.64 ns1.300.020.0839704 B3.03
UrlPathEncode_PR325.4 ns4.59 ns4.30 ns1.250.020.0658552 B2.38

}

private static byte[] UrlEncodeNonAscii(byte[] bytes, int offset, int count)
private static string UrlEncodeNonAscii(Span<byte> bytes)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods are still allocating quite a bit (the original string even if it's already Ascii, substring and combined string in UrlPathEncodeImpl ...).

The 1 caller of this helper is using Encoding.UTF8 and wants to stop after you see a ?.
Assuming there's sufficient test coverage, I would change this to instead look something like

privatestaticstringUrlPathEncodeImpl(stringvalue){inti=value.AsSpan().IndexOfAnyExceptInRange((char)0x21,(char)0x7F);if(i<0){returnvalue;}intindexOfQuery=value.IndexOf('?');if(indexOfQuery>=0&&indexOfQuery<i){// Everything before the Query is valid ASCIIreturnvalue;}ReadOnlySpan<char>toEncode=indexOfQuery>=0?value.AsSpan(i,indexOfQuery-i):value.AsSpan(i);byte[]bytes=ArrayPool<char>.Shared.Rent(Encoding.UTF8.GetMaxByteCount(toEncode.Length));char[]chars=ArrayPool<char>.Shared.Rent(bytes.Length*3);intutf8Length=Encoding.UTF8.GetBytes(toEncode,bytes);intcharCount=0;foreach(bytebinbytes.AsSpan(0,utf8Length)){if(IsNonAsciiOrSpaceByte(b)){chars[charCount++]='%';chars[charCount++]=HexConverter.ToCharLower(b>>4);chars[charCount++]=HexConverter.ToCharLower(b);}else{chars[charCount++]=(char)b;}}ArrayPool<byte>.Shared.Return(bytes);stringresult=string.Concat(value.AsSpan(0,i),chars.AsSpan(0,charCount),indexOfQuery>=0?value.AsSpan(indexOfQuery):ReadOnlySpan<char>.Empty);ArrayPool<char>.Shared.Return(chars);returnresult;}

This would avoid a bunch of intermediate allocations and transcoding.

Comment on lines +595 to +597
schemeAndAuthority = default(Range);
path = Range.All;
queryAndFragment = default(Range);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
schemeAndAuthority=default(Range);
path=Range.All;
queryAndFragment=default(Range);
returnUrlPathEncodeImpl(value);

Otherwise you would end up allocating a substring for value.AsSpan(path).ToString() + a substring for the string.Concat.

}

return schemeAndAuthority + UrlPathEncodeImpl(path) + queryAndFragment;
return string.Concat(value.AsSpan(schemeAndAuthority), UrlPathEncodeImpl(value.AsSpan(path).ToString()), value.AsSpan(queryAndFragment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be combined into the whole UrlPathEncodeImpl to avoid the substring allocations

// Even if this method returns true, one or more of the out parameters might contain a null or empty string, e.g. if there is no query / fragment.
// Concatenating the pieces back together will form the original input string.
internal static bool TrySplitUriForPathEncode(string input, [NotNullWhen(true)] out string? schemeAndAuthority, [NotNullWhen(true)] out string? path, out string? queryAndFragment)
internal static bool TrySplitUriForPathEncode(string input, out Range schemeAndAuthority, out Range path, out Range queryAndFragment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By forcing everything into spans and ranges, you are now also allocating substrings where you might not have needed to previously. E.g. if pathRange is Range.All, you're reallocating the original string.

You could inline ExtractQueryAndFragment into this method, and always use Substring for the path portion.
I would also suggest returning Span instead of Range as it's easier to use given that the callers only need the range to make the span anyway.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

@MihaZupan Sorry I messed git while trying to rebase after merging previous PRs.
Sorry for the stupid question, but can you tell me how you are updating older branches to latest version without closing existing PR.
Meanwhile I will try to open another PR with your comments integrated.

@MihaZupan

Copy link
Copy Markdown
Member

Assuming upstream is this repo and origin is your fork, my workflow looks something like

git checkout main
git pull upstream
git checkout your-branch
git rebase main // And now fix any conflicts
git push origin -f

@karelzkarelz added this to the 9.0.0 milestone Jun 24, 2024
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Jul 25, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Netcommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TrayanZapryanov@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Optimize HttpUtility.UrlPathEncode allocations. - #102823

Closed
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main
Closed

Optimize HttpUtility.UrlPathEncode allocations.#102823
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main

Conversation

@TrayanZapryanov

Copy link
Copy Markdown
Contributor
  1. Omit intermediate string allocations by using Ranges.
  2. Use stackalloc arrays when input is small enough for encoding and ASCII conversions.
  3. Remove string.Contains(' ') loop.

@ghostghost added the area-System.Net label May 29, 2024
@dotnet-policy-servicedotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label May 29, 2024
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

Benchmark:

[MemoryDiagnoser]publicclassHttpUtilityBenchmarks{[Benchmark(Baseline=true)]publicstringUrlTryCreate()=>Uri.TryCreate("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds",UriKind.Absolute,outvaruri)?uri.Authority:"error";[Benchmark]publicstringUrlPathEncode()=>HttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");[Benchmark]publicstringUrlPathEncode_PR()=>MyHttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");}

Results:

MethodMeanErrorStdDevRatioRatioSDGen0AllocatedAlloc Ratio
UrlTryCreate260.7 ns2.41 ns2.01 ns1.000.000.0277232 B1.00
UrlPathEncode338.5 ns6.03 ns5.64 ns1.300.020.0839704 B3.03
UrlPathEncode_PR325.4 ns4.59 ns4.30 ns1.250.020.0658552 B2.38

}

private static byte[] UrlEncodeNonAscii(byte[] bytes, int offset, int count)
private static string UrlEncodeNonAscii(Span<byte> bytes)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods are still allocating quite a bit (the original string even if it's already Ascii, substring and combined string in UrlPathEncodeImpl ...).

The 1 caller of this helper is using Encoding.UTF8 and wants to stop after you see a ?.
Assuming there's sufficient test coverage, I would change this to instead look something like

privatestaticstringUrlPathEncodeImpl(stringvalue){inti=value.AsSpan().IndexOfAnyExceptInRange((char)0x21,(char)0x7F);if(i<0){returnvalue;}intindexOfQuery=value.IndexOf('?');if(indexOfQuery>=0&&indexOfQuery<i){// Everything before the Query is valid ASCIIreturnvalue;}ReadOnlySpan<char>toEncode=indexOfQuery>=0?value.AsSpan(i,indexOfQuery-i):value.AsSpan(i);byte[]bytes=ArrayPool<char>.Shared.Rent(Encoding.UTF8.GetMaxByteCount(toEncode.Length));char[]chars=ArrayPool<char>.Shared.Rent(bytes.Length*3);intutf8Length=Encoding.UTF8.GetBytes(toEncode,bytes);intcharCount=0;foreach(bytebinbytes.AsSpan(0,utf8Length)){if(IsNonAsciiOrSpaceByte(b)){chars[charCount++]='%';chars[charCount++]=HexConverter.ToCharLower(b>>4);chars[charCount++]=HexConverter.ToCharLower(b);}else{chars[charCount++]=(char)b;}}ArrayPool<byte>.Shared.Return(bytes);stringresult=string.Concat(value.AsSpan(0,i),chars.AsSpan(0,charCount),indexOfQuery>=0?value.AsSpan(indexOfQuery):ReadOnlySpan<char>.Empty);ArrayPool<char>.Shared.Return(chars);returnresult;}

This would avoid a bunch of intermediate allocations and transcoding.

Comment on lines +595 to +597
schemeAndAuthority = default(Range);
path = Range.All;
queryAndFragment = default(Range);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
schemeAndAuthority=default(Range);
path=Range.All;
queryAndFragment=default(Range);
returnUrlPathEncodeImpl(value);

Otherwise you would end up allocating a substring for value.AsSpan(path).ToString() + a substring for the string.Concat.

}

return schemeAndAuthority + UrlPathEncodeImpl(path) + queryAndFragment;
return string.Concat(value.AsSpan(schemeAndAuthority), UrlPathEncodeImpl(value.AsSpan(path).ToString()), value.AsSpan(queryAndFragment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be combined into the whole UrlPathEncodeImpl to avoid the substring allocations

// Even if this method returns true, one or more of the out parameters might contain a null or empty string, e.g. if there is no query / fragment.
// Concatenating the pieces back together will form the original input string.
internal static bool TrySplitUriForPathEncode(string input, [NotNullWhen(true)] out string? schemeAndAuthority, [NotNullWhen(true)] out string? path, out string? queryAndFragment)
internal static bool TrySplitUriForPathEncode(string input, out Range schemeAndAuthority, out Range path, out Range queryAndFragment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By forcing everything into spans and ranges, you are now also allocating substrings where you might not have needed to previously. E.g. if pathRange is Range.All, you're reallocating the original string.

You could inline ExtractQueryAndFragment into this method, and always use Substring for the path portion.
I would also suggest returning Span instead of Range as it's easier to use given that the callers only need the range to make the span anyway.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

@MihaZupan Sorry I messed git while trying to rebase after merging previous PRs.
Sorry for the stupid question, but can you tell me how you are updating older branches to latest version without closing existing PR.
Meanwhile I will try to open another PR with your comments integrated.

@MihaZupan

Copy link
Copy Markdown
Member

Assuming upstream is this repo and origin is your fork, my workflow looks something like

git checkout main
git pull upstream
git checkout your-branch
git rebase main // And now fix any conflicts
git push origin -f

@karelzkarelz added this to the 9.0.0 milestone Jun 24, 2024
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Jul 25, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Netcommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TrayanZapryanov@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Optimize HttpUtility.UrlPathEncode allocations. - #102823

Closed
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main
Closed

Optimize HttpUtility.UrlPathEncode allocations.#102823
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main

Conversation

@TrayanZapryanov

Copy link
Copy Markdown
Contributor
  1. Omit intermediate string allocations by using Ranges.
  2. Use stackalloc arrays when input is small enough for encoding and ASCII conversions.
  3. Remove string.Contains(' ') loop.

@ghostghost added the area-System.Net label May 29, 2024
@dotnet-policy-servicedotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label May 29, 2024
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

Benchmark:

[MemoryDiagnoser]publicclassHttpUtilityBenchmarks{[Benchmark(Baseline=true)]publicstringUrlTryCreate()=>Uri.TryCreate("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds",UriKind.Absolute,outvaruri)?uri.Authority:"error";[Benchmark]publicstringUrlPathEncode()=>HttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");[Benchmark]publicstringUrlPathEncode_PR()=>MyHttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");}

Results:

MethodMeanErrorStdDevRatioRatioSDGen0AllocatedAlloc Ratio
UrlTryCreate260.7 ns2.41 ns2.01 ns1.000.000.0277232 B1.00
UrlPathEncode338.5 ns6.03 ns5.64 ns1.300.020.0839704 B3.03
UrlPathEncode_PR325.4 ns4.59 ns4.30 ns1.250.020.0658552 B2.38

}

private static byte[] UrlEncodeNonAscii(byte[] bytes, int offset, int count)
private static string UrlEncodeNonAscii(Span<byte> bytes)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods are still allocating quite a bit (the original string even if it's already Ascii, substring and combined string in UrlPathEncodeImpl ...).

The 1 caller of this helper is using Encoding.UTF8 and wants to stop after you see a ?.
Assuming there's sufficient test coverage, I would change this to instead look something like

privatestaticstringUrlPathEncodeImpl(stringvalue){inti=value.AsSpan().IndexOfAnyExceptInRange((char)0x21,(char)0x7F);if(i<0){returnvalue;}intindexOfQuery=value.IndexOf('?');if(indexOfQuery>=0&&indexOfQuery<i){// Everything before the Query is valid ASCIIreturnvalue;}ReadOnlySpan<char>toEncode=indexOfQuery>=0?value.AsSpan(i,indexOfQuery-i):value.AsSpan(i);byte[]bytes=ArrayPool<char>.Shared.Rent(Encoding.UTF8.GetMaxByteCount(toEncode.Length));char[]chars=ArrayPool<char>.Shared.Rent(bytes.Length*3);intutf8Length=Encoding.UTF8.GetBytes(toEncode,bytes);intcharCount=0;foreach(bytebinbytes.AsSpan(0,utf8Length)){if(IsNonAsciiOrSpaceByte(b)){chars[charCount++]='%';chars[charCount++]=HexConverter.ToCharLower(b>>4);chars[charCount++]=HexConverter.ToCharLower(b);}else{chars[charCount++]=(char)b;}}ArrayPool<byte>.Shared.Return(bytes);stringresult=string.Concat(value.AsSpan(0,i),chars.AsSpan(0,charCount),indexOfQuery>=0?value.AsSpan(indexOfQuery):ReadOnlySpan<char>.Empty);ArrayPool<char>.Shared.Return(chars);returnresult;}

This would avoid a bunch of intermediate allocations and transcoding.

Comment on lines +595 to +597
schemeAndAuthority = default(Range);
path = Range.All;
queryAndFragment = default(Range);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
schemeAndAuthority=default(Range);
path=Range.All;
queryAndFragment=default(Range);
returnUrlPathEncodeImpl(value);

Otherwise you would end up allocating a substring for value.AsSpan(path).ToString() + a substring for the string.Concat.

}

return schemeAndAuthority + UrlPathEncodeImpl(path) + queryAndFragment;
return string.Concat(value.AsSpan(schemeAndAuthority), UrlPathEncodeImpl(value.AsSpan(path).ToString()), value.AsSpan(queryAndFragment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be combined into the whole UrlPathEncodeImpl to avoid the substring allocations

// Even if this method returns true, one or more of the out parameters might contain a null or empty string, e.g. if there is no query / fragment.
// Concatenating the pieces back together will form the original input string.
internal static bool TrySplitUriForPathEncode(string input, [NotNullWhen(true)] out string? schemeAndAuthority, [NotNullWhen(true)] out string? path, out string? queryAndFragment)
internal static bool TrySplitUriForPathEncode(string input, out Range schemeAndAuthority, out Range path, out Range queryAndFragment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By forcing everything into spans and ranges, you are now also allocating substrings where you might not have needed to previously. E.g. if pathRange is Range.All, you're reallocating the original string.

You could inline ExtractQueryAndFragment into this method, and always use Substring for the path portion.
I would also suggest returning Span instead of Range as it's easier to use given that the callers only need the range to make the span anyway.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

@MihaZupan Sorry I messed git while trying to rebase after merging previous PRs.
Sorry for the stupid question, but can you tell me how you are updating older branches to latest version without closing existing PR.
Meanwhile I will try to open another PR with your comments integrated.

@MihaZupan

Copy link
Copy Markdown
Member

Assuming upstream is this repo and origin is your fork, my workflow looks something like

git checkout main
git pull upstream
git checkout your-branch
git rebase main // And now fix any conflicts
git push origin -f

@karelzkarelz added this to the 9.0.0 milestone Jun 24, 2024
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Jul 25, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Netcommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TrayanZapryanov@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Optimize HttpUtility.UrlPathEncode allocations. - #102823

Closed
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main
Closed

Optimize HttpUtility.UrlPathEncode allocations.#102823
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main

Conversation

@TrayanZapryanov

Copy link
Copy Markdown
Contributor
  1. Omit intermediate string allocations by using Ranges.
  2. Use stackalloc arrays when input is small enough for encoding and ASCII conversions.
  3. Remove string.Contains(' ') loop.

@ghostghost added the area-System.Net label May 29, 2024
@dotnet-policy-servicedotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label May 29, 2024
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

Benchmark:

[MemoryDiagnoser]publicclassHttpUtilityBenchmarks{[Benchmark(Baseline=true)]publicstringUrlTryCreate()=>Uri.TryCreate("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds",UriKind.Absolute,outvaruri)?uri.Authority:"error";[Benchmark]publicstringUrlPathEncode()=>HttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");[Benchmark]publicstringUrlPathEncode_PR()=>MyHttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");}

Results:

MethodMeanErrorStdDevRatioRatioSDGen0AllocatedAlloc Ratio
UrlTryCreate260.7 ns2.41 ns2.01 ns1.000.000.0277232 B1.00
UrlPathEncode338.5 ns6.03 ns5.64 ns1.300.020.0839704 B3.03
UrlPathEncode_PR325.4 ns4.59 ns4.30 ns1.250.020.0658552 B2.38

}

private static byte[] UrlEncodeNonAscii(byte[] bytes, int offset, int count)
private static string UrlEncodeNonAscii(Span<byte> bytes)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods are still allocating quite a bit (the original string even if it's already Ascii, substring and combined string in UrlPathEncodeImpl ...).

The 1 caller of this helper is using Encoding.UTF8 and wants to stop after you see a ?.
Assuming there's sufficient test coverage, I would change this to instead look something like

privatestaticstringUrlPathEncodeImpl(stringvalue){inti=value.AsSpan().IndexOfAnyExceptInRange((char)0x21,(char)0x7F);if(i<0){returnvalue;}intindexOfQuery=value.IndexOf('?');if(indexOfQuery>=0&&indexOfQuery<i){// Everything before the Query is valid ASCIIreturnvalue;}ReadOnlySpan<char>toEncode=indexOfQuery>=0?value.AsSpan(i,indexOfQuery-i):value.AsSpan(i);byte[]bytes=ArrayPool<char>.Shared.Rent(Encoding.UTF8.GetMaxByteCount(toEncode.Length));char[]chars=ArrayPool<char>.Shared.Rent(bytes.Length*3);intutf8Length=Encoding.UTF8.GetBytes(toEncode,bytes);intcharCount=0;foreach(bytebinbytes.AsSpan(0,utf8Length)){if(IsNonAsciiOrSpaceByte(b)){chars[charCount++]='%';chars[charCount++]=HexConverter.ToCharLower(b>>4);chars[charCount++]=HexConverter.ToCharLower(b);}else{chars[charCount++]=(char)b;}}ArrayPool<byte>.Shared.Return(bytes);stringresult=string.Concat(value.AsSpan(0,i),chars.AsSpan(0,charCount),indexOfQuery>=0?value.AsSpan(indexOfQuery):ReadOnlySpan<char>.Empty);ArrayPool<char>.Shared.Return(chars);returnresult;}

This would avoid a bunch of intermediate allocations and transcoding.

Comment on lines +595 to +597
schemeAndAuthority = default(Range);
path = Range.All;
queryAndFragment = default(Range);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
schemeAndAuthority=default(Range);
path=Range.All;
queryAndFragment=default(Range);
returnUrlPathEncodeImpl(value);

Otherwise you would end up allocating a substring for value.AsSpan(path).ToString() + a substring for the string.Concat.

}

return schemeAndAuthority + UrlPathEncodeImpl(path) + queryAndFragment;
return string.Concat(value.AsSpan(schemeAndAuthority), UrlPathEncodeImpl(value.AsSpan(path).ToString()), value.AsSpan(queryAndFragment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be combined into the whole UrlPathEncodeImpl to avoid the substring allocations

// Even if this method returns true, one or more of the out parameters might contain a null or empty string, e.g. if there is no query / fragment.
// Concatenating the pieces back together will form the original input string.
internal static bool TrySplitUriForPathEncode(string input, [NotNullWhen(true)] out string? schemeAndAuthority, [NotNullWhen(true)] out string? path, out string? queryAndFragment)
internal static bool TrySplitUriForPathEncode(string input, out Range schemeAndAuthority, out Range path, out Range queryAndFragment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By forcing everything into spans and ranges, you are now also allocating substrings where you might not have needed to previously. E.g. if pathRange is Range.All, you're reallocating the original string.

You could inline ExtractQueryAndFragment into this method, and always use Substring for the path portion.
I would also suggest returning Span instead of Range as it's easier to use given that the callers only need the range to make the span anyway.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

@MihaZupan Sorry I messed git while trying to rebase after merging previous PRs.
Sorry for the stupid question, but can you tell me how you are updating older branches to latest version without closing existing PR.
Meanwhile I will try to open another PR with your comments integrated.

@MihaZupan

Copy link
Copy Markdown
Member

Assuming upstream is this repo and origin is your fork, my workflow looks something like

git checkout main
git pull upstream
git checkout your-branch
git rebase main // And now fix any conflicts
git push origin -f

@karelzkarelz added this to the 9.0.0 milestone Jun 24, 2024
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Jul 25, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Netcommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TrayanZapryanov@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Optimize HttpUtility.UrlPathEncode allocations. - #102823

Closed
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main
Closed

Optimize HttpUtility.UrlPathEncode allocations.#102823
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main

Conversation

@TrayanZapryanov

Copy link
Copy Markdown
Contributor
  1. Omit intermediate string allocations by using Ranges.
  2. Use stackalloc arrays when input is small enough for encoding and ASCII conversions.
  3. Remove string.Contains(' ') loop.

@ghostghost added the area-System.Net label May 29, 2024
@dotnet-policy-servicedotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label May 29, 2024
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

Benchmark:

[MemoryDiagnoser]publicclassHttpUtilityBenchmarks{[Benchmark(Baseline=true)]publicstringUrlTryCreate()=>Uri.TryCreate("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds",UriKind.Absolute,outvaruri)?uri.Authority:"error";[Benchmark]publicstringUrlPathEncode()=>HttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");[Benchmark]publicstringUrlPathEncode_PR()=>MyHttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");}

Results:

MethodMeanErrorStdDevRatioRatioSDGen0AllocatedAlloc Ratio
UrlTryCreate260.7 ns2.41 ns2.01 ns1.000.000.0277232 B1.00
UrlPathEncode338.5 ns6.03 ns5.64 ns1.300.020.0839704 B3.03
UrlPathEncode_PR325.4 ns4.59 ns4.30 ns1.250.020.0658552 B2.38

}

private static byte[] UrlEncodeNonAscii(byte[] bytes, int offset, int count)
private static string UrlEncodeNonAscii(Span<byte> bytes)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods are still allocating quite a bit (the original string even if it's already Ascii, substring and combined string in UrlPathEncodeImpl ...).

The 1 caller of this helper is using Encoding.UTF8 and wants to stop after you see a ?.
Assuming there's sufficient test coverage, I would change this to instead look something like

privatestaticstringUrlPathEncodeImpl(stringvalue){inti=value.AsSpan().IndexOfAnyExceptInRange((char)0x21,(char)0x7F);if(i<0){returnvalue;}intindexOfQuery=value.IndexOf('?');if(indexOfQuery>=0&&indexOfQuery<i){// Everything before the Query is valid ASCIIreturnvalue;}ReadOnlySpan<char>toEncode=indexOfQuery>=0?value.AsSpan(i,indexOfQuery-i):value.AsSpan(i);byte[]bytes=ArrayPool<char>.Shared.Rent(Encoding.UTF8.GetMaxByteCount(toEncode.Length));char[]chars=ArrayPool<char>.Shared.Rent(bytes.Length*3);intutf8Length=Encoding.UTF8.GetBytes(toEncode,bytes);intcharCount=0;foreach(bytebinbytes.AsSpan(0,utf8Length)){if(IsNonAsciiOrSpaceByte(b)){chars[charCount++]='%';chars[charCount++]=HexConverter.ToCharLower(b>>4);chars[charCount++]=HexConverter.ToCharLower(b);}else{chars[charCount++]=(char)b;}}ArrayPool<byte>.Shared.Return(bytes);stringresult=string.Concat(value.AsSpan(0,i),chars.AsSpan(0,charCount),indexOfQuery>=0?value.AsSpan(indexOfQuery):ReadOnlySpan<char>.Empty);ArrayPool<char>.Shared.Return(chars);returnresult;}

This would avoid a bunch of intermediate allocations and transcoding.

Comment on lines +595 to +597
schemeAndAuthority = default(Range);
path = Range.All;
queryAndFragment = default(Range);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
schemeAndAuthority=default(Range);
path=Range.All;
queryAndFragment=default(Range);
returnUrlPathEncodeImpl(value);

Otherwise you would end up allocating a substring for value.AsSpan(path).ToString() + a substring for the string.Concat.

}

return schemeAndAuthority + UrlPathEncodeImpl(path) + queryAndFragment;
return string.Concat(value.AsSpan(schemeAndAuthority), UrlPathEncodeImpl(value.AsSpan(path).ToString()), value.AsSpan(queryAndFragment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be combined into the whole UrlPathEncodeImpl to avoid the substring allocations

// Even if this method returns true, one or more of the out parameters might contain a null or empty string, e.g. if there is no query / fragment.
// Concatenating the pieces back together will form the original input string.
internal static bool TrySplitUriForPathEncode(string input, [NotNullWhen(true)] out string? schemeAndAuthority, [NotNullWhen(true)] out string? path, out string? queryAndFragment)
internal static bool TrySplitUriForPathEncode(string input, out Range schemeAndAuthority, out Range path, out Range queryAndFragment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By forcing everything into spans and ranges, you are now also allocating substrings where you might not have needed to previously. E.g. if pathRange is Range.All, you're reallocating the original string.

You could inline ExtractQueryAndFragment into this method, and always use Substring for the path portion.
I would also suggest returning Span instead of Range as it's easier to use given that the callers only need the range to make the span anyway.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

@MihaZupan Sorry I messed git while trying to rebase after merging previous PRs.
Sorry for the stupid question, but can you tell me how you are updating older branches to latest version without closing existing PR.
Meanwhile I will try to open another PR with your comments integrated.

@MihaZupan

Copy link
Copy Markdown
Member

Assuming upstream is this repo and origin is your fork, my workflow looks something like

git checkout main
git pull upstream
git checkout your-branch
git rebase main // And now fix any conflicts
git push origin -f

@karelzkarelz added this to the 9.0.0 milestone Jun 24, 2024
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Jul 25, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Netcommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TrayanZapryanov@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Optimize HttpUtility.UrlPathEncode allocations. - #102823

Closed
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main
Closed

Optimize HttpUtility.UrlPathEncode allocations.#102823
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main

Conversation

@TrayanZapryanov

Copy link
Copy Markdown
Contributor
  1. Omit intermediate string allocations by using Ranges.
  2. Use stackalloc arrays when input is small enough for encoding and ASCII conversions.
  3. Remove string.Contains(' ') loop.

@ghostghost added the area-System.Net label May 29, 2024
@dotnet-policy-servicedotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label May 29, 2024
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

Benchmark:

[MemoryDiagnoser]publicclassHttpUtilityBenchmarks{[Benchmark(Baseline=true)]publicstringUrlTryCreate()=>Uri.TryCreate("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds",UriKind.Absolute,outvaruri)?uri.Authority:"error";[Benchmark]publicstringUrlPathEncode()=>HttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");[Benchmark]publicstringUrlPathEncode_PR()=>MyHttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");}

Results:

MethodMeanErrorStdDevRatioRatioSDGen0AllocatedAlloc Ratio
UrlTryCreate260.7 ns2.41 ns2.01 ns1.000.000.0277232 B1.00
UrlPathEncode338.5 ns6.03 ns5.64 ns1.300.020.0839704 B3.03
UrlPathEncode_PR325.4 ns4.59 ns4.30 ns1.250.020.0658552 B2.38

}

private static byte[] UrlEncodeNonAscii(byte[] bytes, int offset, int count)
private static string UrlEncodeNonAscii(Span<byte> bytes)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods are still allocating quite a bit (the original string even if it's already Ascii, substring and combined string in UrlPathEncodeImpl ...).

The 1 caller of this helper is using Encoding.UTF8 and wants to stop after you see a ?.
Assuming there's sufficient test coverage, I would change this to instead look something like

privatestaticstringUrlPathEncodeImpl(stringvalue){inti=value.AsSpan().IndexOfAnyExceptInRange((char)0x21,(char)0x7F);if(i<0){returnvalue;}intindexOfQuery=value.IndexOf('?');if(indexOfQuery>=0&&indexOfQuery<i){// Everything before the Query is valid ASCIIreturnvalue;}ReadOnlySpan<char>toEncode=indexOfQuery>=0?value.AsSpan(i,indexOfQuery-i):value.AsSpan(i);byte[]bytes=ArrayPool<char>.Shared.Rent(Encoding.UTF8.GetMaxByteCount(toEncode.Length));char[]chars=ArrayPool<char>.Shared.Rent(bytes.Length*3);intutf8Length=Encoding.UTF8.GetBytes(toEncode,bytes);intcharCount=0;foreach(bytebinbytes.AsSpan(0,utf8Length)){if(IsNonAsciiOrSpaceByte(b)){chars[charCount++]='%';chars[charCount++]=HexConverter.ToCharLower(b>>4);chars[charCount++]=HexConverter.ToCharLower(b);}else{chars[charCount++]=(char)b;}}ArrayPool<byte>.Shared.Return(bytes);stringresult=string.Concat(value.AsSpan(0,i),chars.AsSpan(0,charCount),indexOfQuery>=0?value.AsSpan(indexOfQuery):ReadOnlySpan<char>.Empty);ArrayPool<char>.Shared.Return(chars);returnresult;}

This would avoid a bunch of intermediate allocations and transcoding.

Comment on lines +595 to +597
schemeAndAuthority = default(Range);
path = Range.All;
queryAndFragment = default(Range);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
schemeAndAuthority=default(Range);
path=Range.All;
queryAndFragment=default(Range);
returnUrlPathEncodeImpl(value);

Otherwise you would end up allocating a substring for value.AsSpan(path).ToString() + a substring for the string.Concat.

}

return schemeAndAuthority + UrlPathEncodeImpl(path) + queryAndFragment;
return string.Concat(value.AsSpan(schemeAndAuthority), UrlPathEncodeImpl(value.AsSpan(path).ToString()), value.AsSpan(queryAndFragment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be combined into the whole UrlPathEncodeImpl to avoid the substring allocations

// Even if this method returns true, one or more of the out parameters might contain a null or empty string, e.g. if there is no query / fragment.
// Concatenating the pieces back together will form the original input string.
internal static bool TrySplitUriForPathEncode(string input, [NotNullWhen(true)] out string? schemeAndAuthority, [NotNullWhen(true)] out string? path, out string? queryAndFragment)
internal static bool TrySplitUriForPathEncode(string input, out Range schemeAndAuthority, out Range path, out Range queryAndFragment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By forcing everything into spans and ranges, you are now also allocating substrings where you might not have needed to previously. E.g. if pathRange is Range.All, you're reallocating the original string.

You could inline ExtractQueryAndFragment into this method, and always use Substring for the path portion.
I would also suggest returning Span instead of Range as it's easier to use given that the callers only need the range to make the span anyway.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

@MihaZupan Sorry I messed git while trying to rebase after merging previous PRs.
Sorry for the stupid question, but can you tell me how you are updating older branches to latest version without closing existing PR.
Meanwhile I will try to open another PR with your comments integrated.

@MihaZupan

Copy link
Copy Markdown
Member

Assuming upstream is this repo and origin is your fork, my workflow looks something like

git checkout main
git pull upstream
git checkout your-branch
git rebase main // And now fix any conflicts
git push origin -f

@karelzkarelz added this to the 9.0.0 milestone Jun 24, 2024
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Jul 25, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Netcommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TrayanZapryanov@MihaZupan@karelz
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Optimize HttpUtility.UrlPathEncode allocations. - #102823

Closed
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main
Closed

Optimize HttpUtility.UrlPathEncode allocations.#102823
TrayanZapryanov wants to merge 0 commit into
dotnet:mainfrom
TrayanZapryanov:main

Conversation

@TrayanZapryanov

Copy link
Copy Markdown
Contributor
  1. Omit intermediate string allocations by using Ranges.
  2. Use stackalloc arrays when input is small enough for encoding and ASCII conversions.
  3. Remove string.Contains(' ') loop.

@ghostghost added the area-System.Net label May 29, 2024
@dotnet-policy-servicedotnet-policy-serviceBot added the community-contribution Indicates that the PR has been added by a community member label May 29, 2024
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

Benchmark:

[MemoryDiagnoser]publicclassHttpUtilityBenchmarks{[Benchmark(Baseline=true)]publicstringUrlTryCreate()=>Uri.TryCreate("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds",UriKind.Absolute,outvaruri)?uri.Authority:"error";[Benchmark]publicstringUrlPathEncode()=>HttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");[Benchmark]publicstringUrlPathEncode_PR()=>MyHttpUtility.UrlPathEncode("http://EXAMPLE.NET/d\\u00E9fault.xxx?sdsd=sds");}

Results:

MethodMeanErrorStdDevRatioRatioSDGen0AllocatedAlloc Ratio
UrlTryCreate260.7 ns2.41 ns2.01 ns1.000.000.0277232 B1.00
UrlPathEncode338.5 ns6.03 ns5.64 ns1.300.020.0839704 B3.03
UrlPathEncode_PR325.4 ns4.59 ns4.30 ns1.250.020.0658552 B2.38

}

private static byte[] UrlEncodeNonAscii(byte[] bytes, int offset, int count)
private static string UrlEncodeNonAscii(Span<byte> bytes)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These methods are still allocating quite a bit (the original string even if it's already Ascii, substring and combined string in UrlPathEncodeImpl ...).

The 1 caller of this helper is using Encoding.UTF8 and wants to stop after you see a ?.
Assuming there's sufficient test coverage, I would change this to instead look something like

privatestaticstringUrlPathEncodeImpl(stringvalue){inti=value.AsSpan().IndexOfAnyExceptInRange((char)0x21,(char)0x7F);if(i<0){returnvalue;}intindexOfQuery=value.IndexOf('?');if(indexOfQuery>=0&&indexOfQuery<i){// Everything before the Query is valid ASCIIreturnvalue;}ReadOnlySpan<char>toEncode=indexOfQuery>=0?value.AsSpan(i,indexOfQuery-i):value.AsSpan(i);byte[]bytes=ArrayPool<char>.Shared.Rent(Encoding.UTF8.GetMaxByteCount(toEncode.Length));char[]chars=ArrayPool<char>.Shared.Rent(bytes.Length*3);intutf8Length=Encoding.UTF8.GetBytes(toEncode,bytes);intcharCount=0;foreach(bytebinbytes.AsSpan(0,utf8Length)){if(IsNonAsciiOrSpaceByte(b)){chars[charCount++]='%';chars[charCount++]=HexConverter.ToCharLower(b>>4);chars[charCount++]=HexConverter.ToCharLower(b);}else{chars[charCount++]=(char)b;}}ArrayPool<byte>.Shared.Return(bytes);stringresult=string.Concat(value.AsSpan(0,i),chars.AsSpan(0,charCount),indexOfQuery>=0?value.AsSpan(indexOfQuery):ReadOnlySpan<char>.Empty);ArrayPool<char>.Shared.Return(chars);returnresult;}

This would avoid a bunch of intermediate allocations and transcoding.

Comment on lines +595 to +597
schemeAndAuthority = default(Range);
path = Range.All;
queryAndFragment = default(Range);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
schemeAndAuthority=default(Range);
path=Range.All;
queryAndFragment=default(Range);
returnUrlPathEncodeImpl(value);

Otherwise you would end up allocating a substring for value.AsSpan(path).ToString() + a substring for the string.Concat.

}

return schemeAndAuthority + UrlPathEncodeImpl(path) + queryAndFragment;
return string.Concat(value.AsSpan(schemeAndAuthority), UrlPathEncodeImpl(value.AsSpan(path).ToString()), value.AsSpan(queryAndFragment));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be combined into the whole UrlPathEncodeImpl to avoid the substring allocations

// Even if this method returns true, one or more of the out parameters might contain a null or empty string, e.g. if there is no query / fragment.
// Concatenating the pieces back together will form the original input string.
internal static bool TrySplitUriForPathEncode(string input, [NotNullWhen(true)] out string? schemeAndAuthority, [NotNullWhen(true)] out string? path, out string? queryAndFragment)
internal static bool TrySplitUriForPathEncode(string input, out Range schemeAndAuthority, out Range path, out Range queryAndFragment)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By forcing everything into spans and ranges, you are now also allocating substrings where you might not have needed to previously. E.g. if pathRange is Range.All, you're reallocating the original string.

You could inline ExtractQueryAndFragment into this method, and always use Substring for the path portion.
I would also suggest returning Span instead of Range as it's easier to use given that the callers only need the range to make the span anyway.

@TrayanZapryanov

Copy link
Copy Markdown
ContributorAuthor

@MihaZupan Sorry I messed git while trying to rebase after merging previous PRs.
Sorry for the stupid question, but can you tell me how you are updating older branches to latest version without closing existing PR.
Meanwhile I will try to open another PR with your comments integrated.

@MihaZupan

Copy link
Copy Markdown
Member

Assuming upstream is this repo and origin is your fork, my workflow looks something like

git checkout main
git pull upstream
git checkout your-branch
git rebase main // And now fix any conflicts
git push origin -f

@karelzkarelz added this to the 9.0.0 milestone Jun 24, 2024
@github-actionsgithub-actionsBot locked and limited conversation to collaborators Jul 25, 2024
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Netcommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@TrayanZapryanov@MihaZupan@karelz