Adding AVX512 path to Base64 encoding/Decoding#92241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tannergooding merged 5 commits into from
Oct 25, 2023
Merged

Adding AVX512 path to Base64 encoding/Decoding #92241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
911b517
Adding AVX512 path to Base64 encoding/Decoding
DeepakRajendrakumaran Sep 18, 2023
7b351ad
Addressing review Comments.
DeepakRajendrakumaran Sep 19, 2023
7fb00e3
Removing fallback path.
DeepakRajendrakumaran Sep 20, 2023
2969ddb
Updating Third Party Notice.
DeepakRajendrakumaran Sep 21, 2023
fc05beb
Addressing review comments
DeepakRajendrakumaran Oct 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions THIRD-PARTY-NOTICES.TXT
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -1297,3 +1297,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

License notice for Avx512Vbmi base64 encoding / decoding
--------------------------------------------------------

Copyright (c) 2015-2018, Wojciech Muła
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------

Aspects of base64 encoding / decoding are based on algorithm described in "Base64 encoding and decoding at almost the speed of a memory
copy", Wojciech Muła and Daniel Lemire. https://arxiv.org/pdf/1910.05109.pdf
85 changes: 84 additions & 1 deletion src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Decoder.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -68,7 +68,18 @@ private static unsafe OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> utf8, Sp

if (maxSrcLength >= 24)
{
byte* end = srcMax - 45;
byte* end = srcMax - 88;
if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
{
Avx512Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

if (src == srcEnd)
{
goto DoneExit;
}
}

end = srcMax - 45;
if (Avx2.IsSupported && (end >= src))
{
Avx2Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -616,6 +627,78 @@ private static OperationStatus DecodeWithWhiteSpaceFromUtf8InPlace(Span<byte> ut
return status;
}

[MethodImpl(MethodImplOptions.AggressiveInlining)]
[CompExactlyDependsOn(typeof(Avx512BW))]
[CompExactlyDependsOn(typeof(Avx512Vbmi))]
private static unsafe void Avx512Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
{
// Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/decode
// If we have AVX512 support, pick off 64 bytes at a time for as long as we can,
// but make sure that we quit before seeing any == markers at the end of the
// string. Also, because we write 16 zeroes at the end of the output, ensure
// that there are at least 22 valid bytes of input data remaining to close the
// gap. 64 + 2 + 22 = 88 bytes.
byte* src = srcBytes;
byte* dest = destBytes;

// The JIT won't hoist these "constants", so help it
Vector512<sbyte> vbmiLookup0 = Vector512.Create(
0x80808080, 0x80808080, 0x80808080, 0x80808080,
0x80808080, 0x80808080, 0x80808080, 0x80808080,
0x80808080, 0x80808080, 0x3e808080, 0x3f808080,
0x37363534, 0x3b3a3938, 0x80803d3c, 0x80808080).AsSByte();
Vector512<sbyte> vbmiLookup1 = Vector512.Create(
0x02010080, 0x06050403, 0x0a090807, 0x0e0d0c0b,
0x1211100f, 0x16151413, 0x80191817, 0x80808080,
0x1c1b1a80, 0x201f1e1d, 0x24232221, 0x28272625,
0x2c2b2a29, 0x302f2e2d, 0x80333231, 0x80808080).AsSByte();
Vector512<byte> vbmiPackedLanesControl = Vector512.Create(
0x06000102, 0x090a0405, 0x0c0d0e08, 0x16101112,
0x191a1415, 0x1c1d1e18, 0x26202122, 0x292a2425,
0x2c2d2e28, 0x36303132, 0x393a3435, 0x3c3d3e38,
0x00000000, 0x00000000, 0x00000000, 0x00000000).AsByte();

Vector512<sbyte> mergeConstant0 = Vector512.Create(0x01400140).AsSByte();
Vector512<short> mergeConstant1 = Vector512.Create(0x00011000).AsInt16();

// This algorithm requires AVX512VBMI support.
// Vbmi was first introduced in CannonLake and is avaialable from IceLake on.
do
{
AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
Vector512<sbyte> str = Vector512.Load(src).AsSByte();

// Step 1: Translate encoded Base64 input to their original indices
// This step also checks for invalid inputs and exits.
// After this, we have indices which are verified to have upper 2 bits set to 0 in each byte.
// origIndex = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
Vector512<sbyte> origIndex = Avx512Vbmi.PermuteVar64x8x2(vbmiLookup0, str, vbmiLookup1);
Vector512<sbyte> errorVec = (origIndex.AsInt32() | str.AsInt32()).AsSByte();
if (errorVec.ExtractMostSignificantBits() != 0)
Comment thread
tannergooding marked this conversation as resolved.
Outdated
{
break;
}

// Step 2: Now we need to reshuffle bits to remove the 0 bits.
// multiAdd1: [...|0000cccc|ccdddddd|0000aaaa|aabbbbbb]
Vector512<short> multiAdd1 = Avx512BW.MultiplyAddAdjacent(origIndex.AsByte(), mergeConstant0);
// multiAdd1: [...|00000000|aaaaaabb|bbbbcccc|ccdddddd]
Vector512<int> multiAdd2 = Avx512BW.MultiplyAddAdjacent(multiAdd1, mergeConstant1);

// Step 3: Pack 48 bytes
str = Avx512Vbmi.PermuteVar64x8(multiAdd2.AsByte(), vbmiPackedLanesControl).AsSByte();

AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
str.Store((sbyte*)dest);
src += 64;
dest += 48;
}
while (src <= srcEnd);

srcBytes = src;
destBytes = dest;
}

[MethodImpl(MethodImplOptions.AggressiveInlining)]
[CompExactlyDependsOn(typeof(Avx2))]
private static unsafe void Avx2Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down
85 changes: 84 additions & 1 deletion src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Encoder.cs
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -67,7 +67,16 @@ public static unsafe OperationStatus EncodeToUtf8(ReadOnlySpan<byte> bytes, Span

if (maxSrcLength >= 16)
{
byte* end = srcMax - 32;
byte* end = srcMax - 64;
if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
{
Avx512Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

if (src == srcEnd)
goto DoneExit;
}

end = srcMax - 64;
if (Avx2.IsSupported && (end >= src))
{
Avx2Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -226,6 +235,80 @@ public static unsafe OperationStatus EncodeToUtf8InPlace(Span<byte> buffer, int
}
}

[MethodImpl(MethodImplOptions.AggressiveInlining)]
[CompExactlyDependsOn(typeof(Avx512BW))]
[CompExactlyDependsOn(typeof(Avx512Vbmi))]
private static unsafe void Avx512Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
{
// Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/encode
// If we have AVX512 support, pick off 48 bytes at a time for as long as we can.
// But because we read 64 bytes at a time, ensure we have enough room to do a
// full 64-byte read without segfaulting.

byte* src = srcBytes;
byte* dest = destBytes;

// The JIT won't hoist these "constants", so help it
Vector512<sbyte> shuffleVecVbmi = Vector512.Create(
0x01020001, 0x04050304, 0x07080607, 0x0a0b090a,
0x0d0e0c0d, 0x10110f10, 0x13141213, 0x16171516,
0x191a1819, 0x1c1d1b1c, 0x1f201e1f, 0x22232122,
0x25262425, 0x28292728, 0x2b2c2a2b, 0x2e2f2d2e).AsSByte();
Vector512<sbyte> vbmiLookup = Vector512.Create("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"u8).AsSByte();

Vector512<ushort> maskAC = Vector512.Create((uint)0x0fc0fc00).AsUInt16();
Vector512<uint> maskBB = Vector512.Create((uint)0x3f003f00);
Vector512<ushort> shiftAC = Vector512.Create((uint)0x0006000a).AsUInt16();
Vector512<ushort> shiftBB = Vector512.Create((uint)0x00080004).AsUInt16();

AssertRead<Vector256<sbyte>>(src, srcStart, sourceLength);

// This algorithm requires AVX512VBMI support.
// Vbmi was first introduced in CannonLake and is avaialable from IceLake on.

// str = [...|PONM|LKJI|HGFE|DCBA]
Vector512<sbyte> str = Vector512.Load(src).AsSByte();

while (true)
Comment thread
MihaZupan marked this conversation as resolved.
Outdated
{
// Step 1 : Split 48 bytes into 64 bytes with each byte using 6-bits from input
// str = [...|KLJK|HIGH|EFDE|BCAB]
str = Avx512Vbmi.PermuteVar64x8(str, shuffleVecVbmi);

// TO-DO- This can be achieved faster with multishift
// Consider the first 4 bytes - BCAB
// temp1 = [...|0000cccc|cc000000|aaaaaa00|00000000]
Vector512<ushort> temp1 = (str.AsUInt16() & maskAC);

// temp2 = [...|00000000|00cccccc|00000000|00aaaaaa]
Vector512<ushort> temp2 = Avx512BW.ShiftRightLogicalVariable(temp1, shiftAC).AsUInt16();

// temp3 = [...|ccdddddd|00000000|aabbbbbb|cccc0000]
Vector512<ushort> temp3 = Avx512BW.ShiftLeftLogicalVariable(str.AsUInt16(), shiftBB).AsUInt16();

// str = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
str = Vector512.ConditionalSelect(maskBB, temp3.AsUInt32(), temp2.AsUInt32()).AsSByte();

// Step 2: Now we have the indices calculated. Next step is to use these indices to translate.
str = Avx512Vbmi.PermuteVar64x8(vbmiLookup, str);

AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
str.Store((sbyte*)dest);

src += 48;
dest += 64;

if (src > srcEnd)
break;

AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
str = Vector512.Load(src).AsSByte();
}

srcBytes = src;
destBytes = dest;
}

[MethodImpl(MethodImplOptions.AggressiveInlining)]
[CompExactlyDependsOn(typeof(Avx2))]
private static unsafe void Avx2Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Adding AVX512 path to Base64 encoding/Decoding#92241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

tannergooding
merged 5 commits into

from

Oct 25, 2023
 Merged


Adding AVX512 path to Base64 encoding/Decoding
#92241
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
911b517

Adding AVX512 path to Base64 encoding/Decoding


DeepakRajendrakumaran Sep 18, 2023
7b351ad

Addressing review Comments.


DeepakRajendrakumaran Sep 19, 2023
7fb00e3

Removing fallback path.


DeepakRajendrakumaran Sep 20, 2023
2969ddb

Updating Third Party Notice.


DeepakRajendrakumaran Sep 21, 2023
fc05beb

Addressing review comments


DeepakRajendrakumaran Oct 24, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


34 changes: 34 additions & 0 deletions
THIRD-PARTY-NOTICES.TXT

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1297,3 +1297,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

License notice for Avx512Vbmi base64 encoding / decoding
--------------------------------------------------------

Copyright (c) 2015-2018, Wojciech Muła
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------

Aspects of base64 encoding / decoding are based on algorithm described in "Base64 encoding and decoding at almost the speed of a memory
copy", Wojciech Muła and Daniel Lemire. https://arxiv.org/pdf/1910.05109.pdf

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Decoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -68,7 +68,18 @@ private static unsafe OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> utf8, Sp

 if (maxSrcLength >= 24)
 {
 byte* end = srcMax - 45;
 byte* end = srcMax - 88;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 {
 goto DoneExit;
 }
 }

 end = srcMax - 45;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -616,6 +627,78 @@ private static OperationStatus DecodeWithWhiteSpaceFromUtf8InPlace(Span<byte> ut
 return status;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/decode
 // If we have AVX512 support, pick off 64 bytes at a time for as long as we can,
 // but make sure that we quit before seeing any == markers at the end of the
 // string. Also, because we write 16 zeroes at the end of the output, ensure
 // that there are at least 22 valid bytes of input data remaining to close the
 // gap. 64 + 2 + 22 = 88 bytes.
 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> vbmiLookup0 = Vector512.Create(
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x3e808080, 0x3f808080,
 0x37363534, 0x3b3a3938, 0x80803d3c, 0x80808080).AsSByte();
 Vector512<sbyte> vbmiLookup1 = Vector512.Create(
 0x02010080, 0x06050403, 0x0a090807, 0x0e0d0c0b,
 0x1211100f, 0x16151413, 0x80191817, 0x80808080,
 0x1c1b1a80, 0x201f1e1d, 0x24232221, 0x28272625,
 0x2c2b2a29, 0x302f2e2d, 0x80333231, 0x80808080).AsSByte();
 Vector512<byte> vbmiPackedLanesControl = Vector512.Create(
 0x06000102, 0x090a0405, 0x0c0d0e08, 0x16101112,
 0x191a1415, 0x1c1d1e18, 0x26202122, 0x292a2425,
 0x2c2d2e28, 0x36303132, 0x393a3435, 0x3c3d3e38,
 0x00000000, 0x00000000, 0x00000000, 0x00000000).AsByte();

 Vector512<sbyte> mergeConstant0 = Vector512.Create(0x01400140).AsSByte();
 Vector512<short> mergeConstant1 = Vector512.Create(0x00011000).AsInt16();

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.
 do
 {
 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 // Step 1: Translate encoded Base64 input to their original indices
 // This step also checks for invalid inputs and exits.
 // After this, we have indices which are verified to have upper 2 bits set to 0 in each byte.
 // origIndex = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 Vector512<sbyte> origIndex = Avx512Vbmi.PermuteVar64x8x2(vbmiLookup0, str, vbmiLookup1);
 Vector512<sbyte> errorVec = (origIndex.AsInt32() | str.AsInt32()).AsSByte();
 if (errorVec.ExtractMostSignificantBits() != 0)
Comment thread
tannergooding marked this conversation as resolved.


Outdated

 {
 break;
 }

 // Step 2: Now we need to reshuffle bits to remove the 0 bits.
 // multiAdd1: [...|0000cccc|ccdddddd|0000aaaa|aabbbbbb]
 Vector512<short> multiAdd1 = Avx512BW.MultiplyAddAdjacent(origIndex.AsByte(), mergeConstant0);
 // multiAdd1: [...|00000000|aaaaaabb|bbbbcccc|ccdddddd]
 Vector512<int> multiAdd2 = Avx512BW.MultiplyAddAdjacent(multiAdd1, mergeConstant1);

 // Step 3: Pack 48 bytes
 str = Avx512Vbmi.PermuteVar64x8(multiAdd2.AsByte(), vbmiPackedLanesControl).AsSByte();

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);
 src += 64;
 dest += 48;
 }
 while (src <= srcEnd);

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Encoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -67,7 +67,16 @@ public static unsafe OperationStatus EncodeToUtf8(ReadOnlySpan<byte> bytes, Span

 if (maxSrcLength >= 16)
 {
 byte* end = srcMax - 32;
 byte* end = srcMax - 64;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 goto DoneExit;
 }

 end = srcMax - 64;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -226,6 +235,80 @@ public static unsafe OperationStatus EncodeToUtf8InPlace(Span<byte> buffer, int
 }
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/encode
 // If we have AVX512 support, pick off 48 bytes at a time for as long as we can.
 // But because we read 64 bytes at a time, ensure we have enough room to do a
 // full 64-byte read without segfaulting.

 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> shuffleVecVbmi = Vector512.Create(
 0x01020001, 0x04050304, 0x07080607, 0x0a0b090a,
 0x0d0e0c0d, 0x10110f10, 0x13141213, 0x16171516,
 0x191a1819, 0x1c1d1b1c, 0x1f201e1f, 0x22232122,
 0x25262425, 0x28292728, 0x2b2c2a2b, 0x2e2f2d2e).AsSByte();
 Vector512<sbyte> vbmiLookup = Vector512.Create("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"u8).AsSByte();

 Vector512<ushort> maskAC = Vector512.Create((uint)0x0fc0fc00).AsUInt16();
 Vector512<uint> maskBB = Vector512.Create((uint)0x3f003f00);
 Vector512<ushort> shiftAC = Vector512.Create((uint)0x0006000a).AsUInt16();
 Vector512<ushort> shiftBB = Vector512.Create((uint)0x00080004).AsUInt16();

 AssertRead<Vector256<sbyte>>(src, srcStart, sourceLength);

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.

 // str = [...|PONM|LKJI|HGFE|DCBA]
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 while (true)
Comment thread
MihaZupan marked this conversation as resolved.


Outdated

 {
 // Step 1 : Split 48 bytes into 64 bytes with each byte using 6-bits from input
 // str = [...|KLJK|HIGH|EFDE|BCAB]
 str = Avx512Vbmi.PermuteVar64x8(str, shuffleVecVbmi);

 // TO-DO- This can be achieved faster with multishift
 // Consider the first 4 bytes - BCAB
 // temp1 = [...|0000cccc|cc000000|aaaaaa00|00000000]
 Vector512<ushort> temp1 = (str.AsUInt16() & maskAC);

 // temp2 = [...|00000000|00cccccc|00000000|00aaaaaa]
 Vector512<ushort> temp2 = Avx512BW.ShiftRightLogicalVariable(temp1, shiftAC).AsUInt16();

 // temp3 = [...|ccdddddd|00000000|aabbbbbb|cccc0000]
 Vector512<ushort> temp3 = Avx512BW.ShiftLeftLogicalVariable(str.AsUInt16(), shiftBB).AsUInt16();

 // str = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 str = Vector512.ConditionalSelect(maskBB, temp3.AsUInt32(), temp2.AsUInt32()).AsSByte();

 // Step 2: Now we have the indices calculated. Next step is to use these indices to translate.
 str = Avx512Vbmi.PermuteVar64x8(vbmiLookup, str);

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);

 src += 48;
 dest += 64;

 if (src > srcEnd)
 break;

 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 str = Vector512.Load(src).AsSByte();
 }

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Adding AVX512 path to Base64 encoding/Decoding#92241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

tannergooding
merged 5 commits into

from

Oct 25, 2023
 Merged


Adding AVX512 path to Base64 encoding/Decoding
#92241
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
911b517

Adding AVX512 path to Base64 encoding/Decoding


DeepakRajendrakumaran Sep 18, 2023
7b351ad

Addressing review Comments.


DeepakRajendrakumaran Sep 19, 2023
7fb00e3

Removing fallback path.


DeepakRajendrakumaran Sep 20, 2023
2969ddb

Updating Third Party Notice.


DeepakRajendrakumaran Sep 21, 2023
fc05beb

Addressing review comments


DeepakRajendrakumaran Oct 24, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


34 changes: 34 additions & 0 deletions
THIRD-PARTY-NOTICES.TXT

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1297,3 +1297,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

License notice for Avx512Vbmi base64 encoding / decoding
--------------------------------------------------------

Copyright (c) 2015-2018, Wojciech Muła
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------

Aspects of base64 encoding / decoding are based on algorithm described in "Base64 encoding and decoding at almost the speed of a memory
copy", Wojciech Muła and Daniel Lemire. https://arxiv.org/pdf/1910.05109.pdf

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Decoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -68,7 +68,18 @@ private static unsafe OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> utf8, Sp

 if (maxSrcLength >= 24)
 {
 byte* end = srcMax - 45;
 byte* end = srcMax - 88;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 {
 goto DoneExit;
 }
 }

 end = srcMax - 45;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -616,6 +627,78 @@ private static OperationStatus DecodeWithWhiteSpaceFromUtf8InPlace(Span<byte> ut
 return status;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/decode
 // If we have AVX512 support, pick off 64 bytes at a time for as long as we can,
 // but make sure that we quit before seeing any == markers at the end of the
 // string. Also, because we write 16 zeroes at the end of the output, ensure
 // that there are at least 22 valid bytes of input data remaining to close the
 // gap. 64 + 2 + 22 = 88 bytes.
 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> vbmiLookup0 = Vector512.Create(
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x3e808080, 0x3f808080,
 0x37363534, 0x3b3a3938, 0x80803d3c, 0x80808080).AsSByte();
 Vector512<sbyte> vbmiLookup1 = Vector512.Create(
 0x02010080, 0x06050403, 0x0a090807, 0x0e0d0c0b,
 0x1211100f, 0x16151413, 0x80191817, 0x80808080,
 0x1c1b1a80, 0x201f1e1d, 0x24232221, 0x28272625,
 0x2c2b2a29, 0x302f2e2d, 0x80333231, 0x80808080).AsSByte();
 Vector512<byte> vbmiPackedLanesControl = Vector512.Create(
 0x06000102, 0x090a0405, 0x0c0d0e08, 0x16101112,
 0x191a1415, 0x1c1d1e18, 0x26202122, 0x292a2425,
 0x2c2d2e28, 0x36303132, 0x393a3435, 0x3c3d3e38,
 0x00000000, 0x00000000, 0x00000000, 0x00000000).AsByte();

 Vector512<sbyte> mergeConstant0 = Vector512.Create(0x01400140).AsSByte();
 Vector512<short> mergeConstant1 = Vector512.Create(0x00011000).AsInt16();

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.
 do
 {
 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 // Step 1: Translate encoded Base64 input to their original indices
 // This step also checks for invalid inputs and exits.
 // After this, we have indices which are verified to have upper 2 bits set to 0 in each byte.
 // origIndex = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 Vector512<sbyte> origIndex = Avx512Vbmi.PermuteVar64x8x2(vbmiLookup0, str, vbmiLookup1);
 Vector512<sbyte> errorVec = (origIndex.AsInt32() | str.AsInt32()).AsSByte();
 if (errorVec.ExtractMostSignificantBits() != 0)
Comment thread
tannergooding marked this conversation as resolved.


Outdated

 {
 break;
 }

 // Step 2: Now we need to reshuffle bits to remove the 0 bits.
 // multiAdd1: [...|0000cccc|ccdddddd|0000aaaa|aabbbbbb]
 Vector512<short> multiAdd1 = Avx512BW.MultiplyAddAdjacent(origIndex.AsByte(), mergeConstant0);
 // multiAdd1: [...|00000000|aaaaaabb|bbbbcccc|ccdddddd]
 Vector512<int> multiAdd2 = Avx512BW.MultiplyAddAdjacent(multiAdd1, mergeConstant1);

 // Step 3: Pack 48 bytes
 str = Avx512Vbmi.PermuteVar64x8(multiAdd2.AsByte(), vbmiPackedLanesControl).AsSByte();

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);
 src += 64;
 dest += 48;
 }
 while (src <= srcEnd);

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Encoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -67,7 +67,16 @@ public static unsafe OperationStatus EncodeToUtf8(ReadOnlySpan<byte> bytes, Span

 if (maxSrcLength >= 16)
 {
 byte* end = srcMax - 32;
 byte* end = srcMax - 64;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 goto DoneExit;
 }

 end = srcMax - 64;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -226,6 +235,80 @@ public static unsafe OperationStatus EncodeToUtf8InPlace(Span<byte> buffer, int
 }
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/encode
 // If we have AVX512 support, pick off 48 bytes at a time for as long as we can.
 // But because we read 64 bytes at a time, ensure we have enough room to do a
 // full 64-byte read without segfaulting.

 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> shuffleVecVbmi = Vector512.Create(
 0x01020001, 0x04050304, 0x07080607, 0x0a0b090a,
 0x0d0e0c0d, 0x10110f10, 0x13141213, 0x16171516,
 0x191a1819, 0x1c1d1b1c, 0x1f201e1f, 0x22232122,
 0x25262425, 0x28292728, 0x2b2c2a2b, 0x2e2f2d2e).AsSByte();
 Vector512<sbyte> vbmiLookup = Vector512.Create("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"u8).AsSByte();

 Vector512<ushort> maskAC = Vector512.Create((uint)0x0fc0fc00).AsUInt16();
 Vector512<uint> maskBB = Vector512.Create((uint)0x3f003f00);
 Vector512<ushort> shiftAC = Vector512.Create((uint)0x0006000a).AsUInt16();
 Vector512<ushort> shiftBB = Vector512.Create((uint)0x00080004).AsUInt16();

 AssertRead<Vector256<sbyte>>(src, srcStart, sourceLength);

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.

 // str = [...|PONM|LKJI|HGFE|DCBA]
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 while (true)
Comment thread
MihaZupan marked this conversation as resolved.


Outdated

 {
 // Step 1 : Split 48 bytes into 64 bytes with each byte using 6-bits from input
 // str = [...|KLJK|HIGH|EFDE|BCAB]
 str = Avx512Vbmi.PermuteVar64x8(str, shuffleVecVbmi);

 // TO-DO- This can be achieved faster with multishift
 // Consider the first 4 bytes - BCAB
 // temp1 = [...|0000cccc|cc000000|aaaaaa00|00000000]
 Vector512<ushort> temp1 = (str.AsUInt16() & maskAC);

 // temp2 = [...|00000000|00cccccc|00000000|00aaaaaa]
 Vector512<ushort> temp2 = Avx512BW.ShiftRightLogicalVariable(temp1, shiftAC).AsUInt16();

 // temp3 = [...|ccdddddd|00000000|aabbbbbb|cccc0000]
 Vector512<ushort> temp3 = Avx512BW.ShiftLeftLogicalVariable(str.AsUInt16(), shiftBB).AsUInt16();

 // str = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 str = Vector512.ConditionalSelect(maskBB, temp3.AsUInt32(), temp2.AsUInt32()).AsSByte();

 // Step 2: Now we have the indices calculated. Next step is to use these indices to translate.
 str = Avx512Vbmi.PermuteVar64x8(vbmiLookup, str);

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);

 src += 48;
 dest += 64;

 if (src > srcEnd)
 break;

 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 str = Vector512.Load(src).AsSByte();
 }

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Adding AVX512 path to Base64 encoding/Decoding#92241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

tannergooding
merged 5 commits into

from

Oct 25, 2023
 Merged


Adding AVX512 path to Base64 encoding/Decoding
#92241
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
911b517

Adding AVX512 path to Base64 encoding/Decoding


DeepakRajendrakumaran Sep 18, 2023
7b351ad

Addressing review Comments.


DeepakRajendrakumaran Sep 19, 2023
7fb00e3

Removing fallback path.


DeepakRajendrakumaran Sep 20, 2023
2969ddb

Updating Third Party Notice.


DeepakRajendrakumaran Sep 21, 2023
fc05beb

Addressing review comments


DeepakRajendrakumaran Oct 24, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


34 changes: 34 additions & 0 deletions
THIRD-PARTY-NOTICES.TXT

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1297,3 +1297,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

License notice for Avx512Vbmi base64 encoding / decoding
--------------------------------------------------------

Copyright (c) 2015-2018, Wojciech Muła
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------

Aspects of base64 encoding / decoding are based on algorithm described in "Base64 encoding and decoding at almost the speed of a memory
copy", Wojciech Muła and Daniel Lemire. https://arxiv.org/pdf/1910.05109.pdf

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Decoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -68,7 +68,18 @@ private static unsafe OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> utf8, Sp

 if (maxSrcLength >= 24)
 {
 byte* end = srcMax - 45;
 byte* end = srcMax - 88;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 {
 goto DoneExit;
 }
 }

 end = srcMax - 45;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -616,6 +627,78 @@ private static OperationStatus DecodeWithWhiteSpaceFromUtf8InPlace(Span<byte> ut
 return status;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/decode
 // If we have AVX512 support, pick off 64 bytes at a time for as long as we can,
 // but make sure that we quit before seeing any == markers at the end of the
 // string. Also, because we write 16 zeroes at the end of the output, ensure
 // that there are at least 22 valid bytes of input data remaining to close the
 // gap. 64 + 2 + 22 = 88 bytes.
 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> vbmiLookup0 = Vector512.Create(
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x3e808080, 0x3f808080,
 0x37363534, 0x3b3a3938, 0x80803d3c, 0x80808080).AsSByte();
 Vector512<sbyte> vbmiLookup1 = Vector512.Create(
 0x02010080, 0x06050403, 0x0a090807, 0x0e0d0c0b,
 0x1211100f, 0x16151413, 0x80191817, 0x80808080,
 0x1c1b1a80, 0x201f1e1d, 0x24232221, 0x28272625,
 0x2c2b2a29, 0x302f2e2d, 0x80333231, 0x80808080).AsSByte();
 Vector512<byte> vbmiPackedLanesControl = Vector512.Create(
 0x06000102, 0x090a0405, 0x0c0d0e08, 0x16101112,
 0x191a1415, 0x1c1d1e18, 0x26202122, 0x292a2425,
 0x2c2d2e28, 0x36303132, 0x393a3435, 0x3c3d3e38,
 0x00000000, 0x00000000, 0x00000000, 0x00000000).AsByte();

 Vector512<sbyte> mergeConstant0 = Vector512.Create(0x01400140).AsSByte();
 Vector512<short> mergeConstant1 = Vector512.Create(0x00011000).AsInt16();

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.
 do
 {
 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 // Step 1: Translate encoded Base64 input to their original indices
 // This step also checks for invalid inputs and exits.
 // After this, we have indices which are verified to have upper 2 bits set to 0 in each byte.
 // origIndex = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 Vector512<sbyte> origIndex = Avx512Vbmi.PermuteVar64x8x2(vbmiLookup0, str, vbmiLookup1);
 Vector512<sbyte> errorVec = (origIndex.AsInt32() | str.AsInt32()).AsSByte();
 if (errorVec.ExtractMostSignificantBits() != 0)
Comment thread
tannergooding marked this conversation as resolved.


Outdated

 {
 break;
 }

 // Step 2: Now we need to reshuffle bits to remove the 0 bits.
 // multiAdd1: [...|0000cccc|ccdddddd|0000aaaa|aabbbbbb]
 Vector512<short> multiAdd1 = Avx512BW.MultiplyAddAdjacent(origIndex.AsByte(), mergeConstant0);
 // multiAdd1: [...|00000000|aaaaaabb|bbbbcccc|ccdddddd]
 Vector512<int> multiAdd2 = Avx512BW.MultiplyAddAdjacent(multiAdd1, mergeConstant1);

 // Step 3: Pack 48 bytes
 str = Avx512Vbmi.PermuteVar64x8(multiAdd2.AsByte(), vbmiPackedLanesControl).AsSByte();

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);
 src += 64;
 dest += 48;
 }
 while (src <= srcEnd);

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Encoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -67,7 +67,16 @@ public static unsafe OperationStatus EncodeToUtf8(ReadOnlySpan<byte> bytes, Span

 if (maxSrcLength >= 16)
 {
 byte* end = srcMax - 32;
 byte* end = srcMax - 64;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 goto DoneExit;
 }

 end = srcMax - 64;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -226,6 +235,80 @@ public static unsafe OperationStatus EncodeToUtf8InPlace(Span<byte> buffer, int
 }
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/encode
 // If we have AVX512 support, pick off 48 bytes at a time for as long as we can.
 // But because we read 64 bytes at a time, ensure we have enough room to do a
 // full 64-byte read without segfaulting.

 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> shuffleVecVbmi = Vector512.Create(
 0x01020001, 0x04050304, 0x07080607, 0x0a0b090a,
 0x0d0e0c0d, 0x10110f10, 0x13141213, 0x16171516,
 0x191a1819, 0x1c1d1b1c, 0x1f201e1f, 0x22232122,
 0x25262425, 0x28292728, 0x2b2c2a2b, 0x2e2f2d2e).AsSByte();
 Vector512<sbyte> vbmiLookup = Vector512.Create("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"u8).AsSByte();

 Vector512<ushort> maskAC = Vector512.Create((uint)0x0fc0fc00).AsUInt16();
 Vector512<uint> maskBB = Vector512.Create((uint)0x3f003f00);
 Vector512<ushort> shiftAC = Vector512.Create((uint)0x0006000a).AsUInt16();
 Vector512<ushort> shiftBB = Vector512.Create((uint)0x00080004).AsUInt16();

 AssertRead<Vector256<sbyte>>(src, srcStart, sourceLength);

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.

 // str = [...|PONM|LKJI|HGFE|DCBA]
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 while (true)
Comment thread
MihaZupan marked this conversation as resolved.


Outdated

 {
 // Step 1 : Split 48 bytes into 64 bytes with each byte using 6-bits from input
 // str = [...|KLJK|HIGH|EFDE|BCAB]
 str = Avx512Vbmi.PermuteVar64x8(str, shuffleVecVbmi);

 // TO-DO- This can be achieved faster with multishift
 // Consider the first 4 bytes - BCAB
 // temp1 = [...|0000cccc|cc000000|aaaaaa00|00000000]
 Vector512<ushort> temp1 = (str.AsUInt16() & maskAC);

 // temp2 = [...|00000000|00cccccc|00000000|00aaaaaa]
 Vector512<ushort> temp2 = Avx512BW.ShiftRightLogicalVariable(temp1, shiftAC).AsUInt16();

 // temp3 = [...|ccdddddd|00000000|aabbbbbb|cccc0000]
 Vector512<ushort> temp3 = Avx512BW.ShiftLeftLogicalVariable(str.AsUInt16(), shiftBB).AsUInt16();

 // str = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 str = Vector512.ConditionalSelect(maskBB, temp3.AsUInt32(), temp2.AsUInt32()).AsSByte();

 // Step 2: Now we have the indices calculated. Next step is to use these indices to translate.
 str = Avx512Vbmi.PermuteVar64x8(vbmiLookup, str);

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);

 src += 48;
 dest += 64;

 if (src > srcEnd)
 break;

 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 str = Vector512.Load(src).AsSByte();
 }

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Adding AVX512 path to Base64 encoding/Decoding#92241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

tannergooding
merged 5 commits into

from

Oct 25, 2023
 Merged


Adding AVX512 path to Base64 encoding/Decoding
#92241
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
911b517

Adding AVX512 path to Base64 encoding/Decoding


DeepakRajendrakumaran Sep 18, 2023
7b351ad

Addressing review Comments.


DeepakRajendrakumaran Sep 19, 2023
7fb00e3

Removing fallback path.


DeepakRajendrakumaran Sep 20, 2023
2969ddb

Updating Third Party Notice.


DeepakRajendrakumaran Sep 21, 2023
fc05beb

Addressing review comments


DeepakRajendrakumaran Oct 24, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


34 changes: 34 additions & 0 deletions
THIRD-PARTY-NOTICES.TXT

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1297,3 +1297,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

License notice for Avx512Vbmi base64 encoding / decoding
--------------------------------------------------------

Copyright (c) 2015-2018, Wojciech Muła
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------

Aspects of base64 encoding / decoding are based on algorithm described in "Base64 encoding and decoding at almost the speed of a memory
copy", Wojciech Muła and Daniel Lemire. https://arxiv.org/pdf/1910.05109.pdf

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Decoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -68,7 +68,18 @@ private static unsafe OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> utf8, Sp

 if (maxSrcLength >= 24)
 {
 byte* end = srcMax - 45;
 byte* end = srcMax - 88;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 {
 goto DoneExit;
 }
 }

 end = srcMax - 45;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -616,6 +627,78 @@ private static OperationStatus DecodeWithWhiteSpaceFromUtf8InPlace(Span<byte> ut
 return status;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/decode
 // If we have AVX512 support, pick off 64 bytes at a time for as long as we can,
 // but make sure that we quit before seeing any == markers at the end of the
 // string. Also, because we write 16 zeroes at the end of the output, ensure
 // that there are at least 22 valid bytes of input data remaining to close the
 // gap. 64 + 2 + 22 = 88 bytes.
 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> vbmiLookup0 = Vector512.Create(
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x3e808080, 0x3f808080,
 0x37363534, 0x3b3a3938, 0x80803d3c, 0x80808080).AsSByte();
 Vector512<sbyte> vbmiLookup1 = Vector512.Create(
 0x02010080, 0x06050403, 0x0a090807, 0x0e0d0c0b,
 0x1211100f, 0x16151413, 0x80191817, 0x80808080,
 0x1c1b1a80, 0x201f1e1d, 0x24232221, 0x28272625,
 0x2c2b2a29, 0x302f2e2d, 0x80333231, 0x80808080).AsSByte();
 Vector512<byte> vbmiPackedLanesControl = Vector512.Create(
 0x06000102, 0x090a0405, 0x0c0d0e08, 0x16101112,
 0x191a1415, 0x1c1d1e18, 0x26202122, 0x292a2425,
 0x2c2d2e28, 0x36303132, 0x393a3435, 0x3c3d3e38,
 0x00000000, 0x00000000, 0x00000000, 0x00000000).AsByte();

 Vector512<sbyte> mergeConstant0 = Vector512.Create(0x01400140).AsSByte();
 Vector512<short> mergeConstant1 = Vector512.Create(0x00011000).AsInt16();

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.
 do
 {
 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 // Step 1: Translate encoded Base64 input to their original indices
 // This step also checks for invalid inputs and exits.
 // After this, we have indices which are verified to have upper 2 bits set to 0 in each byte.
 // origIndex = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 Vector512<sbyte> origIndex = Avx512Vbmi.PermuteVar64x8x2(vbmiLookup0, str, vbmiLookup1);
 Vector512<sbyte> errorVec = (origIndex.AsInt32() | str.AsInt32()).AsSByte();
 if (errorVec.ExtractMostSignificantBits() != 0)
Comment thread
tannergooding marked this conversation as resolved.


Outdated

 {
 break;
 }

 // Step 2: Now we need to reshuffle bits to remove the 0 bits.
 // multiAdd1: [...|0000cccc|ccdddddd|0000aaaa|aabbbbbb]
 Vector512<short> multiAdd1 = Avx512BW.MultiplyAddAdjacent(origIndex.AsByte(), mergeConstant0);
 // multiAdd1: [...|00000000|aaaaaabb|bbbbcccc|ccdddddd]
 Vector512<int> multiAdd2 = Avx512BW.MultiplyAddAdjacent(multiAdd1, mergeConstant1);

 // Step 3: Pack 48 bytes
 str = Avx512Vbmi.PermuteVar64x8(multiAdd2.AsByte(), vbmiPackedLanesControl).AsSByte();

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);
 src += 64;
 dest += 48;
 }
 while (src <= srcEnd);

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Encoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -67,7 +67,16 @@ public static unsafe OperationStatus EncodeToUtf8(ReadOnlySpan<byte> bytes, Span

 if (maxSrcLength >= 16)
 {
 byte* end = srcMax - 32;
 byte* end = srcMax - 64;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 goto DoneExit;
 }

 end = srcMax - 64;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -226,6 +235,80 @@ public static unsafe OperationStatus EncodeToUtf8InPlace(Span<byte> buffer, int
 }
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/encode
 // If we have AVX512 support, pick off 48 bytes at a time for as long as we can.
 // But because we read 64 bytes at a time, ensure we have enough room to do a
 // full 64-byte read without segfaulting.

 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> shuffleVecVbmi = Vector512.Create(
 0x01020001, 0x04050304, 0x07080607, 0x0a0b090a,
 0x0d0e0c0d, 0x10110f10, 0x13141213, 0x16171516,
 0x191a1819, 0x1c1d1b1c, 0x1f201e1f, 0x22232122,
 0x25262425, 0x28292728, 0x2b2c2a2b, 0x2e2f2d2e).AsSByte();
 Vector512<sbyte> vbmiLookup = Vector512.Create("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"u8).AsSByte();

 Vector512<ushort> maskAC = Vector512.Create((uint)0x0fc0fc00).AsUInt16();
 Vector512<uint> maskBB = Vector512.Create((uint)0x3f003f00);
 Vector512<ushort> shiftAC = Vector512.Create((uint)0x0006000a).AsUInt16();
 Vector512<ushort> shiftBB = Vector512.Create((uint)0x00080004).AsUInt16();

 AssertRead<Vector256<sbyte>>(src, srcStart, sourceLength);

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.

 // str = [...|PONM|LKJI|HGFE|DCBA]
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 while (true)
Comment thread
MihaZupan marked this conversation as resolved.


Outdated

 {
 // Step 1 : Split 48 bytes into 64 bytes with each byte using 6-bits from input
 // str = [...|KLJK|HIGH|EFDE|BCAB]
 str = Avx512Vbmi.PermuteVar64x8(str, shuffleVecVbmi);

 // TO-DO- This can be achieved faster with multishift
 // Consider the first 4 bytes - BCAB
 // temp1 = [...|0000cccc|cc000000|aaaaaa00|00000000]
 Vector512<ushort> temp1 = (str.AsUInt16() & maskAC);

 // temp2 = [...|00000000|00cccccc|00000000|00aaaaaa]
 Vector512<ushort> temp2 = Avx512BW.ShiftRightLogicalVariable(temp1, shiftAC).AsUInt16();

 // temp3 = [...|ccdddddd|00000000|aabbbbbb|cccc0000]
 Vector512<ushort> temp3 = Avx512BW.ShiftLeftLogicalVariable(str.AsUInt16(), shiftBB).AsUInt16();

 // str = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 str = Vector512.ConditionalSelect(maskBB, temp3.AsUInt32(), temp2.AsUInt32()).AsSByte();

 // Step 2: Now we have the indices calculated. Next step is to use these indices to translate.
 str = Avx512Vbmi.PermuteVar64x8(vbmiLookup, str);

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);

 src += 48;
 dest += 64;

 if (src > srcEnd)
 break;

 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 str = Vector512.Load(src).AsSByte();
 }

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Adding AVX512 path to Base64 encoding/Decoding#92241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

tannergooding
merged 5 commits into

from

Oct 25, 2023
 Merged


Adding AVX512 path to Base64 encoding/Decoding
#92241
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
911b517

Adding AVX512 path to Base64 encoding/Decoding


DeepakRajendrakumaran Sep 18, 2023
7b351ad

Addressing review Comments.


DeepakRajendrakumaran Sep 19, 2023
7fb00e3

Removing fallback path.


DeepakRajendrakumaran Sep 20, 2023
2969ddb

Updating Third Party Notice.


DeepakRajendrakumaran Sep 21, 2023
fc05beb

Addressing review comments


DeepakRajendrakumaran Oct 24, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


34 changes: 34 additions & 0 deletions
THIRD-PARTY-NOTICES.TXT

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1297,3 +1297,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

License notice for Avx512Vbmi base64 encoding / decoding
--------------------------------------------------------

Copyright (c) 2015-2018, Wojciech Muła
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------

Aspects of base64 encoding / decoding are based on algorithm described in "Base64 encoding and decoding at almost the speed of a memory
copy", Wojciech Muła and Daniel Lemire. https://arxiv.org/pdf/1910.05109.pdf

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Decoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -68,7 +68,18 @@ private static unsafe OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> utf8, Sp

 if (maxSrcLength >= 24)
 {
 byte* end = srcMax - 45;
 byte* end = srcMax - 88;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 {
 goto DoneExit;
 }
 }

 end = srcMax - 45;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -616,6 +627,78 @@ private static OperationStatus DecodeWithWhiteSpaceFromUtf8InPlace(Span<byte> ut
 return status;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/decode
 // If we have AVX512 support, pick off 64 bytes at a time for as long as we can,
 // but make sure that we quit before seeing any == markers at the end of the
 // string. Also, because we write 16 zeroes at the end of the output, ensure
 // that there are at least 22 valid bytes of input data remaining to close the
 // gap. 64 + 2 + 22 = 88 bytes.
 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> vbmiLookup0 = Vector512.Create(
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x3e808080, 0x3f808080,
 0x37363534, 0x3b3a3938, 0x80803d3c, 0x80808080).AsSByte();
 Vector512<sbyte> vbmiLookup1 = Vector512.Create(
 0x02010080, 0x06050403, 0x0a090807, 0x0e0d0c0b,
 0x1211100f, 0x16151413, 0x80191817, 0x80808080,
 0x1c1b1a80, 0x201f1e1d, 0x24232221, 0x28272625,
 0x2c2b2a29, 0x302f2e2d, 0x80333231, 0x80808080).AsSByte();
 Vector512<byte> vbmiPackedLanesControl = Vector512.Create(
 0x06000102, 0x090a0405, 0x0c0d0e08, 0x16101112,
 0x191a1415, 0x1c1d1e18, 0x26202122, 0x292a2425,
 0x2c2d2e28, 0x36303132, 0x393a3435, 0x3c3d3e38,
 0x00000000, 0x00000000, 0x00000000, 0x00000000).AsByte();

 Vector512<sbyte> mergeConstant0 = Vector512.Create(0x01400140).AsSByte();
 Vector512<short> mergeConstant1 = Vector512.Create(0x00011000).AsInt16();

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.
 do
 {
 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 // Step 1: Translate encoded Base64 input to their original indices
 // This step also checks for invalid inputs and exits.
 // After this, we have indices which are verified to have upper 2 bits set to 0 in each byte.
 // origIndex = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 Vector512<sbyte> origIndex = Avx512Vbmi.PermuteVar64x8x2(vbmiLookup0, str, vbmiLookup1);
 Vector512<sbyte> errorVec = (origIndex.AsInt32() | str.AsInt32()).AsSByte();
 if (errorVec.ExtractMostSignificantBits() != 0)
Comment thread
tannergooding marked this conversation as resolved.


Outdated

 {
 break;
 }

 // Step 2: Now we need to reshuffle bits to remove the 0 bits.
 // multiAdd1: [...|0000cccc|ccdddddd|0000aaaa|aabbbbbb]
 Vector512<short> multiAdd1 = Avx512BW.MultiplyAddAdjacent(origIndex.AsByte(), mergeConstant0);
 // multiAdd1: [...|00000000|aaaaaabb|bbbbcccc|ccdddddd]
 Vector512<int> multiAdd2 = Avx512BW.MultiplyAddAdjacent(multiAdd1, mergeConstant1);

 // Step 3: Pack 48 bytes
 str = Avx512Vbmi.PermuteVar64x8(multiAdd2.AsByte(), vbmiPackedLanesControl).AsSByte();

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);
 src += 64;
 dest += 48;
 }
 while (src <= srcEnd);

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Encoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -67,7 +67,16 @@ public static unsafe OperationStatus EncodeToUtf8(ReadOnlySpan<byte> bytes, Span

 if (maxSrcLength >= 16)
 {
 byte* end = srcMax - 32;
 byte* end = srcMax - 64;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 goto DoneExit;
 }

 end = srcMax - 64;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -226,6 +235,80 @@ public static unsafe OperationStatus EncodeToUtf8InPlace(Span<byte> buffer, int
 }
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/encode
 // If we have AVX512 support, pick off 48 bytes at a time for as long as we can.
 // But because we read 64 bytes at a time, ensure we have enough room to do a
 // full 64-byte read without segfaulting.

 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> shuffleVecVbmi = Vector512.Create(
 0x01020001, 0x04050304, 0x07080607, 0x0a0b090a,
 0x0d0e0c0d, 0x10110f10, 0x13141213, 0x16171516,
 0x191a1819, 0x1c1d1b1c, 0x1f201e1f, 0x22232122,
 0x25262425, 0x28292728, 0x2b2c2a2b, 0x2e2f2d2e).AsSByte();
 Vector512<sbyte> vbmiLookup = Vector512.Create("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"u8).AsSByte();

 Vector512<ushort> maskAC = Vector512.Create((uint)0x0fc0fc00).AsUInt16();
 Vector512<uint> maskBB = Vector512.Create((uint)0x3f003f00);
 Vector512<ushort> shiftAC = Vector512.Create((uint)0x0006000a).AsUInt16();
 Vector512<ushort> shiftBB = Vector512.Create((uint)0x00080004).AsUInt16();

 AssertRead<Vector256<sbyte>>(src, srcStart, sourceLength);

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.

 // str = [...|PONM|LKJI|HGFE|DCBA]
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 while (true)
Comment thread
MihaZupan marked this conversation as resolved.


Outdated

 {
 // Step 1 : Split 48 bytes into 64 bytes with each byte using 6-bits from input
 // str = [...|KLJK|HIGH|EFDE|BCAB]
 str = Avx512Vbmi.PermuteVar64x8(str, shuffleVecVbmi);

 // TO-DO- This can be achieved faster with multishift
 // Consider the first 4 bytes - BCAB
 // temp1 = [...|0000cccc|cc000000|aaaaaa00|00000000]
 Vector512<ushort> temp1 = (str.AsUInt16() & maskAC);

 // temp2 = [...|00000000|00cccccc|00000000|00aaaaaa]
 Vector512<ushort> temp2 = Avx512BW.ShiftRightLogicalVariable(temp1, shiftAC).AsUInt16();

 // temp3 = [...|ccdddddd|00000000|aabbbbbb|cccc0000]
 Vector512<ushort> temp3 = Avx512BW.ShiftLeftLogicalVariable(str.AsUInt16(), shiftBB).AsUInt16();

 // str = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 str = Vector512.ConditionalSelect(maskBB, temp3.AsUInt32(), temp2.AsUInt32()).AsSByte();

 // Step 2: Now we have the indices calculated. Next step is to use these indices to translate.
 str = Avx512Vbmi.PermuteVar64x8(vbmiLookup, str);

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);

 src += 48;
 dest += 64;

 if (src > srcEnd)
 break;

 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 str = Vector512.Load(src).AsSByte();
 }

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Adding AVX512 path to Base64 encoding/Decoding#92241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

tannergooding
merged 5 commits into

from

Oct 25, 2023
 Merged


Adding AVX512 path to Base64 encoding/Decoding
#92241
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
911b517

Adding AVX512 path to Base64 encoding/Decoding


DeepakRajendrakumaran Sep 18, 2023
7b351ad

Addressing review Comments.


DeepakRajendrakumaran Sep 19, 2023
7fb00e3

Removing fallback path.


DeepakRajendrakumaran Sep 20, 2023
2969ddb

Updating Third Party Notice.


DeepakRajendrakumaran Sep 21, 2023
fc05beb

Addressing review comments


DeepakRajendrakumaran Oct 24, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


34 changes: 34 additions & 0 deletions
THIRD-PARTY-NOTICES.TXT

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1297,3 +1297,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

License notice for Avx512Vbmi base64 encoding / decoding
--------------------------------------------------------

Copyright (c) 2015-2018, Wojciech Muła
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------

Aspects of base64 encoding / decoding are based on algorithm described in "Base64 encoding and decoding at almost the speed of a memory
copy", Wojciech Muła and Daniel Lemire. https://arxiv.org/pdf/1910.05109.pdf

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Decoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -68,7 +68,18 @@ private static unsafe OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> utf8, Sp

 if (maxSrcLength >= 24)
 {
 byte* end = srcMax - 45;
 byte* end = srcMax - 88;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 {
 goto DoneExit;
 }
 }

 end = srcMax - 45;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -616,6 +627,78 @@ private static OperationStatus DecodeWithWhiteSpaceFromUtf8InPlace(Span<byte> ut
 return status;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/decode
 // If we have AVX512 support, pick off 64 bytes at a time for as long as we can,
 // but make sure that we quit before seeing any == markers at the end of the
 // string. Also, because we write 16 zeroes at the end of the output, ensure
 // that there are at least 22 valid bytes of input data remaining to close the
 // gap. 64 + 2 + 22 = 88 bytes.
 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> vbmiLookup0 = Vector512.Create(
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x3e808080, 0x3f808080,
 0x37363534, 0x3b3a3938, 0x80803d3c, 0x80808080).AsSByte();
 Vector512<sbyte> vbmiLookup1 = Vector512.Create(
 0x02010080, 0x06050403, 0x0a090807, 0x0e0d0c0b,
 0x1211100f, 0x16151413, 0x80191817, 0x80808080,
 0x1c1b1a80, 0x201f1e1d, 0x24232221, 0x28272625,
 0x2c2b2a29, 0x302f2e2d, 0x80333231, 0x80808080).AsSByte();
 Vector512<byte> vbmiPackedLanesControl = Vector512.Create(
 0x06000102, 0x090a0405, 0x0c0d0e08, 0x16101112,
 0x191a1415, 0x1c1d1e18, 0x26202122, 0x292a2425,
 0x2c2d2e28, 0x36303132, 0x393a3435, 0x3c3d3e38,
 0x00000000, 0x00000000, 0x00000000, 0x00000000).AsByte();

 Vector512<sbyte> mergeConstant0 = Vector512.Create(0x01400140).AsSByte();
 Vector512<short> mergeConstant1 = Vector512.Create(0x00011000).AsInt16();

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.
 do
 {
 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 // Step 1: Translate encoded Base64 input to their original indices
 // This step also checks for invalid inputs and exits.
 // After this, we have indices which are verified to have upper 2 bits set to 0 in each byte.
 // origIndex = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 Vector512<sbyte> origIndex = Avx512Vbmi.PermuteVar64x8x2(vbmiLookup0, str, vbmiLookup1);
 Vector512<sbyte> errorVec = (origIndex.AsInt32() | str.AsInt32()).AsSByte();
 if (errorVec.ExtractMostSignificantBits() != 0)
Comment thread
tannergooding marked this conversation as resolved.


Outdated

 {
 break;
 }

 // Step 2: Now we need to reshuffle bits to remove the 0 bits.
 // multiAdd1: [...|0000cccc|ccdddddd|0000aaaa|aabbbbbb]
 Vector512<short> multiAdd1 = Avx512BW.MultiplyAddAdjacent(origIndex.AsByte(), mergeConstant0);
 // multiAdd1: [...|00000000|aaaaaabb|bbbbcccc|ccdddddd]
 Vector512<int> multiAdd2 = Avx512BW.MultiplyAddAdjacent(multiAdd1, mergeConstant1);

 // Step 3: Pack 48 bytes
 str = Avx512Vbmi.PermuteVar64x8(multiAdd2.AsByte(), vbmiPackedLanesControl).AsSByte();

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);
 src += 64;
 dest += 48;
 }
 while (src <= srcEnd);

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Encoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -67,7 +67,16 @@ public static unsafe OperationStatus EncodeToUtf8(ReadOnlySpan<byte> bytes, Span

 if (maxSrcLength >= 16)
 {
 byte* end = srcMax - 32;
 byte* end = srcMax - 64;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 goto DoneExit;
 }

 end = srcMax - 64;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -226,6 +235,80 @@ public static unsafe OperationStatus EncodeToUtf8InPlace(Span<byte> buffer, int
 }
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/encode
 // If we have AVX512 support, pick off 48 bytes at a time for as long as we can.
 // But because we read 64 bytes at a time, ensure we have enough room to do a
 // full 64-byte read without segfaulting.

 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> shuffleVecVbmi = Vector512.Create(
 0x01020001, 0x04050304, 0x07080607, 0x0a0b090a,
 0x0d0e0c0d, 0x10110f10, 0x13141213, 0x16171516,
 0x191a1819, 0x1c1d1b1c, 0x1f201e1f, 0x22232122,
 0x25262425, 0x28292728, 0x2b2c2a2b, 0x2e2f2d2e).AsSByte();
 Vector512<sbyte> vbmiLookup = Vector512.Create("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"u8).AsSByte();

 Vector512<ushort> maskAC = Vector512.Create((uint)0x0fc0fc00).AsUInt16();
 Vector512<uint> maskBB = Vector512.Create((uint)0x3f003f00);
 Vector512<ushort> shiftAC = Vector512.Create((uint)0x0006000a).AsUInt16();
 Vector512<ushort> shiftBB = Vector512.Create((uint)0x00080004).AsUInt16();

 AssertRead<Vector256<sbyte>>(src, srcStart, sourceLength);

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.

 // str = [...|PONM|LKJI|HGFE|DCBA]
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 while (true)
Comment thread
MihaZupan marked this conversation as resolved.


Outdated

 {
 // Step 1 : Split 48 bytes into 64 bytes with each byte using 6-bits from input
 // str = [...|KLJK|HIGH|EFDE|BCAB]
 str = Avx512Vbmi.PermuteVar64x8(str, shuffleVecVbmi);

 // TO-DO- This can be achieved faster with multishift
 // Consider the first 4 bytes - BCAB
 // temp1 = [...|0000cccc|cc000000|aaaaaa00|00000000]
 Vector512<ushort> temp1 = (str.AsUInt16() & maskAC);

 // temp2 = [...|00000000|00cccccc|00000000|00aaaaaa]
 Vector512<ushort> temp2 = Avx512BW.ShiftRightLogicalVariable(temp1, shiftAC).AsUInt16();

 // temp3 = [...|ccdddddd|00000000|aabbbbbb|cccc0000]
 Vector512<ushort> temp3 = Avx512BW.ShiftLeftLogicalVariable(str.AsUInt16(), shiftBB).AsUInt16();

 // str = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 str = Vector512.ConditionalSelect(maskBB, temp3.AsUInt32(), temp2.AsUInt32()).AsSByte();

 // Step 2: Now we have the indices calculated. Next step is to use these indices to translate.
 str = Avx512Vbmi.PermuteVar64x8(vbmiLookup, str);

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);

 src += 48;
 dest += 64;

 if (src > srcEnd)
 break;

 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 str = Vector512.Load(src).AsSByte();
 }

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Adding AVX512 path to Base64 encoding/Decoding#92241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

 Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and
privacy statement. We’ll occasionally send you account related emails.

Already on GitHub?
Sign in
to your account

 Merged

tannergooding
merged 5 commits into

from

Oct 25, 2023
 Merged


Adding AVX512 path to Base64 encoding/Decoding
#92241
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


5 commits


Select commit
Hold shift + click to select a range
911b517

Adding AVX512 path to Base64 encoding/Decoding


DeepakRajendrakumaran Sep 18, 2023
7b351ad

Addressing review Comments.


DeepakRajendrakumaran Sep 19, 2023
7fb00e3

Removing fallback path.


DeepakRajendrakumaran Sep 20, 2023
2969ddb

Updating Third Party Notice.


DeepakRajendrakumaran Sep 21, 2023
fc05beb

Addressing review comments


DeepakRajendrakumaran Oct 24, 2023

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


34 changes: 34 additions & 0 deletions
THIRD-PARTY-NOTICES.TXT

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -1297,3 +1297,37 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

License notice for Avx512Vbmi base64 encoding / decoding
--------------------------------------------------------

Copyright (c) 2015-2018, Wojciech Muła
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:

1. Redistributions of source code must retain the above copyright
 notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright
 notice, this list of conditions and the following disclaimer in the
 documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED
TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--------------------------------------------------------

Aspects of base64 encoding / decoding are based on algorithm described in "Base64 encoding and decoding at almost the speed of a memory
copy", Wojciech Muła and Daniel Lemire. https://arxiv.org/pdf/1910.05109.pdf

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Decoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -68,7 +68,18 @@ private static unsafe OperationStatus DecodeFromUtf8(ReadOnlySpan<byte> utf8, Sp

 if (maxSrcLength >= 24)
 {
 byte* end = srcMax - 45;
 byte* end = srcMax - 88;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 {
 goto DoneExit;
 }
 }

 end = srcMax - 45;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Decode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -616,6 +627,78 @@ private static OperationStatus DecodeWithWhiteSpaceFromUtf8InPlace(Span<byte> ut
 return status;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/decode
 // If we have AVX512 support, pick off 64 bytes at a time for as long as we can,
 // but make sure that we quit before seeing any == markers at the end of the
 // string. Also, because we write 16 zeroes at the end of the output, ensure
 // that there are at least 22 valid bytes of input data remaining to close the
 // gap. 64 + 2 + 22 = 88 bytes.
 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> vbmiLookup0 = Vector512.Create(
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x80808080, 0x80808080,
 0x80808080, 0x80808080, 0x3e808080, 0x3f808080,
 0x37363534, 0x3b3a3938, 0x80803d3c, 0x80808080).AsSByte();
 Vector512<sbyte> vbmiLookup1 = Vector512.Create(
 0x02010080, 0x06050403, 0x0a090807, 0x0e0d0c0b,
 0x1211100f, 0x16151413, 0x80191817, 0x80808080,
 0x1c1b1a80, 0x201f1e1d, 0x24232221, 0x28272625,
 0x2c2b2a29, 0x302f2e2d, 0x80333231, 0x80808080).AsSByte();
 Vector512<byte> vbmiPackedLanesControl = Vector512.Create(
 0x06000102, 0x090a0405, 0x0c0d0e08, 0x16101112,
 0x191a1415, 0x1c1d1e18, 0x26202122, 0x292a2425,
 0x2c2d2e28, 0x36303132, 0x393a3435, 0x3c3d3e38,
 0x00000000, 0x00000000, 0x00000000, 0x00000000).AsByte();

 Vector512<sbyte> mergeConstant0 = Vector512.Create(0x01400140).AsSByte();
 Vector512<short> mergeConstant1 = Vector512.Create(0x00011000).AsInt16();

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.
 do
 {
 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 // Step 1: Translate encoded Base64 input to their original indices
 // This step also checks for invalid inputs and exits.
 // After this, we have indices which are verified to have upper 2 bits set to 0 in each byte.
 // origIndex = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 Vector512<sbyte> origIndex = Avx512Vbmi.PermuteVar64x8x2(vbmiLookup0, str, vbmiLookup1);
 Vector512<sbyte> errorVec = (origIndex.AsInt32() | str.AsInt32()).AsSByte();
 if (errorVec.ExtractMostSignificantBits() != 0)
Comment thread
tannergooding marked this conversation as resolved.


Outdated

 {
 break;
 }

 // Step 2: Now we need to reshuffle bits to remove the 0 bits.
 // multiAdd1: [...|0000cccc|ccdddddd|0000aaaa|aabbbbbb]
 Vector512<short> multiAdd1 = Avx512BW.MultiplyAddAdjacent(origIndex.AsByte(), mergeConstant0);
 // multiAdd1: [...|00000000|aaaaaabb|bbbbcccc|ccdddddd]
 Vector512<int> multiAdd2 = Avx512BW.MultiplyAddAdjacent(multiAdd1, mergeConstant1);

 // Step 3: Pack 48 bytes
 str = Avx512Vbmi.PermuteVar64x8(multiAdd2.AsByte(), vbmiPackedLanesControl).AsSByte();

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);
 src += 64;
 dest += 48;
 }
 while (src <= srcEnd);

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Decode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down

85 changes: 84 additions & 1 deletion
src/libraries/System.Private.CoreLib/src/System/Buffers/Text/Base64Encoder.cs

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -67,7 +67,16 @@ public static unsafe OperationStatus EncodeToUtf8(ReadOnlySpan<byte> bytes, Span

 if (maxSrcLength >= 16)
 {
 byte* end = srcMax - 32;
 byte* end = srcMax - 64;
 if (Vector512.IsHardwareAccelerated && Avx512Vbmi.IsSupported && (end >= src))
 {
 Avx512Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);

 if (src == srcEnd)
 goto DoneExit;
 }

 end = srcMax - 64;
 if (Avx2.IsSupported && (end >= src))
 {
 Avx2Encode(ref src, ref dest, end, maxSrcLength, destLength, srcBytes, destBytes);
Expand DownExpand Up@@ -226,6 +235,80 @@ public static unsafe OperationStatus EncodeToUtf8InPlace(Span<byte> buffer, int
 }
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx512BW))]
 [CompExactlyDependsOn(typeof(Avx512Vbmi))]
 private static unsafe void Avx512Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
 {
 // Reference for VBMI implementation : https://github.com/WojciechMula/base64simd/tree/master/encode
 // If we have AVX512 support, pick off 48 bytes at a time for as long as we can.
 // But because we read 64 bytes at a time, ensure we have enough room to do a
 // full 64-byte read without segfaulting.

 byte* src = srcBytes;
 byte* dest = destBytes;

 // The JIT won't hoist these "constants", so help it
 Vector512<sbyte> shuffleVecVbmi = Vector512.Create(
 0x01020001, 0x04050304, 0x07080607, 0x0a0b090a,
 0x0d0e0c0d, 0x10110f10, 0x13141213, 0x16171516,
 0x191a1819, 0x1c1d1b1c, 0x1f201e1f, 0x22232122,
 0x25262425, 0x28292728, 0x2b2c2a2b, 0x2e2f2d2e).AsSByte();
 Vector512<sbyte> vbmiLookup = Vector512.Create("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"u8).AsSByte();

 Vector512<ushort> maskAC = Vector512.Create((uint)0x0fc0fc00).AsUInt16();
 Vector512<uint> maskBB = Vector512.Create((uint)0x3f003f00);
 Vector512<ushort> shiftAC = Vector512.Create((uint)0x0006000a).AsUInt16();
 Vector512<ushort> shiftBB = Vector512.Create((uint)0x00080004).AsUInt16();

 AssertRead<Vector256<sbyte>>(src, srcStart, sourceLength);

 // This algorithm requires AVX512VBMI support.
 // Vbmi was first introduced in CannonLake and is avaialable from IceLake on.

 // str = [...|PONM|LKJI|HGFE|DCBA]
 Vector512<sbyte> str = Vector512.Load(src).AsSByte();

 while (true)
Comment thread
MihaZupan marked this conversation as resolved.


Outdated

 {
 // Step 1 : Split 48 bytes into 64 bytes with each byte using 6-bits from input
 // str = [...|KLJK|HIGH|EFDE|BCAB]
 str = Avx512Vbmi.PermuteVar64x8(str, shuffleVecVbmi);

 // TO-DO- This can be achieved faster with multishift
 // Consider the first 4 bytes - BCAB
 // temp1 = [...|0000cccc|cc000000|aaaaaa00|00000000]
 Vector512<ushort> temp1 = (str.AsUInt16() & maskAC);

 // temp2 = [...|00000000|00cccccc|00000000|00aaaaaa]
 Vector512<ushort> temp2 = Avx512BW.ShiftRightLogicalVariable(temp1, shiftAC).AsUInt16();

 // temp3 = [...|ccdddddd|00000000|aabbbbbb|cccc0000]
 Vector512<ushort> temp3 = Avx512BW.ShiftLeftLogicalVariable(str.AsUInt16(), shiftBB).AsUInt16();

 // str = [...|00dddddd|00cccccc|00bbbbbb|00aaaaaa]
 str = Vector512.ConditionalSelect(maskBB, temp3.AsUInt32(), temp2.AsUInt32()).AsSByte();

 // Step 2: Now we have the indices calculated. Next step is to use these indices to translate.
 str = Avx512Vbmi.PermuteVar64x8(vbmiLookup, str);

 AssertWrite<Vector512<sbyte>>(dest, destStart, destLength);
 str.Store((sbyte*)dest);

 src += 48;
 dest += 64;

 if (src > srcEnd)
 break;

 AssertRead<Vector512<sbyte>>(src, srcStart, sourceLength);
 str = Vector512.Load(src).AsSByte();
 }

 srcBytes = src;
 destBytes = dest;
 }

 [MethodImpl(MethodImplOptions.AggressiveInlining)]
 [CompExactlyDependsOn(typeof(Avx2))]
 private static unsafe void Avx2Encode(ref byte* srcBytes, ref byte* destBytes, byte* srcEnd, int sourceLength, int destLength, byte* srcStart, byte* destStart)
Expand Down