<regex> mishandles locale-based character classes outside of the char range

[AlexGuteniev]

Describe the bug
Regex does not handle non-ASCII characters.

Billy O'Neal (@BillyONeal) comments:

This is a longstanding bug in our regex engine -- when we form negated character classes (like \S), we negate the bitmap used for encoding units in the range [0-255], but don't have correct handling for encoding units outside that. We've known about this problem since at least October 5 of 2016, but it's ABI breaking to fix :(.

Command-line test case

d:\Temp2>type repro.cpp
#include <regex>
#include <iostream>

bool test(std::wstring line, std::wstring query)
{
    std::wregex regex(query);
    std::wsmatch res;
    return std::regex_search(line, res, regex);
}

int main()
{
    std::cout << test(L"xxxYxx\x0078xxxZxxx", L"Y\\S*Z") << std::endl; // 0078 is small Latin X
    std::cout << test(L"xxxYxx\xCF87xxxZxxx", L"Y\\S*Z") << std::endl; // CF87 is small Greek Chi 
}

d:\Temp2>cl /EHsc /W4 /WX .\repro.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 19.27.29009.1 for x86
Copyright (C) Microsoft Corporation.  All rights reserved.

repro.cpp
Microsoft (R) Incremental Linker Version 14.27.29009.1
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:repro.exe
repro.obj

d:\Temp2>.\repro.exe
1
0

Expected behavior
Given example should match, the correct output is:

1
1

STL version

Microsoft Visual Studio Professional 2019 Preview
Version 16.7.0 Preview 3.1

Additional context
Original repro:

#include <regex>
#include <iostream>

int main()
{
	std::wstring line = L"受注、製造、購買オーダのスケジューリングと自動補充生産機能";
	std::wstring query = L"造\\S*オ";
	std::wregex regex(query);
	std::wsmatch res;
	bool found = std::regex_search(line, res, regex);

	std::cout << found << std::endl;
}

This item is also tracked on Developer Community as DevCom-984204 and by Microsoft-internal VSO-273702 / AB#273702.

See also #405

vNext note: Resolving this issue will require breaking binary compatibility. We won't be able to accept pull requests for this issue until the vNext branch is available. See #169 for more information.

[StephanTLavavej]

Keeping this open as Julian Müller (@muellerj2) noted in #5160:

After this change, class escapes are handled correctly when they appear outside character class definitions, but not when they are used inside them. So regular expressions like \S or \D do work correctly now, but [\S], [\D] or [0\D] still show the defective behavior.

I'm going to resolve the DevCom/VSO bugs since the original test case was fixed, and I'll mention that we're keeping this GitHub bug open to track the remaining problem.