<format>: format should statically detect the default latin-1252 character set

[barcharcraz]

currently format can detect if you compile with /utf-8 and activate a fast path in the string parser (we can search for control characters using a simple byte search, instead of decoding all the code-points). In reality we can do this search for almost any encoding, with the exception of non-self-synchronizing multi-byte encodings.

In micro benchmarks the fast path can be 10-15x faster than the slow path, so getting the fast path for 1252 would have major performance benefits for users.

[CaseyCarter]

This should be almost trivial for MSVC which defines _MSVC_EXECUTION_CHARACTER_SET to the codepage used for encoding narrow strings. Ideally, we should also use _MSVC_EXECUTION_CHARACTER_SET to detect UTF-8 (codepage 65001) when compiling with MSVC.

[barcharcraz]

yep! we just need to remember to do it!

[AlexGuteniev]

Would linking 1251 and 1252 source produce ABI break with this optimization?

[CaseyCarter]

Would linking 1251 and 1252 source produce ABI break with this optimization?

I think you mean an ODR violation, and I believe the answer is "the optimization doesn't make this case any more broken than it was without the optimization." If you pass strings with two different encodings to the same function which must necessarily choose one encoding, you're going to have a bad time. (Unless, of course, you restrict your usage to only characters whose encoded values are the same across both of your encodings.)

[barcharcraz]

yep, if you mix literal encodings a huge amount of stuff breaks. It happens to sort-of work most of the time because almost all encodings are almost the same for the first 127 code-units.

This will cause new and more exciting breakage if you mix shift encodings and non-shift encodings. If the shift-encoding version is picked the decoding is nonsensical and may produce errors, if the non-shift one is picked then you'll probably get almost correct behavior, but sometimes (rarely) see malformed format strings because a multi-byte sequence got interpreted as a control character.

If you mix self-synchronizing encodings (such as 1251 and 1252, or 1251/1252 and UTF-8) things should continue to work.

[barcharcraz]

The real badness here is mixing CP936/CP932 and any of the non-shift charsets (1252/1251/utf-8/etc), but that was already really bad.

actually come to think of it now that we can pull the charset from the compiler we can do the "shift" decoding stuff using that, instead of the active system codepage at runtime