`<regex>`: What names can and should `regex_traits::lookup_collatename()` recognize?

[muellerj2]

The title states this as one question, but it's really two.

First is an actual choice: Should the portable character names in the POSIX standard be recognized and translated bylookup_collatename()?

Cons:

• These aren't actually names for locale-specific collating elements (which are defined in Chapter 7); they are symbolic names of characters for internal use in the POSIX standard.
• TR1 was deliberately changed to not mandate that these are recognized by lookup_collatename() (see the end of Section 2 of N1623).

Pro:

• Boost.Regex and the other standard libraries recognize these names as an extension.
• After <regex>: Properly parse and match collating symbols and equivalences #5392, these names are the final puzzle piece to make a few more libcxx tests pass.

The second question is a technical issue that I don't have an answer to yet: Can we actually access the set of locale-specific (multi-character) collating elements or recognize them in a reasonable way using some Windows API? Or is there some other reasonable approach to recognize locale-specific collating elements like "ch" in Czech or "dzs" in Hungarian?

[muellerj2]

I have been looking a bit into the recognition of locale-specific collating elements:

• I failed to find a simple Windows API to query collating elements of a locale.
• In ICU, there is the concept of contractions, which is similar to POSIX' collating elements. ICU supports querying the list of contractions for a specific locale, so maybe we could solve this by querying ICU. But do we want to make regex dependent on ICU?
• It's possible to deduce indirectly from collate::transform() (which is based on LCMapStringEx) whether a character sequence is a collating element: Because collating elements behave like single characters, sort keys of the same length are produced for simple alphabetic characters and collating elements (on top of a few more structural similarities between the sort keys). The problem is that the structure of this sort key doesn't seem documented, so I'm not sure to what extent the stability of the sort key structure is guaranteed for LCMapStringEx or whether this observation relies on an implementation detail subject to change. (This "sort key sniffing" approach already drives some locale-specific regex behavior in Boost.Regex and libc++, including recognition of collating elements in libc++; Boost.Regex instead hardcodes a few collating elements independent of the actual imbued locale. )

[muellerj2]

After #6005 is merged, I think this is the most relevant remaining issue in <regex> unrelated to performance. It is also the source for most of the libc++ test failures.

I don't know if there is a sensible way to recognize locale-specific collating symbols using Windows Natural Language Support. It can be done by examining the structure of the search key produced by LCMapStringKeyEx, but that approach strikes me as fairly fragile. (To take note, libc++ relies on this approach, but recently some OSes changed the format of these keys and locale-related tests started failing. This led to the removal of locale-related regex tests.) I think this question needs to be asked to an expert on WinNLS.

Alternatively, ICU provides the API ucol_getContractionsAndExpansions(), and we could accept a string as a collating symbol if it is a contraction according to ICU in this locale. But relying on this function means that <regex> will have to take ICU as a dependency. Furthermore, ICU and WinNLS might disagree on the set of collating symbols for a language, and in all other contexts we rely on WinNLS in <regex>.