getStringFromCodePoints() silently drops surrogate code points, causing lossy string transformation

[masakielastic]

Summary

UnicodeString::getStringFromCodePoints() currently skips code points in the surrogate range U+D800..U+DFFF.

Surrogate code points must not be encoded as UTF-8, so avoiding UTF-8 generation for this range is correct. However, silently dropping them changes the semantic value of the input.

For security-sensitive use cases, this behavior can be problematic because different code point sequences may collapse into the same output string.

Current behavior

The current implementation contains the following logic:

if ($code >= 0xD800 && $code <= 0xDFFF) {
 continue;
}

This means that invalid Unicode scalar values are silently removed from the output.

For example, the following code point sequence:

[
 0x61, // a
 0x74, // t
 0x74, // t
 0x61, // a
 0xD800, // surrogate code point
 0x63, // c
 0x6B, // k
]

is converted to:

attack

In other words:

"atta" + U+D800 + "ck"

becomes:

"attack"

Why this is problematic

The issue is not that surrogate code points are rejected for UTF-8 encoding. That part is correct.

The problem is that invalid input is not rejected or visibly replaced. It is silently removed.

This is a lossy transformation. It can make distinct inputs produce the same output:

getStringFromCodePoints([
 0x61, 0x74, 0x74, 0x61, 0x63, 0x6B
]);

getStringFromCodePoints([
 0x61, 0x74, 0x74, 0x61, 0xD800, 0x63, 0x6B
]);

Both result in:

attack

In security-sensitive contexts, this kind of implicit normalization can lead to validation bypasses, identifier collisions, inconsistent comparison results, or misleading logs.

Comparison with PHP standard APIs

PHP standard APIs provide a useful comparison.

htmlspecialchars() supports ENT_SUBSTITUTE for invalid code unit sequences. In modern PHP versions, ENT_SUBSTITUTE is also included in the default flags.

JSON APIs also distinguish between two different behaviors:

• JSON_INVALID_UTF8_IGNORE
• JSON_INVALID_UTF8_SUBSTITUTE

This distinction is important. Ignoring invalid input is a lossy behavior, while substituting invalid input with U+FFFD preserves the fact that invalid data was present.

The current behavior of getStringFromCodePoints() is closer to an implicit “ignore invalid data” mode. For a Unicode conversion routine, that behavior should probably not be implicit.

Expected behavior

Invalid Unicode scalar values should not be silently dropped.

Possible alternatives:

1. Throw an exception for invalid code points.

2. Replace invalid code points with U+FFFD.

3. Provide explicit modes such as:

   • strict: reject invalid code points
   • substitute: replace invalid code points with U+FFFD
   • ignore: drop invalid code points, documented as lossy

For security-sensitive string processing, strict or substitute would be safer defaults than silently ignoring invalid code points.

Suggested change

Instead of silently continuing here:

if ($code >= 0xD800 && $code <= 0xDFFF) {
 continue;
}

the implementation could either reject the value:

if ($code < 0 || $code > 0x10FFFF || ($code >= 0xD800 && $code <= 0xDFFF)) {
 throw new InvalidArgumentException('Invalid Unicode scalar value.');
}

or replace it with U+FFFD:

if ($code < 0 || $code > 0x10FFFF || ($code >= 0xD800 && $code <= 0xDFFF)) {
 $code = 0xFFFD;
}

If the current behavior is kept for backward compatibility, it should be documented as an explicit lossy “ignore invalid code points” behavior.

[masakielastic]

The same issue exists in getCharFromCodePoint().

This method returns an empty string for invalid code points, including negative values, surrogate code points, and values above U+10FFFF.

However, an empty string is a valid string value and does not communicate that invalid input was encountered. If this method is used while building a larger string, invalid code points are silently removed, leading to the same lossy transformation issue as getStringFromCodePoints().

For example, concatenating getCharFromCodePoint() results for the sequence:

"atta" + U+D800 + "ck"

produces:

"attack"

because U+D800 is converted to an empty string.

Therefore, the issue is not limited to getStringFromCodePoints(). The lower-level getCharFromCodePoint() API should also define how invalid Unicode scalar values are handled.

Possible alternatives:

• throw an exception for invalid code points,
• return null or false in a try-style API,
• replace invalid code points with U+FFFD,
• or keep the current behavior only as an explicitly documented lossy ignore mode.