Skip to content

UTF16 buffer boundary XALANJ-2725#166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
maxfortun wants to merge 3 commits into
base:master
Choose a base branch
from
Open

UTF16 buffer boundary XALANJ-2725 #166

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
490b5ab
updated
maxfortun Jan 24, 2024
e92e170
updated
maxfortun Jan 24, 2024
0006abf
updated
maxfortun Jan 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 32 additions & 11 deletions serializer/src/main/java/org/apache/xml/serializer/ToStream.java
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -175,6 +175,10 @@ abstract public class ToStream extends SerializerBase
*/
private boolean m_expandDTDEntities = true;

/**
* Track multibyte character in order to serialize when the whole byte sequence is available.
*/
private char m_highUTF16Surrogate = 0;

/**
* Default constructor
Expand DownExpand Up@@ -1595,23 +1599,40 @@ else if (m_encodingInfo.isInEncoding(ch)) {
// not in the normal ASCII range, we also
// just leave it get added on to the clean characters
}
else if (Encodings.isHighUTF16Surrogate(ch) && i < end-1 && Encodings.isLowUTF16Surrogate(chars[i+1])) {
// So, this is a (valid) surrogate pair
if (! m_encodingInfo.isInEncoding(ch, chars[i+1])) {
int codepoint = Encodings.toCodePoint(ch, chars[i+1]);
writeOutCleanChars(chars, i, lastDirtyCharProcessed);
writer.write("&#");
writer.write(Integer.toString(codepoint));
writer.write(';');
lastDirtyCharProcessed = i+1;
}
i++; // skip the low surrogate, too
else if (Encodings.isHighUTF16Surrogate(ch)) {
// Store for later processing. We may be at the end of a buffer,
// and must wait till low surrogate arrives
// before we can do anything with this.
writeOutCleanChars(chars, i, lastDirtyCharProcessed);
m_highUTF16Surrogate = ch;
lastDirtyCharProcessed = i;
}
else if (m_highUTF16Surrogate != 0 && Encodings.isLowUTF16Surrogate(ch)) {
// The complete utf16 byte sequence is now available and may be serialized.
if (! m_encodingInfo.isInEncoding(m_highUTF16Surrogate, ch)) {

@maxfortunmaxfortunJan 24, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1602: If ch is high surrogate, dump clean chars and retain ch. This works within the same buffer or across buffer boundaries
1610: if we have a retained high surrogate and ch is a valid low surrogate, we have 2 use cases.
1. encoding doesn't support the multibyte char, escape entity.
2. encoding does support the multibyte char, output chars as is.

int codepoint = Encodings.toCodePoint(m_highUTF16Surrogate, ch);
writer.write("&#");
writer.write(Integer.toString(codepoint));
writer.write(';');
} else {
writer.write(m_highUTF16Surrogate);
writer.write(ch);
}
lastDirtyCharProcessed = i;
m_highUTF16Surrogate = 0;
}
else {
// This is a fallback plan, we get here if the
// encoding doesn't contain ch and it's not part
// of a surrogate pair
// The right thing is to write out an entity
if(m_highUTF16Surrogate != 0) {

@maxfortunmaxfortunJan 24, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is probably not a valid scenario and should throw an error. I am not certain what is the correct way to address a situation where we have seen a high surrogate not followed by a low one.

@maxfortunmaxfortunJan 24, 2024

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, I did encounter this scenario, that's why I coded this. But in theory this should not be happening.

writer.write("&#");
writer.write(Integer.toString(m_highUTF16Surrogate));
writer.write(';');
m_highUTF16Surrogate = 0;
}

writeOutCleanChars(chars, i, lastDirtyCharProcessed);
writer.write("&#");
writer.write(Integer.toString(ch));
Expand Down