`<print>`: `std::println("{}", ...)` UTF-8 Truncation Bug with `/utf-8` (256-byte Buffer Split)

[739C1AE2]

Describe the bug

When using the MSVC compiler with /utf-8, std::println truncates overly long UTF-8 strings at internal buffer boundaries (replaced with U+FFFD replacement characters) when formatting arguments are used (std::println("{}", str)).

Reproduction Code and Output

#include <print>

int main()
{
    std::println("{}", "这是一段超长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长的文本。");
}

Compilation Command: cl.exe /std:c++latest /utf-8 repro.cpp

Compiler Version: 用于 x86 的 Microsoft (R) C/C++ 优化编译器 19.50.35718 版

Expected Behavior: The UTF-8 string is output completely and correctly.

Observed Behavior: 这是一段超长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长���的文本。

Possible Cause

The issue is likely caused by the output mechanism splitting the long string into small, fixed-size chunks (e.g., 256 bytes) before sending them to the console.

The failure is suspected to lie in the specialized handler responsible for committing these chunks to the console: _Fmt_iterator_flush<_Print_to_unicode_console_it>.

This handler, which manages the UTF-8 to Console conversion, appears to simply pass the raw byte chunk's range (_First to _Last) to the underlying write function without ensuring the chunk contains complete UTF-8 characters:

STL/stl/inc/print

Lines 107 to 114 in 5459853

	template <>
	struct _Fmt_iterator_flush<_Print_to_unicode_console_it> {
	static _Print_to_unicode_console_it _Flush(
	const char* const _First, const char* const _Last, _Print_to_unicode_console_it _Output) {
	_STD _Print_noformat_unicode_to_console_nonlocking(_Output._Get_console_handle(), {_First, _Last});
	return _Output;
	}
	};

If a chunk ends in the middle of a multi-byte UTF-8 character, committing the incomplete sequence at this point may cause the downstream MultiByteToWideChar conversion to fail, resulting in the observed U+FFFD characters. This suggests the necessary UTF-8 boundary check logic may be missing from this specific specialization before the data is written to the console handle.

[YexuanXiao]

I believe the issue is in __msvc_ranges_tuple_formatter.hpp. Essentially, print.cpp L148-L178 correctly handles the UTF-8 code unit, but __msvc_ranges_tuple_formatter.hpp L801,L837 truncates the code unit prematurely.

Adjusting the value of _Fmt_buffer_size can change the position where u+fffd appears.

STL/stl/inc/__msvc_ranges_tuple_formatter.hpp

Lines 800 to 802 in 5459853

	inline constexpr size_t _Fmt_buffer_size = 256;

STL/stl/inc/__msvc_ranges_tuple_formatter.hpp

Lines 832 to 837 in 5459853

	template <class _OutputIt, class _Ty, class _Traits = _Fmt_buffer_traits>
	class _Fmt_iterator_buffer final : public _Traits, public _Fmt_buffer<_Ty> {
	private:
	_OutputIt _Output;
	_Ty _Data[_Fmt_buffer_size];

STL/stl/src/print.cpp

Lines 148 to 178 in 5459853

	[[nodiscard]] _Minimal_string_view _Get_next_utf8_string_segment(
	const char* const _Str, const size_t _Str_size) noexcept {
	constexpr size_t _Max_str_segment_size = 8192;
	if (_Str_size <= _Max_str_segment_size) {
	return _Minimal_string_view{_Str, _Str_size};
	}
	// Let's refer to _Max_str_segment_size as M.
	// Now we know _Str_size > M, so we can read _Str[M].
	// We might need to shrink this segment down to M - 3 bytes, in this worst case scenario:
	// Values: [byte1][byte2][byte3] | [byte4]
	// Indices: [M - 4][M - 3][M - 2][M - 1] | [ M ]
	// Sizes: [M - 3][M - 2][M - 1][ M ] | [M + 1]
	// Maximum segment boundary ^
	for (size_t _Shrink = 0; _Shrink < 3; ++_Shrink) {
	const size_t _Kx = _Max_str_segment_size - _Shrink; // consider a segment of _Kx bytes
	// The first byte after the segment is at index _Kx, which we can read (see above).
	// If _Str[_Kx] is a non-trailing byte, then it's the beginning of a code point.
	const bool _Trailing_byte = (static_cast<unsigned char>(_Str[_Kx]) >> 6) == 0b10;
	if (!_Trailing_byte) {
	return _Minimal_string_view{_Str, _Kx}; // found a boundary between code points
	}
	}
	return _Minimal_string_view{_Str, _Max_str_segment_size - 3}; // worst case scenario
	}

[YexuanXiao]

Potential solutions:

1. Bypass __msvc_ranges_tuple_formatter.hpp.

2. Perform transcoding in print.cpp only when the buffer is full or upon output completion.

[Demonese]

PS C:\Users\Admin\AppData\Local\Temp> cl /std:c++latest /utf-8 print.cpp
Microsoft (R) C/C++ Optimizing Compiler Version 19.51.36247 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

/std:c++latest is provided as a preview of language features from the latest C++
working draft, and we're eager to hear about bugs and suggestions for improvements.
However, note that these features are provided as-is without support, and subject
to changes or removal as the working draft evolves. See
https://go.microsoft.com/fwlink/?linkid=2045807 for details.

print.cpp
Microsoft (R) Incremental Linker Version 14.51.36247.0
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:print.exe
print.obj
PS C:\Users\Admin\AppData\Local\Temp> ./print
这是一段超长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长长���的文本。
PS C:\Users\Admin\AppData\Local\Temp>

Stephan T. Lavavej (@StephanTLavavej) This issue still exists in 14.51. It's been around for a long time.