Sporadic unresolved test failures

[StephanTLavavej]

We're encountering sporadic unresolved failures. So far, they've always happened with x86 Clang in this test:

1: Unresolved Tests (1):
1:   libc++ :: std/input.output/string.streams/stringbuf/stringbuf.members/view.pass.cpp:2

Test File

https://github.com/llvm/llvm-project/blob/57f42a8765cd3d878be4fb59ad44c85f8a7ca223/libcxx/test/std/input.output/string.streams/stringbuf/stringbuf.members/view.pass.cpp

Occurrences

Date	PR	Arch	Azure Pipelines Run
2024-01-25	#4274	x86	https://dev.azure.com/vclibs/STL/_build/results?buildId=15987&view=results
2024-01-27	#4345	x86	https://dev.azure.com/vclibs/STL/_build/results?buildId=16004&view=results
2024-02-03	#4365	x86	https://dev.azure.com/vclibs/STL/_build/results?buildId=16061&view=results

Output

Thanks to #4323, we have some useful output:

UnicodeDecodeError('charmap', b'minkernel\\crts\\ucrt\\src\\appcrt\\heap\\debug_heap.cpp(904) : Assertion failed: _CrtIsValidHeapPointer(block)\r\nminkernel\\crts\\ucrt\\src\\appcrt\\heap\\debug_heap.cpp(908) : Assertion failed: is_block_type_valid(header->_block_use)\r\nHEAP CORRUPTION DETECTED: before R\xe8\xa5j\xfe\xff\x83\xc4\x08\x8dE\x08P\x8dM\xf0\xe8F6\xfe\xffPj\x05\xe8\x0e\xfc\xff\xff\x83\xc4\x08\x8dM\x08Q\x8dM\xec\xe8/6\xfe\xffPj\x04\xe8\x87\xfb\xff\xff\x83\xc4\x08\x8bM\xf4d\x89\r block (#-33686019) at 0x00EF2CD4.\nCRT detected that the application wrote to memory before start of heap buffer.\nHEAP CORRUPTION DETECTED: after R\xe8\xa5j\xfe\xff\x83\xc4\x08\x8dE\x08P\x8dM\xf0\xe8F6\xfe\xffPj\x05\xe8\x0e\xfc\xff\xff\x83\xc4\x08\x8dM\x08Q\x8dM\xec\xe8/6\xfe\xffPj\x04\xe8\x87\xfb\xff\xff\x83\xc4\x08\x8bM\xf4d\x89\r block (#-33686019) at 0x00EF2CD4.\nCRT detected that the application wrote to memory after end of heap buffer.\nminkernel\\crts\\ucrt\\src\\appcrt\\heap\\debug_heap.cpp(996) : Assertion failed: __acrt_first_block == header\r\n', 266, 267, 'character maps to <undefined>') at D:\a\_work\1\s\tests\utils\stl\test\format.py:231

Open Questions

• What's causing this _CrtIsValidHeapPointer assertion, which is the underlying spurious failure?
• Why is this specific to x86 Clang and stringbuf.members/view.pass.cpp? Is it a library bug in our basic_stringbuf, a test bug, or a compiler bug?
• Why is the Python machinery encountering a UnicodeDecodeError? Is something about the test output, like the heap contents, tripping it up? Do we need to fix the Python code somewhere, or does the UCRT need to fix its output?

[cpplearner]

I got the same error with the std/utilities/format/format.functions/escaped_output.ascii.pass.cpp test.

stl-lit.py: D:\STL\tests\utils\stl\test\format.py:250: error: UnicodeDecodeError('gbk', b"Assertion failure: out == expected D:\\STL\\llvm-project\\libcxx\\test\\std\\utilities\\format\\format.functions\\escaped_output.ascii.pass.cpp 46\r\n\r\nFormat string   {:?}\r\nExpected output '\x80'\r\nActual output   '\\x{80}'\r\nabort() has been called", 198, 199, 'illegal multibyte sequence') at D:\STL\tests\utils\stl\test\format.py:231
UNRESOLVED: libc++ :: std/utilities/format/format.functions/escaped_output.ascii.pass.cpp:1 (16 of 63)
******************** TEST 'libc++ :: std/utilities/format/format.functions/escaped_output.ascii.pass.cpp:1' FAILED ********************
Exception during script execution:
Traceback (most recent call last):
  File "D:\STL/llvm-project/llvm\utils\lit\lit\worker.py", line 77, in _execute_test_handle_errors
    return _adapt_result(result)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\STL/llvm-project/llvm\utils\lit\lit\worker.py", line 92, in _adapt_result
    assert isinstance(result, tuple)
AssertionError


********************

It seems that STLTestFormat.execute returns None when an exception has been raised, but llvm's _adapt_result expects it to return something meaningful (either lit.Test.Result or tuple).

[StephanTLavavej]

(Transferred this comment's info to the issue above.)

[frederick-vs-ja]

[ ] What's causing this _CrtIsValidHeapPointer assertion, which is the underlying spurious failure?

I believe this is #4268. (The public_setg function calls setg, and then changes the return value of eback, which is unfortunately depended by MSVC STL.)

The underlying failure is actually stable, which generally causes FAIL.

It seems that the exception essentially comes from here:

STL/tests/utils/stl/util.py

Lines 74 to 81 in 202e382

	def decodeOutput(bytes):
	# MSVC's output is encoded in the active code page
	# EDG's (`cl /BE`) output is encoded in UTF-8
	try:
	return bytes.decode()
	except UnicodeError:
	import locale
	return bytes.decode(locale.getpreferredencoding(do_setlocale=False))

Perhaps the UnicodeDecodeError exception was triggered by particular data pattern in the corrupted heap. Such data seemed directly included in the output of Clang.

[StephanTLavavej]

Changing a runtime test to contain this:

// Copyright (c) Microsoft Corporation.
// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception

#include <cstdio>
using namespace std;

int main() {
    puts("HEAP CORRUPTION DETECTED: before \x8d");
}

caused it to reliably emit:

code: UNRESOLVED
name: std :: tests/P0586R2_integer_comparison:04
output: UnicodeDecodeError('charmap', b'HEAP CORRUPTION DETECTED: before \x8d\r\n', 33, 34, 'character maps to <undefined>') at D:\GitHub\STL\tests\utils\stl\test\format.py:231

I can fix this.

I'm still not sure exactly how the heap corruption is causing the UCRT to get confused when it tries to print the block type, but that's not something we can do anything about (we can fix iostreams someday, and we can fix the Python now).