Sporadic test failures after VS 2019 16.8 Preview 1 toolset update

[StephanTLavavej]

After merging #1153 followed by other PRs, we had 6 consecutive successful CI builds of master. We also had successful PR builds of #1177 and #1178 followed by successful CI builds of feature/format and feature/spaceship.

However, other PRs appear to be encountering sporadic failures that we haven't seen before. statementreply's #935 and #1173 failed, and their test runs (here and here) appear to involve the compiler abruptly failing without printing anything:

stl-lit.py: C:\agent\_work\1\s\tests\utils\stl\test\format.py:203: note: Build step failed unexpectedly.
Command: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.28.29115\bin\HostX64\x64\cl.EXE" "/FeD:\build\x64\tests\libcxx\std\localization\locale.categories\category.ctype\facet.ctype.special\facet.ctype.char.members\tolower_many.pass\0\tolower_many.pass.exe" "/FoD:\build\x64\tests\libcxx\std\localization\locale.categories\category.ctype\facet.ctype.special\facet.ctype.char.members\tolower_many.pass\0\tolower_many.pass.obj" "/ID:\build\x64\out\inc" "/IC:/agent/_work/1/s/llvm-project/libcxx/test/support" "/nologo" "/Od" "/W4" "/w14061" "/w14242" "/w14265" "/w14582" "/w14583" "/w14587" "/w14588" "/w14749" "/w14841" "/w14842" "/w15038" "/w15214" "/w15215" "/w15216" "/w15217" "/sdl" "/WX" "/Zc:strictStrings" "/D_ENABLE_STL_INTERNAL_CHECK" "/bigobj" "/EHsc" "/MTd" "/std:c++latest" "/permissive-" "/FImsvc_stdlib_force_include.h" "/wd4643" "/analyze" "/Zc:preprocessor" "/analyze:logD:\build\x64\tests\libcxx\std\localization\locale.categories\category.ctype\facet.ctype.special\facet.ctype.char.members\tolower_many.pass\0\tolower_many.pass.nativecodeanalysis.xml" "C:\agent\_work\1\s\llvm-project\libcxx\test\std\localization\locale.categories\category.ctype\facet.ctype.special\facet.ctype.char.members\tolower_many.pass.cpp" "/link" "/LIBPATH:D:\build\x64\out\lib\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.28.29115\lib\x64" "/MANIFEST:EMBED"
Exit Code: 2
Standard Output:
--
tolower_many.pass.cpp
--

stl-lit.py: C:\agent\_work\1\s\tests\utils\stl\test\format.py:203: note: Build step failed unexpectedly.
Command: "C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.28.29115\bin\HostX64\x64\cl.EXE" "/FeD:\build\x64\tests\std\tests\VSO_0105317_expression_sfinae\15\VSO_0105317_expression_sfinae.exe" "/FoD:\build\x64\tests\std\tests\VSO_0105317_expression_sfinae\15\VSO_0105317_expression_sfinae.obj" "/ID:\build\x64\out\inc" "/IC:/agent/_work/1/s/llvm-project/libcxx/test/support" "/IC:/agent/_work/1/s/tests/std/include" "/nologo" "/Od" "/W4" "/w14061" "/w14242" "/w14265" "/w14582" "/w14583" "/w14587" "/w14588" "/w14749" "/w14841" "/w14842" "/w15038" "/w15214" "/w15215" "/w15216" "/w15217" "/sdl" "/WX" "/Zc:strictStrings" "/D_ENABLE_STL_INTERNAL_CHECK" "/bigobj" "/FIforce_include.hpp" "/w14365" "/D_ENFORCE_FACET_SPECIALIZATIONS=1" "/EHsc" "/MTd" "/D_ITERATOR_DEBUG_LEVEL=2" "/std:c++latest" "/await" "/w14640" "/Zc:threadSafeInit-" "C:\agent\_work\1\s\tests\std\tests\VSO_0105317_expression_sfinae\test.cpp" "/link" "/LIBPATH:D:\build\x64\out\lib\amd64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Preview\VC\Tools\MSVC\14.28.29115\lib\x64" "/MANIFEST:EMBED"
Exit Code: 2
Standard Output:
--
test.cpp
--

These failures affected different tests, but both involved x64. One was /analyze, the other wasn't, so that appears to be unrelated.

[BillyONeal]

I do indeed see a lot of memory consumption; I doubt the compiler is hardened against OOM:

so it would be luck of when tests get scheduled for when the systems would run out of commit limit. And x64 would be more likely to trigger it since pointers are twice as big.

I vaguely recall an email thread about a PREfast memory use regression from Xiang that might be related.

[miscco]

Note that in #1083 we consistently run out of heap space again in an unrelated test:

https://github.com/microsoft/STL/pull/1083/checks?check_run_id=964285122

The previous run crashed in a different test:

https://github.com/microsoft/STL/runs/952678437

[BillyONeal]

Yeah the big RAM spikes are all from ranges_ tests so maybe not a /analyze regression hitting us here.

[StephanTLavavej]

After merging #1185, the sporadic failures appear to have vanished (I looked through all of the runs, and the failures all appeared to be actual PR flaws). Since we didn't find and fix the root cause, just doubled the VM RAM, I'm going to close this as not reproducible instead of fixed.