From bfd4cb5217b0b784619f983f83ef33e3507ca376 Mon Sep 17 00:00:00 2001
From: Casey Carter <Casey@Carter.net>
Date: Fri, 6 Dec 2024 13:24:02 -0800
Subject: [PATCH 1/4] Stop using ctest

It's pointless to use ctest to run one of two different LIT commands when we can simply run LIT directly. This will make it easier to add future tests: we can simply add more targets, we won't need to jump through hoops to tell ctest not to run the new test all the time. This change will ease adding a new EDG-only test target for the IntelliSense team.

Detailed changes:
* In `tests/CMakeLists.txt`:
  * Replace `add_test` targets with `add_custom_target`s. The `stl` and `stlasan` tests are now `test` and `extra-asan-tests` targets, respectively.
  * Replace `list(APPEND unset_variable_name a b c d)` with the equivalent `set(unset_variable_name a b c d)` which makes it clear that we expect `unset_variable_name` to be undefined.
  * Merge `${Python_EXECUTABLE}` into `STL_LIT_COMMAND`, and extract `${STL_LIT_TEST_DIRS}`. The makes it easy to reuse `${STL_LIT_COMMAND}` and `${STL_LIT_TEST_DIRS}` with target-specific LIT options.
  * Note that `USES_TERMINAL` in the `add_custom_target`s makes ninja run those targets serially, maintaining the observable effects of the removed `set_tests_properties(stl stlasan PROPERTIES RUN_SERIAL ON)`.
* Remove `enable_testing` from the top-level `CMakeLists.txt`; it does nothing now that we have no `add_test`s. We still use `BUILD_TESTING` to control including the test targets in the generated build system, but I'm not certain there's a good reason to do so. Maybe to enable folks without python to build but not test the STL?
* In `run-tests.yml`, replace `ctestOptions` parameter with a `testTargets` parameter, a list of targets to build via `cmake --build meow --target woof quack`. This needs to be set in `asan-pipeline.yml` and plumbed through `build-and-test.yml` with a default of `test` to run the vanilla STL tests.
* Remove/Replace the `ctest` discussion in `README.md`.
* Drive-by: Report that the `psutil` python module is unavailable at most once per `stl-lit` run. Seeing three repetitions in the CI logs bothered me.
---
 CMakeLists.txt                  |  1 -
 README.md                       | 43 +++++++++------------------------
 azure-devops/asan-pipeline.yml  |  4 +--
 azure-devops/build-and-test.yml |  6 ++---
 azure-devops/run-tests.yml      |  4 +--
 tests/CMakeLists.txt            | 19 +++++----------
 tests/utils/stl/test/params.py  |  6 +++--
 7 files changed, 29 insertions(+), 54 deletions(-)

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 9d6f36924ef..40775a257f2 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -118,7 +118,6 @@ set(VCLIBS_DEBUG_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:/Od>")
 set(VCLIBS_RELEASE_OPTIONS "$<$<COMPILE_LANGUAGE:CXX>:/O2;/Os>")
 
 if(BUILD_TESTING)
-    enable_testing()
     add_subdirectory(tests)
 endif()
 
diff --git a/README.md b/README.md
index 354a729e36e..93fb9052b15 100644
--- a/README.md
+++ b/README.md
@@ -271,9 +271,12 @@ C:\Users\username\Desktop>dumpbin /DEPENDENTS .\example.exe | findstr msvcp
 
 ## Running All The Tests
 
-After configuring and building the project, running `ctest` from the build output directory will run all the tests.
-CTest will only display the standard error output of tests that failed. In order to get more details from CTest's
-`lit` invocations, run the tests with `ctest -V`.
+The CMake build defines a few test targets that provide the simplest mechanism for running lots of tests.
+After configuring and building the project, you can build:
+* the plain `test` target to run the "normal" set of tests (the tests run by the
+  [STL-CI][STL-CI-link] pipeline), and/or
+* the `extra-asan-tests` target to run the tests with ASan coverage (as run by the
+  [STL-ASan-CI][STL-ASan-CI-link] pipeline).
 
 ## Running A Subset Of The Tests
 
@@ -286,13 +289,13 @@ under a category in libcxx, or running a single test in `std` and `tr1`.
 These examples assume that your current directory is `C:\Dev\STL\out\x64`.
 
 * This command will run all of the test suites with verbose output.
-  + `ctest -V`
-* This command will also run all of the test suites.
+  + `ninja test extra-asan-tests`
+* This command will run all of the test suites.
   + `python tests\utils\stl-lit\stl-lit.py ..\..\llvm-project\libcxx\test ..\..\tests\std ..\..\tests\tr1`
 * This command will run all of the std test suite.
   + `python tests\utils\stl-lit\stl-lit.py ..\..\tests\std`
 * If you want to run a subset of a test suite, you need to point it to the right place in the sources. The following
-will run the single test found under VSO_0000000_any_calling_conventions.
+will run the single test found under `VSO_0000000_any_calling_conventions`.
   + `python tests\utils\stl-lit\stl-lit.py ..\..\tests\std\tests\VSO_0000000_any_calling_conventions`
 * You can invoke `stl-lit` with any arbitrary subdirectory of a test suite. In libcxx this allows you to have finer
 control over what category of tests you would like to run. The following will run all the libcxx map tests.
@@ -305,31 +308,9 @@ control over what category of tests you would like to run. The following will ru
 
 ## Interpreting The Results Of Tests
 
-### CTest
-
-When running the tests via CTest, all of the test suites are considered to be a single test. If any single test in a
-test suite fails, CTest will simply report that the `stl` test failed.
-
-Example:
-```
-0% tests passed, 1 tests failed out of 1
-
-Total Test time (real) = 2441.55 sec
-
-The following tests FAILED:
-      1 - stl (Failed)
-```
-
-The primary utility of CTest in this case is to conveniently invoke `stl-lit.py` with the correct set of arguments.
-
-CTest will output everything that was sent to stderr for each of the failed test suites, which can be used to identify
-which individual test within the test suite failed. It can sometimes be helpful to run CTest with the `-V` option in
-order to see the stdout of the tests.
-
-### stl-lit
-
-When running the tests directly via the generated `stl-lit.py` script the result of each test will be printed. The
-format of each result is `{Result Code}: {Test Suite Name} :: {Test Name}:{Configuration Number}`.
+Whether running the tests via building the cmake test targets or directly via the generated `stl-lit.py` script, the
+result of each test will be printed. The format of each result is
+`{Result Code}: {Test Suite Name} :: {Test Name}:{Configuration Number}`.
 
 Example:
 ```
diff --git a/azure-devops/asan-pipeline.yml b/azure-devops/asan-pipeline.yml
index 5107cd41b4a..3cd92cdf93e 100644
--- a/azure-devops/asan-pipeline.yml
+++ b/azure-devops/asan-pipeline.yml
@@ -27,7 +27,7 @@ stages:
           hostArch: x64
           targetArch: x64
           asanBuild: true
-          ctestOptions: '--tests-regex stlasan'
+          testTargets: extra-asan-tests
 
   - stage: Build_And_Test_x86
     displayName: 'Build and Test x86'
@@ -41,6 +41,6 @@ stages:
           hostArch: x86
           targetArch: x86
           asanBuild: true
-          ctestOptions: '--tests-regex stlasan'
+          testTargets: extra-asan-tests
 
   # no coverage for ARM and ARM64
diff --git a/azure-devops/build-and-test.yml b/azure-devops/build-and-test.yml
index 6798be4d1b7..d3526a6fdc6 100644
--- a/azure-devops/build-and-test.yml
+++ b/azure-devops/build-and-test.yml
@@ -15,9 +15,9 @@ parameters:
 - name: buildBenchmarks
   type: boolean
   default: false
-- name: ctestOptions
+- name: testTargets
   type: string
-  default: '--exclude-regex stlasan'
+  default: 'test'
 - name: numShards
   type: number
   default: 8
@@ -61,5 +61,5 @@ jobs:
     parameters:
       hostArch: ${{ parameters.hostArch }}
       targetArch: ${{ parameters.targetArch }}
-      ctestOptions: ${{ parameters.ctestOptions }}
+      testTargets: ${{ parameters.testTargets }}
       skipTesting: ${{ parameters.skipTesting }}
diff --git a/azure-devops/run-tests.yml b/azure-devops/run-tests.yml
index 27ce640f32f..a6e2c1363d3 100644
--- a/azure-devops/run-tests.yml
+++ b/azure-devops/run-tests.yml
@@ -6,7 +6,7 @@ parameters:
   type: string
 - name: targetArch
   type: string
-- name: ctestOptions
+- name: testTargets
   type: string
 - name: skipTesting
   type: boolean
@@ -14,7 +14,7 @@ steps:
 - script: |
     call "%ProgramFiles%\Microsoft Visual Studio\2022\Preview\Common7\Tools\VsDevCmd.bat" ^
     -host_arch=${{ parameters.hostArch }} -arch=${{ parameters.targetArch }} -no_logo
-    ctest --verbose ${{ parameters.ctestOptions }}
+    ninja --verbose -k 0 ${{ parameters.testTargets }}
   displayName: 'Build and Run Tests'
   timeoutInMinutes: 30
   condition: and(succeeded(), not(${{ parameters.skipTesting }}))
diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 9dc352a0f25..d873c05d396 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -26,19 +26,12 @@ add_subdirectory(utils/stl-lit)
 find_package(Python "3.13" REQUIRED COMPONENTS Interpreter)
 
 if(NOT DEFINED LIT_FLAGS)
-    list(APPEND LIT_FLAGS "-o" "${CMAKE_CURRENT_BINARY_DIR}/test_results.json")
+    set(LIT_FLAGS "-o" "${CMAKE_CURRENT_BINARY_DIR}/test_results.json")
 endif()
 
+set(STL_LIT_COMMAND ${Python_EXECUTABLE} ${STL_LIT_OUTPUT} ${LIT_FLAGS})
 get_property(STL_LIT_TEST_DIRS GLOBAL PROPERTY STL_LIT_TEST_DIRS)
-list(APPEND STL_LIT_COMMAND "${STL_LIT_OUTPUT}"
-                            "${LIT_FLAGS}"
-                            "-D" "notags=ASAN"
-                            "${STL_LIT_TEST_DIRS}")
-list(APPEND STLASAN_LIT_COMMAND "${STL_LIT_OUTPUT}"
-                                "${LIT_FLAGS}"
-                                "-D" "tags=ASAN"
-                                "${STL_LIT_TEST_DIRS}")
-
-add_test(NAME stl COMMAND ${Python_EXECUTABLE} ${STL_LIT_COMMAND} COMMAND_EXPAND_LISTS)
-add_test(NAME stlasan COMMAND ${Python_EXECUTABLE} ${STLASAN_LIT_COMMAND} COMMAND_EXPAND_LISTS)
-set_tests_properties(stl stlasan PROPERTIES RUN_SERIAL ON)
+
+add_custom_target(test COMMAND ${STL_LIT_COMMAND} -Dnotags=ASAN ${STL_LIT_TEST_DIRS} USES_TERMINAL)
+add_custom_target(extra-asan-tests COMMAND ${STL_LIT_COMMAND} -Dtags=ASAN ${STL_LIT_TEST_DIRS} USES_TERMINAL)
+# add_custom_target(only-edg-tests COMMAND ${STL_LIT_COMMAND} -Dnotags=ASAN --just-BE-things ${STL_LIT_TEST_DIRS} USES_TERMINAL)
diff --git a/tests/utils/stl/test/params.py b/tests/utils/stl/test/params.py
index d032839beaf..c3dd091d5d5 100644
--- a/tests/utils/stl/test/params.py
+++ b/tests/utils/stl/test/params.py
@@ -49,8 +49,10 @@ def beNice(prio: str) -> list[ConfigAction]:
     }
     psutil.Process().nice(priority_map[prio])
   except ImportError:
-    import sys
-    print(f'NOTE: Module "psutil" is not installed, so the priority setting "{prio}" has no effect.', file=sys.stderr)
+    if not hasattr(beNice, 'suppress'):
+      import sys
+      print(f'NOTE: Module "psutil" is not installed, so the priority setting "{prio}" has no effect.', file=sys.stderr)
+      beNice.suppress = True
   return []
 
 

From aed1b48bc47ba9f14e0127bc2a5ce92d7a16dcda Mon Sep 17 00:00:00 2001
From: Casey Carter <Casey@Carter.net>
Date: Mon, 9 Dec 2024 11:49:17 -0800
Subject: [PATCH 2/4] Remove placeholder comment for EDG-only test target

---
 tests/CMakeLists.txt | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index d873c05d396..5193b393164 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -34,4 +34,3 @@ get_property(STL_LIT_TEST_DIRS GLOBAL PROPERTY STL_LIT_TEST_DIRS)
 
 add_custom_target(test COMMAND ${STL_LIT_COMMAND} -Dnotags=ASAN ${STL_LIT_TEST_DIRS} USES_TERMINAL)
 add_custom_target(extra-asan-tests COMMAND ${STL_LIT_COMMAND} -Dtags=ASAN ${STL_LIT_TEST_DIRS} USES_TERMINAL)
-# add_custom_target(only-edg-tests COMMAND ${STL_LIT_COMMAND} -Dnotags=ASAN --just-BE-things ${STL_LIT_TEST_DIRS} USES_TERMINAL)

From 1529e672bd5e0a9eb8a2073bb621d4202b013acd Mon Sep 17 00:00:00 2001
From: Casey Carter <Casey@Carter.net>
Date: Mon, 9 Dec 2024 13:54:12 -0800
Subject: [PATCH 3/4] Rename (test, extra-asan-tests) to (STL-CI, STL-ASan-CI)

---
 README.md                       | 39 ++++++++++++++-------------------
 azure-devops/asan-pipeline.yml  |  4 ++--
 azure-devops/build-and-test.yml |  2 +-
 tests/CMakeLists.txt            |  4 ++--
 4 files changed, 22 insertions(+), 27 deletions(-)

diff --git a/README.md b/README.md
index 93fb9052b15..f91492457f8 100644
--- a/README.md
+++ b/README.md
@@ -269,37 +269,33 @@ C:\Users\username\Desktop>dumpbin /DEPENDENTS .\example.exe | findstr msvcp
     * Otherwise, use [LLVM's installer][] and choose to add LLVM to your `PATH` during installation.
 4. Follow the instructions below.
 
-## Running All The Tests
+## Running The Tests
 
-The CMake build defines a few test targets that provide the simplest mechanism for running lots of tests.
-After configuring and building the project, you can build:
-* the plain `test` target to run the "normal" set of tests (the tests run by the
-  [STL-CI][STL-CI-link] pipeline), and/or
-* the `extra-asan-tests` target to run the tests with ASan coverage (as run by the
-  [STL-ASan-CI][STL-ASan-CI-link] pipeline).
+Our tests are currently split across three test suites that are located at `tests\std`, `tests\tr1`, and
+`llvm-project\libcxx\test\std`. The test runner `${PROJECT_BINARY_DIR}\tests\utils\stl-lit\stl-lit.py` accepts paths to
+directories in the test suites and runs all tests located in the subtree rooted at those paths. This can mean executing
+the entirety of a single test suite, running all tests under a category in `libcxx`, or running a single test in `std`
+and `tr1`.
 
-## Running A Subset Of The Tests
-
-`${PROJECT_BINARY_DIR}\tests\utils\stl-lit\stl-lit.py` can be invoked on a subdirectory of a test suite and will execute
-all the tests under that subdirectory. This can mean executing the entirety of a single test suite, running all tests
-under a category in libcxx, or running a single test in `std` and `tr1`.
+Some useful `stl-lit.py` options:
+* `-v` (verbose) tells `stl-lit.py` to show us output from failed test cases.
+* `-Dnotags=ASAN` disables the "extra ASAn configs" that we typically run only in CI. This is useful to limit runtime
+  for full validation runs, but often omitted when running just a few test cases to enable the extra ASAn coverage.
 
 ## Examples
 
 These examples assume that your current directory is `C:\Dev\STL\out\x64`.
 
-* This command will run all of the test suites with verbose output.
-  + `ninja test extra-asan-tests`
-* This command will run all of the test suites.
-  + `python tests\utils\stl-lit\stl-lit.py ..\..\llvm-project\libcxx\test ..\..\tests\std ..\..\tests\tr1`
-* This command will run all of the std test suite.
-  + `python tests\utils\stl-lit\stl-lit.py ..\..\tests\std`
+* This command will run all of the test suites:
+  + `python tests\utils\stl-lit\stl-lit.py -Dnotags=ASAN ..\..\llvm-project\libcxx\test ..\..\tests\std ..\..\tests\tr1`
+* This command will run only the std test suite.
+  + `python tests\utils\stl-lit\stl-lit.py -Dnotags=ASAN ..\..\tests\std`
 * If you want to run a subset of a test suite, you need to point it to the right place in the sources. The following
 will run the single test found under `VSO_0000000_any_calling_conventions`.
-  + `python tests\utils\stl-lit\stl-lit.py ..\..\tests\std\tests\VSO_0000000_any_calling_conventions`
+  + `python tests\utils\stl-lit\stl-lit.py -Dnotags=ASAN ..\..\tests\std\tests\VSO_0000000_any_calling_conventions`
 * You can invoke `stl-lit` with any arbitrary subdirectory of a test suite. In libcxx this allows you to have finer
 control over what category of tests you would like to run. The following will run all the libcxx map tests.
-  + `python tests\utils\stl-lit\stl-lit.py ..\..\llvm-project\libcxx\test\std\containers\associative\map`
+  + `python tests\utils\stl-lit\stl-lit.py -Dnotags=ASAN ..\..\llvm-project\libcxx\test\std\containers\associative\map`
 * You can also use the `--filter` option to include tests whose names match a regular expression. The following
   command will run tests with "atomic_wait" in their names in both the std and libcxx test suites.
   + `python tests\utils\stl-lit\stl-lit.py ..\..\llvm-project\libcxx\test ..\..\tests\std --filter=atomic_wait`
@@ -308,8 +304,7 @@ control over what category of tests you would like to run. The following will ru
 
 ## Interpreting The Results Of Tests
 
-Whether running the tests via building the cmake test targets or directly via the generated `stl-lit.py` script, the
-result of each test will be printed. The format of each result is
+`stl-lit.py` prints the result of each test. The format of each result is
 `{Result Code}: {Test Suite Name} :: {Test Name}:{Configuration Number}`.
 
 Example:
diff --git a/azure-devops/asan-pipeline.yml b/azure-devops/asan-pipeline.yml
index 3cd92cdf93e..f98953de548 100644
--- a/azure-devops/asan-pipeline.yml
+++ b/azure-devops/asan-pipeline.yml
@@ -27,7 +27,7 @@ stages:
           hostArch: x64
           targetArch: x64
           asanBuild: true
-          testTargets: extra-asan-tests
+          testTargets: STL-ASan-CI
 
   - stage: Build_And_Test_x86
     displayName: 'Build and Test x86'
@@ -41,6 +41,6 @@ stages:
           hostArch: x86
           targetArch: x86
           asanBuild: true
-          testTargets: extra-asan-tests
+          testTargets: STL-ASan-CI
 
   # no coverage for ARM and ARM64
diff --git a/azure-devops/build-and-test.yml b/azure-devops/build-and-test.yml
index d3526a6fdc6..edc9134cd93 100644
--- a/azure-devops/build-and-test.yml
+++ b/azure-devops/build-and-test.yml
@@ -17,7 +17,7 @@ parameters:
   default: false
 - name: testTargets
   type: string
-  default: 'test'
+  default: 'STL-CI'
 - name: numShards
   type: number
   default: 8
diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt
index 5193b393164..980ddcf194b 100644
--- a/tests/CMakeLists.txt
+++ b/tests/CMakeLists.txt
@@ -32,5 +32,5 @@ endif()
 set(STL_LIT_COMMAND ${Python_EXECUTABLE} ${STL_LIT_OUTPUT} ${LIT_FLAGS})
 get_property(STL_LIT_TEST_DIRS GLOBAL PROPERTY STL_LIT_TEST_DIRS)
 
-add_custom_target(test COMMAND ${STL_LIT_COMMAND} -Dnotags=ASAN ${STL_LIT_TEST_DIRS} USES_TERMINAL)
-add_custom_target(extra-asan-tests COMMAND ${STL_LIT_COMMAND} -Dtags=ASAN ${STL_LIT_TEST_DIRS} USES_TERMINAL)
+add_custom_target(STL-CI COMMAND ${STL_LIT_COMMAND} -Dnotags=ASAN ${STL_LIT_TEST_DIRS} USES_TERMINAL)
+add_custom_target(STL-ASan-CI COMMAND ${STL_LIT_COMMAND} -Dtags=ASAN ${STL_LIT_TEST_DIRS} USES_TERMINAL)

From 26a62c8c0e5e0f7b290fb45c6adc480723f6a76b Mon Sep 17 00:00:00 2001
From: Casey Carter <Casey@Carter.net>
Date: Mon, 9 Dec 2024 14:50:58 -0800
Subject: [PATCH 4/4] tpyo

---
 README.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/README.md b/README.md
index f91492457f8..681caeb8183 100644
--- a/README.md
+++ b/README.md
@@ -279,8 +279,8 @@ and `tr1`.
 
 Some useful `stl-lit.py` options:
 * `-v` (verbose) tells `stl-lit.py` to show us output from failed test cases.
-* `-Dnotags=ASAN` disables the "extra ASAn configs" that we typically run only in CI. This is useful to limit runtime
-  for full validation runs, but often omitted when running just a few test cases to enable the extra ASAn coverage.
+* `-Dnotags=ASAN` disables the "extra ASan configs" that we typically run only in CI. This is useful to limit runtime
+  for full validation runs, but often omitted when running just a few test cases to enable the extra ASan coverage.
 
 ## Examples