JIT: Move profile checking back until just before inlining

[AndyAyersMS]

Fixes the following areas with proper profile updates:

• GDV chaining
• instrumentation-introduces flow
• OSR step blocks
• fgSplitEdge (used by instrumentation)

Adds checking bypasses for:

• callfinally pair tails
• original method entries in OSR methods

Contributes to #93020

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

@amanasifkhalid PTAL
cc @dotnet/jit-contrib

Modest number of diffs from some of the profile changes.

FYI getting this checking past inlining is going to require some more work. Need to think about it a bit.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

Stress shows a number of related failures, so this needs more work.

[amanasifkhalid]

Preliminary pass LGTM

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Pull request contains merge conflicts.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

[amanasifkhalid]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

I suppose if inlining and block layout are most likely to benefit from a robust profile, and your current work has ensured profiles are consistent for the majority of methods up to inlining, then maybe it makes sense to scale back our broader goals for profile consistency to just up to inlining? We can always just re-run profile repair right before doing layout.

[AndyAyersMS]

jit-format is failing because we can't extract the compile commands...

;; linux Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -[18:02:22] Running: /tmp/tmpk8nyid8a/jitutils/bin/jit-format -a x64 -b Checked -o linux -r /__w/1/s --verbose --projects dll --cross
Formatting jit directory.
Can't find compile_commands.json file. Running configure.
Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -cross
There was an error running CMake to generate compile_commands.json. Please run build-runtime.sh configureonly
;; windows
Formatting jit directory.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Running: D:\a\_work\1\s\src\coreclr\build-runtime.cmd x64 Checked -configureonly -ninja
BUILD: Starting Build at 18:09:12.60
BUILD: Commencing CoreCLR product build
BUILD: Checking prerequisites
BUILD: Using CMake from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
BUILD: Number of processor cores 2
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
BUILD: Commencing build of native components for windows.x64.Checked
BUILD: Using environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILD: Regenerating the Visual Studio solution
Calling "D:\a\_work\1\s\eng\native\gen-buildsys.cmd" "D:\a\_work\1\s\src\coreclr" "D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Checked" vs2022 x64 windows "-DCMAKE_BUILD_TYPE=Checked" "-DCLR_CMAKE_TARGET_ARCH=x64" "-DCLR_CMAKE_TARGET_OS=windows" "-DCLI_CMAKE_FALLBACK_OS=win10" "-DCLR_CMAKE_PGO_INSTRUMENT=0" "-DCLR_CMAKE_OPTDATA_PATH=" "-DCLR_CMAKE_PGO_OPTIMIZE=0" Not searching for unused variables given on the command line.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/ml64.exe
-- Performing Test HAVE_DIRENT_D_TYPE
-- Performing Test HAVE_DIRENT_D_TYPE - Failed
-- Looking for getauxval

Looks like this is getting fixed in #101297.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

libraries-jitstress and libraries-pgo seem like they may be unrelated.

But pgostress is related. Will keep trying to repro this locally.

[AndyAyersMS]

Finally caught one

BB77 and BB80 are mis-weighted (likely just BB80) -- seems like multi-guess GDV is not quite right.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

@amanasifkhalid think this is ready...

[amanasifkhalid]

LGTM, thanks for seeing this through!

We briefly discussed not continuing this validation past inlining for now, as inlining and block layout (and hot-cold splitting) are probably the opt passes that benefit most from high-quality profile data, and we can simply re-run profile repair right before the latter to ensure it has a useful profile to work with. Plus, the stress failures these consistency checks have triggered seem to be a pain to diagnose. Are you planning on stopping here for now?

[amanasifkhalid]

Suggested change

	// No matter what, the minimum weight is zero
	// No matter what, the minimum weight is zero

[amanasifkhalid]

libraries-pgo failure looks unrelated, and SPMI failures are #101070.

[AndyAyersMS]

Are you planning on stopping here for now?

I will probably keep pushing on this for the next week or two; would be nice to be able to validate more of the flow optimizations (which generally should have local repairs).

Also will fix the comment issue you noted above.

[AndyAyersMS]

Fixes the following areas with proper profile updates:

• GDV chaining
• instrumentation-introduces flow
• OSR step blocks
• fgSplitEdge (used by instrumentation)

Adds checking bypasses for:

• callfinally pair tails
• original method entries in OSR methods

Contributes to #93020

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

@amanasifkhalid PTAL
cc @dotnet/jit-contrib

Modest number of diffs from some of the profile changes.

FYI getting this checking past inlining is going to require some more work. Need to think about it a bit.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

Stress shows a number of related failures, so this needs more work.

[amanasifkhalid]

Preliminary pass LGTM

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Pull request contains merge conflicts.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

[amanasifkhalid]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

I suppose if inlining and block layout are most likely to benefit from a robust profile, and your current work has ensured profiles are consistent for the majority of methods up to inlining, then maybe it makes sense to scale back our broader goals for profile consistency to just up to inlining? We can always just re-run profile repair right before doing layout.

[AndyAyersMS]

jit-format is failing because we can't extract the compile commands...

;; linux Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -[18:02:22] Running: /tmp/tmpk8nyid8a/jitutils/bin/jit-format -a x64 -b Checked -o linux -r /__w/1/s --verbose --projects dll --cross
Formatting jit directory.
Can't find compile_commands.json file. Running configure.
Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -cross
There was an error running CMake to generate compile_commands.json. Please run build-runtime.sh configureonly
;; windows
Formatting jit directory.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Running: D:\a\_work\1\s\src\coreclr\build-runtime.cmd x64 Checked -configureonly -ninja
BUILD: Starting Build at 18:09:12.60
BUILD: Commencing CoreCLR product build
BUILD: Checking prerequisites
BUILD: Using CMake from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
BUILD: Number of processor cores 2
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
BUILD: Commencing build of native components for windows.x64.Checked
BUILD: Using environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILD: Regenerating the Visual Studio solution
Calling "D:\a\_work\1\s\eng\native\gen-buildsys.cmd" "D:\a\_work\1\s\src\coreclr" "D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Checked" vs2022 x64 windows "-DCMAKE_BUILD_TYPE=Checked" "-DCLR_CMAKE_TARGET_ARCH=x64" "-DCLR_CMAKE_TARGET_OS=windows" "-DCLI_CMAKE_FALLBACK_OS=win10" "-DCLR_CMAKE_PGO_INSTRUMENT=0" "-DCLR_CMAKE_OPTDATA_PATH=" "-DCLR_CMAKE_PGO_OPTIMIZE=0" Not searching for unused variables given on the command line.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/ml64.exe
-- Performing Test HAVE_DIRENT_D_TYPE
-- Performing Test HAVE_DIRENT_D_TYPE - Failed
-- Looking for getauxval

Looks like this is getting fixed in #101297.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

libraries-jitstress and libraries-pgo seem like they may be unrelated.

But pgostress is related. Will keep trying to repro this locally.

[AndyAyersMS]

Finally caught one

BB77 and BB80 are mis-weighted (likely just BB80) -- seems like multi-guess GDV is not quite right.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

@amanasifkhalid think this is ready...

[amanasifkhalid]

LGTM, thanks for seeing this through!

We briefly discussed not continuing this validation past inlining for now, as inlining and block layout (and hot-cold splitting) are probably the opt passes that benefit most from high-quality profile data, and we can simply re-run profile repair right before the latter to ensure it has a useful profile to work with. Plus, the stress failures these consistency checks have triggered seem to be a pain to diagnose. Are you planning on stopping here for now?

[amanasifkhalid]

Suggested change

	// No matter what, the minimum weight is zero
	// No matter what, the minimum weight is zero

[amanasifkhalid]

libraries-pgo failure looks unrelated, and SPMI failures are #101070.

[AndyAyersMS]

Are you planning on stopping here for now?

I will probably keep pushing on this for the next week or two; would be nice to be able to validate more of the flow optimizations (which generally should have local repairs).

Also will fix the comment issue you noted above.

[AndyAyersMS]

Fixes the following areas with proper profile updates:

• GDV chaining
• instrumentation-introduces flow
• OSR step blocks
• fgSplitEdge (used by instrumentation)

Adds checking bypasses for:

• callfinally pair tails
• original method entries in OSR methods

Contributes to #93020

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

@amanasifkhalid PTAL
cc @dotnet/jit-contrib

Modest number of diffs from some of the profile changes.

FYI getting this checking past inlining is going to require some more work. Need to think about it a bit.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

Stress shows a number of related failures, so this needs more work.

[amanasifkhalid]

Preliminary pass LGTM

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Pull request contains merge conflicts.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

[amanasifkhalid]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

I suppose if inlining and block layout are most likely to benefit from a robust profile, and your current work has ensured profiles are consistent for the majority of methods up to inlining, then maybe it makes sense to scale back our broader goals for profile consistency to just up to inlining? We can always just re-run profile repair right before doing layout.

[AndyAyersMS]

jit-format is failing because we can't extract the compile commands...

;; linux Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -[18:02:22] Running: /tmp/tmpk8nyid8a/jitutils/bin/jit-format -a x64 -b Checked -o linux -r /__w/1/s --verbose --projects dll --cross
Formatting jit directory.
Can't find compile_commands.json file. Running configure.
Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -cross
There was an error running CMake to generate compile_commands.json. Please run build-runtime.sh configureonly
;; windows
Formatting jit directory.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Running: D:\a\_work\1\s\src\coreclr\build-runtime.cmd x64 Checked -configureonly -ninja
BUILD: Starting Build at 18:09:12.60
BUILD: Commencing CoreCLR product build
BUILD: Checking prerequisites
BUILD: Using CMake from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
BUILD: Number of processor cores 2
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
BUILD: Commencing build of native components for windows.x64.Checked
BUILD: Using environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILD: Regenerating the Visual Studio solution
Calling "D:\a\_work\1\s\eng\native\gen-buildsys.cmd" "D:\a\_work\1\s\src\coreclr" "D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Checked" vs2022 x64 windows "-DCMAKE_BUILD_TYPE=Checked" "-DCLR_CMAKE_TARGET_ARCH=x64" "-DCLR_CMAKE_TARGET_OS=windows" "-DCLI_CMAKE_FALLBACK_OS=win10" "-DCLR_CMAKE_PGO_INSTRUMENT=0" "-DCLR_CMAKE_OPTDATA_PATH=" "-DCLR_CMAKE_PGO_OPTIMIZE=0" Not searching for unused variables given on the command line.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/ml64.exe
-- Performing Test HAVE_DIRENT_D_TYPE
-- Performing Test HAVE_DIRENT_D_TYPE - Failed
-- Looking for getauxval

Looks like this is getting fixed in #101297.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

libraries-jitstress and libraries-pgo seem like they may be unrelated.

But pgostress is related. Will keep trying to repro this locally.

[AndyAyersMS]

Finally caught one

BB77 and BB80 are mis-weighted (likely just BB80) -- seems like multi-guess GDV is not quite right.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

@amanasifkhalid think this is ready...

[amanasifkhalid]

LGTM, thanks for seeing this through!

We briefly discussed not continuing this validation past inlining for now, as inlining and block layout (and hot-cold splitting) are probably the opt passes that benefit most from high-quality profile data, and we can simply re-run profile repair right before the latter to ensure it has a useful profile to work with. Plus, the stress failures these consistency checks have triggered seem to be a pain to diagnose. Are you planning on stopping here for now?

[amanasifkhalid]

Suggested change

	// No matter what, the minimum weight is zero
	// No matter what, the minimum weight is zero

[amanasifkhalid]

libraries-pgo failure looks unrelated, and SPMI failures are #101070.

[AndyAyersMS]

Are you planning on stopping here for now?

I will probably keep pushing on this for the next week or two; would be nice to be able to validate more of the flow optimizations (which generally should have local repairs).

Also will fix the comment issue you noted above.

[AndyAyersMS]

Fixes the following areas with proper profile updates:

• GDV chaining
• instrumentation-introduces flow
• OSR step blocks
• fgSplitEdge (used by instrumentation)

Adds checking bypasses for:

• callfinally pair tails
• original method entries in OSR methods

Contributes to #93020

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

@amanasifkhalid PTAL
cc @dotnet/jit-contrib

Modest number of diffs from some of the profile changes.

FYI getting this checking past inlining is going to require some more work. Need to think about it a bit.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

Stress shows a number of related failures, so this needs more work.

[amanasifkhalid]

Preliminary pass LGTM

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Pull request contains merge conflicts.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

[amanasifkhalid]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

I suppose if inlining and block layout are most likely to benefit from a robust profile, and your current work has ensured profiles are consistent for the majority of methods up to inlining, then maybe it makes sense to scale back our broader goals for profile consistency to just up to inlining? We can always just re-run profile repair right before doing layout.

[AndyAyersMS]

jit-format is failing because we can't extract the compile commands...

;; linux Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -[18:02:22] Running: /tmp/tmpk8nyid8a/jitutils/bin/jit-format -a x64 -b Checked -o linux -r /__w/1/s --verbose --projects dll --cross
Formatting jit directory.
Can't find compile_commands.json file. Running configure.
Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -cross
There was an error running CMake to generate compile_commands.json. Please run build-runtime.sh configureonly
;; windows
Formatting jit directory.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Running: D:\a\_work\1\s\src\coreclr\build-runtime.cmd x64 Checked -configureonly -ninja
BUILD: Starting Build at 18:09:12.60
BUILD: Commencing CoreCLR product build
BUILD: Checking prerequisites
BUILD: Using CMake from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
BUILD: Number of processor cores 2
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
BUILD: Commencing build of native components for windows.x64.Checked
BUILD: Using environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILD: Regenerating the Visual Studio solution
Calling "D:\a\_work\1\s\eng\native\gen-buildsys.cmd" "D:\a\_work\1\s\src\coreclr" "D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Checked" vs2022 x64 windows "-DCMAKE_BUILD_TYPE=Checked" "-DCLR_CMAKE_TARGET_ARCH=x64" "-DCLR_CMAKE_TARGET_OS=windows" "-DCLI_CMAKE_FALLBACK_OS=win10" "-DCLR_CMAKE_PGO_INSTRUMENT=0" "-DCLR_CMAKE_OPTDATA_PATH=" "-DCLR_CMAKE_PGO_OPTIMIZE=0" Not searching for unused variables given on the command line.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/ml64.exe
-- Performing Test HAVE_DIRENT_D_TYPE
-- Performing Test HAVE_DIRENT_D_TYPE - Failed
-- Looking for getauxval

Looks like this is getting fixed in #101297.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

libraries-jitstress and libraries-pgo seem like they may be unrelated.

But pgostress is related. Will keep trying to repro this locally.

[AndyAyersMS]

Finally caught one

BB77 and BB80 are mis-weighted (likely just BB80) -- seems like multi-guess GDV is not quite right.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

@amanasifkhalid think this is ready...

[amanasifkhalid]

LGTM, thanks for seeing this through!

We briefly discussed not continuing this validation past inlining for now, as inlining and block layout (and hot-cold splitting) are probably the opt passes that benefit most from high-quality profile data, and we can simply re-run profile repair right before the latter to ensure it has a useful profile to work with. Plus, the stress failures these consistency checks have triggered seem to be a pain to diagnose. Are you planning on stopping here for now?

[amanasifkhalid]

Suggested change

	// No matter what, the minimum weight is zero
	// No matter what, the minimum weight is zero

[amanasifkhalid]

libraries-pgo failure looks unrelated, and SPMI failures are #101070.

[AndyAyersMS]

Are you planning on stopping here for now?

I will probably keep pushing on this for the next week or two; would be nice to be able to validate more of the flow optimizations (which generally should have local repairs).

Also will fix the comment issue you noted above.

[AndyAyersMS]

Fixes the following areas with proper profile updates:

• GDV chaining
• instrumentation-introduces flow
• OSR step blocks
• fgSplitEdge (used by instrumentation)

Adds checking bypasses for:

• callfinally pair tails
• original method entries in OSR methods

Contributes to #93020

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

@amanasifkhalid PTAL
cc @dotnet/jit-contrib

Modest number of diffs from some of the profile changes.

FYI getting this checking past inlining is going to require some more work. Need to think about it a bit.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

Stress shows a number of related failures, so this needs more work.

[amanasifkhalid]

Preliminary pass LGTM

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Pull request contains merge conflicts.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

[amanasifkhalid]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

I suppose if inlining and block layout are most likely to benefit from a robust profile, and your current work has ensured profiles are consistent for the majority of methods up to inlining, then maybe it makes sense to scale back our broader goals for profile consistency to just up to inlining? We can always just re-run profile repair right before doing layout.

[AndyAyersMS]

jit-format is failing because we can't extract the compile commands...

;; linux Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -[18:02:22] Running: /tmp/tmpk8nyid8a/jitutils/bin/jit-format -a x64 -b Checked -o linux -r /__w/1/s --verbose --projects dll --cross
Formatting jit directory.
Can't find compile_commands.json file. Running configure.
Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -cross
There was an error running CMake to generate compile_commands.json. Please run build-runtime.sh configureonly
;; windows
Formatting jit directory.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Running: D:\a\_work\1\s\src\coreclr\build-runtime.cmd x64 Checked -configureonly -ninja
BUILD: Starting Build at 18:09:12.60
BUILD: Commencing CoreCLR product build
BUILD: Checking prerequisites
BUILD: Using CMake from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
BUILD: Number of processor cores 2
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
BUILD: Commencing build of native components for windows.x64.Checked
BUILD: Using environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILD: Regenerating the Visual Studio solution
Calling "D:\a\_work\1\s\eng\native\gen-buildsys.cmd" "D:\a\_work\1\s\src\coreclr" "D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Checked" vs2022 x64 windows "-DCMAKE_BUILD_TYPE=Checked" "-DCLR_CMAKE_TARGET_ARCH=x64" "-DCLR_CMAKE_TARGET_OS=windows" "-DCLI_CMAKE_FALLBACK_OS=win10" "-DCLR_CMAKE_PGO_INSTRUMENT=0" "-DCLR_CMAKE_OPTDATA_PATH=" "-DCLR_CMAKE_PGO_OPTIMIZE=0" Not searching for unused variables given on the command line.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/ml64.exe
-- Performing Test HAVE_DIRENT_D_TYPE
-- Performing Test HAVE_DIRENT_D_TYPE - Failed
-- Looking for getauxval

Looks like this is getting fixed in #101297.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

libraries-jitstress and libraries-pgo seem like they may be unrelated.

But pgostress is related. Will keep trying to repro this locally.

[AndyAyersMS]

Finally caught one

BB77 and BB80 are mis-weighted (likely just BB80) -- seems like multi-guess GDV is not quite right.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

@amanasifkhalid think this is ready...

[amanasifkhalid]

LGTM, thanks for seeing this through!

We briefly discussed not continuing this validation past inlining for now, as inlining and block layout (and hot-cold splitting) are probably the opt passes that benefit most from high-quality profile data, and we can simply re-run profile repair right before the latter to ensure it has a useful profile to work with. Plus, the stress failures these consistency checks have triggered seem to be a pain to diagnose. Are you planning on stopping here for now?

[amanasifkhalid]

Suggested change

	// No matter what, the minimum weight is zero
	// No matter what, the minimum weight is zero

[amanasifkhalid]

libraries-pgo failure looks unrelated, and SPMI failures are #101070.

[AndyAyersMS]

Are you planning on stopping here for now?

I will probably keep pushing on this for the next week or two; would be nice to be able to validate more of the flow optimizations (which generally should have local repairs).

Also will fix the comment issue you noted above.

[AndyAyersMS]

Fixes the following areas with proper profile updates:

• GDV chaining
• instrumentation-introduces flow
• OSR step blocks
• fgSplitEdge (used by instrumentation)

Adds checking bypasses for:

• callfinally pair tails
• original method entries in OSR methods

Contributes to #93020

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

@amanasifkhalid PTAL
cc @dotnet/jit-contrib

Modest number of diffs from some of the profile changes.

FYI getting this checking past inlining is going to require some more work. Need to think about it a bit.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

Stress shows a number of related failures, so this needs more work.

[amanasifkhalid]

Preliminary pass LGTM

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Pull request contains merge conflicts.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

[amanasifkhalid]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

I suppose if inlining and block layout are most likely to benefit from a robust profile, and your current work has ensured profiles are consistent for the majority of methods up to inlining, then maybe it makes sense to scale back our broader goals for profile consistency to just up to inlining? We can always just re-run profile repair right before doing layout.

[AndyAyersMS]

jit-format is failing because we can't extract the compile commands...

;; linux Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -[18:02:22] Running: /tmp/tmpk8nyid8a/jitutils/bin/jit-format -a x64 -b Checked -o linux -r /__w/1/s --verbose --projects dll --cross
Formatting jit directory.
Can't find compile_commands.json file. Running configure.
Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -cross
There was an error running CMake to generate compile_commands.json. Please run build-runtime.sh configureonly
;; windows
Formatting jit directory.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Running: D:\a\_work\1\s\src\coreclr\build-runtime.cmd x64 Checked -configureonly -ninja
BUILD: Starting Build at 18:09:12.60
BUILD: Commencing CoreCLR product build
BUILD: Checking prerequisites
BUILD: Using CMake from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
BUILD: Number of processor cores 2
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
BUILD: Commencing build of native components for windows.x64.Checked
BUILD: Using environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILD: Regenerating the Visual Studio solution
Calling "D:\a\_work\1\s\eng\native\gen-buildsys.cmd" "D:\a\_work\1\s\src\coreclr" "D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Checked" vs2022 x64 windows "-DCMAKE_BUILD_TYPE=Checked" "-DCLR_CMAKE_TARGET_ARCH=x64" "-DCLR_CMAKE_TARGET_OS=windows" "-DCLI_CMAKE_FALLBACK_OS=win10" "-DCLR_CMAKE_PGO_INSTRUMENT=0" "-DCLR_CMAKE_OPTDATA_PATH=" "-DCLR_CMAKE_PGO_OPTIMIZE=0" Not searching for unused variables given on the command line.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/ml64.exe
-- Performing Test HAVE_DIRENT_D_TYPE
-- Performing Test HAVE_DIRENT_D_TYPE - Failed
-- Looking for getauxval

Looks like this is getting fixed in #101297.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

libraries-jitstress and libraries-pgo seem like they may be unrelated.

But pgostress is related. Will keep trying to repro this locally.

[AndyAyersMS]

Finally caught one

BB77 and BB80 are mis-weighted (likely just BB80) -- seems like multi-guess GDV is not quite right.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

@amanasifkhalid think this is ready...

[amanasifkhalid]

LGTM, thanks for seeing this through!

We briefly discussed not continuing this validation past inlining for now, as inlining and block layout (and hot-cold splitting) are probably the opt passes that benefit most from high-quality profile data, and we can simply re-run profile repair right before the latter to ensure it has a useful profile to work with. Plus, the stress failures these consistency checks have triggered seem to be a pain to diagnose. Are you planning on stopping here for now?

[amanasifkhalid]

Suggested change

	// No matter what, the minimum weight is zero
	// No matter what, the minimum weight is zero

[amanasifkhalid]

libraries-pgo failure looks unrelated, and SPMI failures are #101070.

[AndyAyersMS]

Are you planning on stopping here for now?

I will probably keep pushing on this for the next week or two; would be nice to be able to validate more of the flow optimizations (which generally should have local repairs).

Also will fix the comment issue you noted above.

[AndyAyersMS]

Fixes the following areas with proper profile updates:

• GDV chaining
• instrumentation-introduces flow
• OSR step blocks
• fgSplitEdge (used by instrumentation)

Adds checking bypasses for:

• callfinally pair tails
• original method entries in OSR methods

Contributes to #93020

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

@amanasifkhalid PTAL
cc @dotnet/jit-contrib

Modest number of diffs from some of the profile changes.

FYI getting this checking past inlining is going to require some more work. Need to think about it a bit.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

Stress shows a number of related failures, so this needs more work.

[amanasifkhalid]

Preliminary pass LGTM

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Pull request contains merge conflicts.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

[amanasifkhalid]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

I suppose if inlining and block layout are most likely to benefit from a robust profile, and your current work has ensured profiles are consistent for the majority of methods up to inlining, then maybe it makes sense to scale back our broader goals for profile consistency to just up to inlining? We can always just re-run profile repair right before doing layout.

[AndyAyersMS]

jit-format is failing because we can't extract the compile commands...

;; linux Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -[18:02:22] Running: /tmp/tmpk8nyid8a/jitutils/bin/jit-format -a x64 -b Checked -o linux -r /__w/1/s --verbose --projects dll --cross
Formatting jit directory.
Can't find compile_commands.json file. Running configure.
Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -cross
There was an error running CMake to generate compile_commands.json. Please run build-runtime.sh configureonly
;; windows
Formatting jit directory.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Running: D:\a\_work\1\s\src\coreclr\build-runtime.cmd x64 Checked -configureonly -ninja
BUILD: Starting Build at 18:09:12.60
BUILD: Commencing CoreCLR product build
BUILD: Checking prerequisites
BUILD: Using CMake from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
BUILD: Number of processor cores 2
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
BUILD: Commencing build of native components for windows.x64.Checked
BUILD: Using environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILD: Regenerating the Visual Studio solution
Calling "D:\a\_work\1\s\eng\native\gen-buildsys.cmd" "D:\a\_work\1\s\src\coreclr" "D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Checked" vs2022 x64 windows "-DCMAKE_BUILD_TYPE=Checked" "-DCLR_CMAKE_TARGET_ARCH=x64" "-DCLR_CMAKE_TARGET_OS=windows" "-DCLI_CMAKE_FALLBACK_OS=win10" "-DCLR_CMAKE_PGO_INSTRUMENT=0" "-DCLR_CMAKE_OPTDATA_PATH=" "-DCLR_CMAKE_PGO_OPTIMIZE=0" Not searching for unused variables given on the command line.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/ml64.exe
-- Performing Test HAVE_DIRENT_D_TYPE
-- Performing Test HAVE_DIRENT_D_TYPE - Failed
-- Looking for getauxval

Looks like this is getting fixed in #101297.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

libraries-jitstress and libraries-pgo seem like they may be unrelated.

But pgostress is related. Will keep trying to repro this locally.

[AndyAyersMS]

Finally caught one

BB77 and BB80 are mis-weighted (likely just BB80) -- seems like multi-guess GDV is not quite right.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

@amanasifkhalid think this is ready...

[amanasifkhalid]

LGTM, thanks for seeing this through!

We briefly discussed not continuing this validation past inlining for now, as inlining and block layout (and hot-cold splitting) are probably the opt passes that benefit most from high-quality profile data, and we can simply re-run profile repair right before the latter to ensure it has a useful profile to work with. Plus, the stress failures these consistency checks have triggered seem to be a pain to diagnose. Are you planning on stopping here for now?

[amanasifkhalid]

Suggested change

	// No matter what, the minimum weight is zero
	// No matter what, the minimum weight is zero

[amanasifkhalid]

libraries-pgo failure looks unrelated, and SPMI failures are #101070.

[AndyAyersMS]

Are you planning on stopping here for now?

I will probably keep pushing on this for the next week or two; would be nice to be able to validate more of the flow optimizations (which generally should have local repairs).

Also will fix the comment issue you noted above.

[AndyAyersMS]

Fixes the following areas with proper profile updates:

• GDV chaining
• instrumentation-introduces flow
• OSR step blocks
• fgSplitEdge (used by instrumentation)

Adds checking bypasses for:

• callfinally pair tails
• original method entries in OSR methods

Contributes to #93020

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[AndyAyersMS]

@amanasifkhalid PTAL
cc @dotnet/jit-contrib

Modest number of diffs from some of the profile changes.

FYI getting this checking past inlining is going to require some more work. Need to think about it a bit.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

Stress shows a number of related failures, so this needs more work.

[amanasifkhalid]

Preliminary pass LGTM

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Pull request contains merge conflicts.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[AndyAyersMS]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

[amanasifkhalid]

The failures (at least pgostress so far) are proving difficult to repro. Starting to worry that having this checking on by default is going to be counterproductive, if we keep seeing a long tail of rare cases that don't get properly handled and are hard to capture/reproduce.

I suppose if inlining and block layout are most likely to benefit from a robust profile, and your current work has ensured profiles are consistent for the majority of methods up to inlining, then maybe it makes sense to scale back our broader goals for profile consistency to just up to inlining? We can always just re-run profile repair right before doing layout.

[AndyAyersMS]

jit-format is failing because we can't extract the compile commands...

;; linux Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -[18:02:22] Running: /tmp/tmpk8nyid8a/jitutils/bin/jit-format -a x64 -b Checked -o linux -r /__w/1/s --verbose --projects dll --cross
Formatting jit directory.
Can't find compile_commands.json file. Running configure.
Running: /__w/1/s/src/coreclr/build-runtime.sh x64 Checked configureonly -cmakeargs -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -cross
There was an error running CMake to generate compile_commands.json. Please run build-runtime.sh configureonly
;; windows
Formatting jit directory.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Running: D:\a\_work\1\s\src\coreclr\build-runtime.cmd x64 Checked -configureonly -ninja
BUILD: Starting Build at 18:09:12.60
BUILD: Commencing CoreCLR product build
BUILD: Checking prerequisites
BUILD: Using CMake from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe
BUILD: Number of processor cores 2
1 file(s) copied.
1 file(s) copied.
1 file(s) copied.
BUILD: Commencing build of native components for windows.x64.Checked
BUILD: Using environment: "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.9.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
BUILD: Regenerating the Visual Studio solution
Calling "D:\a\_work\1\s\eng\native\gen-buildsys.cmd" "D:\a\_work\1\s\src\coreclr" "D:\a\_work\1\s\artifacts\obj\coreclr\windows.x64.Checked" vs2022 x64 windows "-DCMAKE_BUILD_TYPE=Checked" "-DCLR_CMAKE_TARGET_ARCH=x64" "-DCLR_CMAKE_TARGET_OS=windows" "-DCLI_CMAKE_FALLBACK_OS=win10" "-DCLR_CMAKE_PGO_INSTRUMENT=0" "-DCLR_CMAKE_OPTDATA_PATH=" "-DCLR_CMAKE_PGO_OPTIMIZE=0" Not searching for unused variables given on the command line.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Enterprise/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/ml64.exe
-- Performing Test HAVE_DIRENT_D_TYPE
-- Performing Test HAVE_DIRENT_D_TYPE - Failed
-- Looking for getauxval

Looks like this is getting fixed in #101297.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

libraries-jitstress and libraries-pgo seem like they may be unrelated.

But pgostress is related. Will keep trying to repro this locally.

[AndyAyersMS]

Finally caught one

BB77 and BB80 are mis-weighted (likely just BB80) -- seems like multi-guess GDV is not quite right.

[AndyAyersMS]

/azp run runtime-coreclr libraries-pgo, runtime-coreclr jitstress, runtime-coreclr pgostress, runtime-coreclr pgo, runtime-coreclr libraries-jitstress

[azure-pipelines]

Azure Pipelines successfully started running 5 pipeline(s).

[AndyAyersMS]

@amanasifkhalid think this is ready...

[amanasifkhalid]

LGTM, thanks for seeing this through!

We briefly discussed not continuing this validation past inlining for now, as inlining and block layout (and hot-cold splitting) are probably the opt passes that benefit most from high-quality profile data, and we can simply re-run profile repair right before the latter to ensure it has a useful profile to work with. Plus, the stress failures these consistency checks have triggered seem to be a pain to diagnose. Are you planning on stopping here for now?

[amanasifkhalid]

Suggested change

	// No matter what, the minimum weight is zero
	// No matter what, the minimum weight is zero

[amanasifkhalid]

libraries-pgo failure looks unrelated, and SPMI failures are #101070.

[AndyAyersMS]

Are you planning on stopping here for now?

I will probably keep pushing on this for the next week or two; would be nice to be able to validate more of the flow optimizations (which generally should have local repairs).

Also will fix the comment issue you noted above.