[MONO] Add JIT_CODE_DEBUG_INFO record functionality for Jitdump

[saitama951]

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

[vargaz]

This should follow the mono coding conventions, see the rest of the file for examples.

[saitama951]

done. let me know if there is anything else

[saitama951]

changed the style to adhere to the mono coding conventions

[saitama951]

done. let me know if there is anything else

[fanyang-mono]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

[fanyang-mono]

Suggested change

	}DebugEntry;
	}DebugEntry;

[fanyang-mono]

Suggested change

	}JitCodeDebug;
	}JitCodeDebug;

[fanyang-mono]

Suggested change

	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);
	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);

[fanyang-mono]

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

[fanyang-mono]

Fix code style

[fanyang-mono]

Suggested change

	if(!(loc)
	if(!loc)

[saitama951]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

[saitama951]

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

[directhex]

@vargaz@lambdageek can someone see if this is ready to merge now?

[shivanirmishra]

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[saitama951]

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

[shivanirmishra]

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

[saitama951]

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

[shivanirmishra]

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

[saitama951]

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

[shivanirmishra]

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

[fanyang-mono]

/backport to release/7.0-staging

[github-actions]

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

[shivanirmishra]

@saitama951@fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

[saitama951]

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

[shivanirmishra]

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

[fanyang-mono]

@saitama951@fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

[fanyang-mono]

@shivanirmishra We will get this fix in the next possible servicing release.

[shivanirmishra]

@saitama951@fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

[saitama951]

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

[shivanirmishra]

Hi @saitama951@fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

[saitama951]

@shivanirmishra have you applied the diff changes ?

[shivanirmishra]

@saitama951 Yes, It's working.
Thank you.

[saitama951]

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

[vargaz]

This should follow the mono coding conventions, see the rest of the file for examples.

[saitama951]

done. let me know if there is anything else

[saitama951]

changed the style to adhere to the mono coding conventions

[saitama951]

done. let me know if there is anything else

[fanyang-mono]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

[fanyang-mono]

Suggested change

	}DebugEntry;
	}DebugEntry;

[fanyang-mono]

Suggested change

	}JitCodeDebug;
	}JitCodeDebug;

[fanyang-mono]

Suggested change

	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);
	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);

[fanyang-mono]

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

[fanyang-mono]

Fix code style

[fanyang-mono]

Suggested change

	if(!(loc)
	if(!loc)

[saitama951]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

[saitama951]

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

[directhex]

@vargaz@lambdageek can someone see if this is ready to merge now?

[shivanirmishra]

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[saitama951]

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

[shivanirmishra]

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

[saitama951]

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

[shivanirmishra]

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

[saitama951]

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

[shivanirmishra]

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

[fanyang-mono]

/backport to release/7.0-staging

[github-actions]

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

[shivanirmishra]

@saitama951@fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

[saitama951]

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

[shivanirmishra]

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

[fanyang-mono]

@saitama951@fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

[fanyang-mono]

@shivanirmishra We will get this fix in the next possible servicing release.

[shivanirmishra]

@saitama951@fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

[saitama951]

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

[shivanirmishra]

Hi @saitama951@fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

[saitama951]

@shivanirmishra have you applied the diff changes ?

[shivanirmishra]

@saitama951 Yes, It's working.
Thank you.

[saitama951]

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

[vargaz]

This should follow the mono coding conventions, see the rest of the file for examples.

[saitama951]

done. let me know if there is anything else

[saitama951]

changed the style to adhere to the mono coding conventions

[saitama951]

done. let me know if there is anything else

[fanyang-mono]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

[fanyang-mono]

Suggested change

	}DebugEntry;
	}DebugEntry;

[fanyang-mono]

Suggested change

	}JitCodeDebug;
	}JitCodeDebug;

[fanyang-mono]

Suggested change

	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);
	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);

[fanyang-mono]

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

[fanyang-mono]

Fix code style

[fanyang-mono]

Suggested change

	if(!(loc)
	if(!loc)

[saitama951]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

[saitama951]

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

[directhex]

@vargaz@lambdageek can someone see if this is ready to merge now?

[shivanirmishra]

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[saitama951]

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

[shivanirmishra]

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

[saitama951]

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

[shivanirmishra]

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

[saitama951]

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

[shivanirmishra]

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

[fanyang-mono]

/backport to release/7.0-staging

[github-actions]

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

[shivanirmishra]

@saitama951@fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

[saitama951]

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

[shivanirmishra]

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

[fanyang-mono]

@saitama951@fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

[fanyang-mono]

@shivanirmishra We will get this fix in the next possible servicing release.

[shivanirmishra]

@saitama951@fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

[saitama951]

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

[shivanirmishra]

Hi @saitama951@fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

[saitama951]

@shivanirmishra have you applied the diff changes ?

[shivanirmishra]

@saitama951 Yes, It's working.
Thank you.

[saitama951]

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

[vargaz]

This should follow the mono coding conventions, see the rest of the file for examples.

[saitama951]

done. let me know if there is anything else

[saitama951]

changed the style to adhere to the mono coding conventions

[saitama951]

done. let me know if there is anything else

[fanyang-mono]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

[fanyang-mono]

Suggested change

	}DebugEntry;
	}DebugEntry;

[fanyang-mono]

Suggested change

	}JitCodeDebug;
	}JitCodeDebug;

[fanyang-mono]

Suggested change

	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);
	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);

[fanyang-mono]

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

[fanyang-mono]

Fix code style

[fanyang-mono]

Suggested change

	if(!(loc)
	if(!loc)

[saitama951]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

[saitama951]

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

[directhex]

@vargaz@lambdageek can someone see if this is ready to merge now?

[shivanirmishra]

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[saitama951]

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

[shivanirmishra]

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

[saitama951]

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

[shivanirmishra]

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

[saitama951]

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

[shivanirmishra]

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

[fanyang-mono]

/backport to release/7.0-staging

[github-actions]

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

[shivanirmishra]

@saitama951@fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

[saitama951]

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

[shivanirmishra]

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

[fanyang-mono]

@saitama951@fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

[fanyang-mono]

@shivanirmishra We will get this fix in the next possible servicing release.

[shivanirmishra]

@saitama951@fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

[saitama951]

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

[shivanirmishra]

Hi @saitama951@fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

[saitama951]

@shivanirmishra have you applied the diff changes ?

[shivanirmishra]

@saitama951 Yes, It's working.
Thank you.

[saitama951]

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

[vargaz]

This should follow the mono coding conventions, see the rest of the file for examples.

[saitama951]

done. let me know if there is anything else

[saitama951]

changed the style to adhere to the mono coding conventions

[saitama951]

done. let me know if there is anything else

[fanyang-mono]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

[fanyang-mono]

Suggested change

	}DebugEntry;
	}DebugEntry;

[fanyang-mono]

Suggested change

	}JitCodeDebug;
	}JitCodeDebug;

[fanyang-mono]

Suggested change

	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);
	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);

[fanyang-mono]

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

[fanyang-mono]

Fix code style

[fanyang-mono]

Suggested change

	if(!(loc)
	if(!loc)

[saitama951]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

[saitama951]

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

[directhex]

@vargaz@lambdageek can someone see if this is ready to merge now?

[shivanirmishra]

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[saitama951]

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

[shivanirmishra]

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

[saitama951]

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

[shivanirmishra]

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

[saitama951]

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

[shivanirmishra]

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

[fanyang-mono]

/backport to release/7.0-staging

[github-actions]

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

[shivanirmishra]

@saitama951@fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

[saitama951]

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

[shivanirmishra]

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

[fanyang-mono]

@saitama951@fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

[fanyang-mono]

@shivanirmishra We will get this fix in the next possible servicing release.

[shivanirmishra]

@saitama951@fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

[saitama951]

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

[shivanirmishra]

Hi @saitama951@fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

[saitama951]

@shivanirmishra have you applied the diff changes ?

[shivanirmishra]

@saitama951 Yes, It's working.
Thank you.

[saitama951]

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

[vargaz]

This should follow the mono coding conventions, see the rest of the file for examples.

[saitama951]

done. let me know if there is anything else

[saitama951]

changed the style to adhere to the mono coding conventions

[saitama951]

done. let me know if there is anything else

[fanyang-mono]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

[fanyang-mono]

Suggested change

	}DebugEntry;
	}DebugEntry;

[fanyang-mono]

Suggested change

	}JitCodeDebug;
	}JitCodeDebug;

[fanyang-mono]

Suggested change

	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);
	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);

[fanyang-mono]

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

[fanyang-mono]

Fix code style

[fanyang-mono]

Suggested change

	if(!(loc)
	if(!loc)

[saitama951]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

[saitama951]

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

[directhex]

@vargaz@lambdageek can someone see if this is ready to merge now?

[shivanirmishra]

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[saitama951]

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

[shivanirmishra]

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

[saitama951]

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

[shivanirmishra]

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

[saitama951]

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

[shivanirmishra]

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

[fanyang-mono]

/backport to release/7.0-staging

[github-actions]

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

[shivanirmishra]

@saitama951@fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

[saitama951]

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

[shivanirmishra]

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

[fanyang-mono]

@saitama951@fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

[fanyang-mono]

@shivanirmishra We will get this fix in the next possible servicing release.

[shivanirmishra]

@saitama951@fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

[saitama951]

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

[shivanirmishra]

Hi @saitama951@fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

[saitama951]

@shivanirmishra have you applied the diff changes ?

[shivanirmishra]

@saitama951 Yes, It's working.
Thank you.

[saitama951]

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

[vargaz]

This should follow the mono coding conventions, see the rest of the file for examples.

[saitama951]

done. let me know if there is anything else

[saitama951]

changed the style to adhere to the mono coding conventions

[saitama951]

done. let me know if there is anything else

[fanyang-mono]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

[fanyang-mono]

Suggested change

	}DebugEntry;
	}DebugEntry;

[fanyang-mono]

Suggested change

	}JitCodeDebug;
	}JitCodeDebug;

[fanyang-mono]

Suggested change

	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);
	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);

[fanyang-mono]

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

[fanyang-mono]

Fix code style

[fanyang-mono]

Suggested change

	if(!(loc)
	if(!loc)

[saitama951]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

[saitama951]

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

[directhex]

@vargaz@lambdageek can someone see if this is ready to merge now?

[shivanirmishra]

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[saitama951]

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

[shivanirmishra]

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

[saitama951]

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

[shivanirmishra]

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

[saitama951]

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

[shivanirmishra]

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

[fanyang-mono]

/backport to release/7.0-staging

[github-actions]

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

[shivanirmishra]

@saitama951@fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

[saitama951]

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

[shivanirmishra]

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

[fanyang-mono]

@saitama951@fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

[fanyang-mono]

@shivanirmishra We will get this fix in the next possible servicing release.

[shivanirmishra]

@saitama951@fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

[saitama951]

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

[shivanirmishra]

Hi @saitama951@fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

[saitama951]

@shivanirmishra have you applied the diff changes ?

[shivanirmishra]

@saitama951 Yes, It's working.
Thank you.

[saitama951]

This is related to #36774 . added support for JIT_CODE_DEBUG_INFO record which helps to annotate source code in perf.

Changed the jitdump version to 1 as the perf sources expect version 1

https://github.com/torvalds/linux/blob/master/tools/perf/util/jitdump.h#L25

Example use:-
MONO_ENV_OPTIONS="--jitdump" perf record -k 1 dotnet <path/to/binary>
perf inject --jit -i perf.data -o perf.jit.data
perf report -i perf.jit.data

cc: @fanyang-mono

[vargaz]

This should follow the mono coding conventions, see the rest of the file for examples.

[saitama951]

done. let me know if there is anything else

[saitama951]

changed the style to adhere to the mono coding conventions

[saitama951]

done. let me know if there is anything else

[fanyang-mono]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

[fanyang-mono]

Suggested change

	}DebugEntry;
	}DebugEntry;

[fanyang-mono]

Suggested change

	}JitCodeDebug;
	}JitCodeDebug;

[fanyang-mono]

Suggested change

	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);
	staticvoidadd_basic_JitCodeDebug_info(JitCodeDebug*record);

[fanyang-mono]

In mono code base, we add a space between the function name and the "(". Please follow this function call style for all the code change in this PR.

[fanyang-mono]

Fix code style

[fanyang-mono]

Suggested change

	if(!(loc)
	if(!loc)

[saitama951]

Please go through your change and fix the coding style. I've pointed out a few examples for you.

Thank you for helping me out with the code style. I have made the changes accordingly.

[saitama951]

@fanyang-mono is there any other changes required to be made?
maybe we can merge the code?

[directhex]

@vargaz@lambdageek can someone see if this is ready to merge now?

[shivanirmishra]

@fanyang-mono
The fix did not work for me.
The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Error:
objdump: /tmp/perf-886285.map: File format not recognized

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[saitama951]

@shivanirmishra The runtime should generate the .dump file. can you cross check that under the /tmp/?
if that is present can you check if the dumps are being read by using the strace?

I faced a similar issue and fixed the same : #82520

[shivanirmishra]

Hi @saitama951, yes runtime is generating the .dump file in the/tmp/ folder.
But Strace is not showing any references to dump files.
The observations we had are with .NET7.0 so is the above fix available only on .NET8 or later?

[saitama951]

@shivanirmishra the fix should be there in .NET 8 preview 4 or you can cherry pick the commit.

[shivanirmishra]

@saitama951
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?

[saitama951]

@shivanirmishra I have fixed the wrong versioning of jitdump in the current PR itself, probably this fix isn't incorporated in this release I guess , I would suggest you can build dotnet from source that should work fine.

[shivanirmishra]

Hi @saitama951, thank you for your help.
This fix is working for the sample dotnet code.
But while building dotnet for the workload from https://github.com/dotnet/performance, we are facing the below issues with the power system:

Any suggestion regarding the above error?
Also, I have changed the target frameworks to .net8 from .net7.

[fanyang-mono]

/backport to release/7.0-staging

[github-actions]

Started backporting to release/7.0-staging: https://github.com/dotnet/runtime/actions/runs/5248244227

[shivanirmishra]

@saitama951@fanyang-mono
We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

[saitama951]

@shivanirmishra It seems source code annotation fails on top of the BenchmarkDotnet library. it works for sample programs without the BDN. I have been manually looking into the jitted-so's, everything looks good there as well.

[shivanirmishra]

Thanks @saitama951
It works for other workloads that run for a shorter duration.
But if we are trying to run a longer workload, then sometimes it's showing "unknown" in the profile on both X86 and Power systems.

For Example: after running the BinaryTrees_5 workload from https://github.com/dotnet/performance, we get the below profiles.

Before Inject:

After Inject:

[fanyang-mono]

@saitama951@fanyang-mono We are facing the above issue on .net8 on power, so will this backporting to 7.0 fix the issue on .net8 as well?

Yes, I am working on backporting the fixes to regressions to 7.0 now.

[fanyang-mono]

@shivanirmishra We will get this fix in the next possible servicing release.

[shivanirmishra]

@saitama951@fanyang-mono
The above fix is working for some applications but not for all.
For example, It's not working if we increase the no of iterations for the same application

This is the profile with 100 iterations of some ml.net workload:

This is the profile with 10k iterations of the same workload:

The above profiles are after perf inject.

Any suggestions or ideas for why this is not working with larger overhead?

[saitama951]

@shivanirmishra I have proposed the fix for the issue that you were facing.

#88373

[shivanirmishra]

Hi @saitama951@fanyang-mono
This is another sample code for matrix multiplication, in which we are seeing the same above issue.

MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 /home/user/dotnet run

Dimension of the matrix: 2000x3 and 3x2000, and running the MultiplyMatrices function once.

perf Report -i perf.data

perf inject -j -i ./perf.data -o ./perf.jit.data

perf report -i perf.jit.data

We are not able to see the main function MultiplyMatrices anywhere in the profile.
And wherever we are seeing the project name (new-matrix) as a command in the profile, we don’t see any functions/annotations for some so it might be splitting the main function into some parts and we are not able to resolve the same.

[saitama951]

@shivanirmishra have you applied the diff changes ?

[shivanirmishra]

@saitama951 Yes, It's working.
Thank you.