Skip to content

[wasm] Endian fix for Webcil - #92274

Merged
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil
Sep 22, 2023
Merged

[wasm] Endian fix for Webcil#92274
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil

Conversation

@saitama951

@saitama951saitama951 commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes#91893
cc: @omajid@iii-i@uweigand

@ghostghost added area-Build-mono community-contribution Indicates that the PR has been added by a community member labels Sep 19, 2023
@saitama951
saitama951force-pushed the fix-endian-webcil branch 3 times, most recently from 2671ac0 to ab6a917CompareSeptember 19, 2023 16:00
Comment on lines 198 to 201

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.PointerToRawData)

Comment on lines 230 to 236

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
webcilHeader.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_major));
webcilHeader.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_minor));
webcilHeader.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.coff_sections));
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_rva));
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_size));
webcilHeader.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_rva));
webcilHeader.pe_debug_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_size));
webcilHeader.version_major=BinaryPrimitives.ReverseEndianess(webcilHeader.version_major);
webcilHeader.version_minor=BinaryPrimitives.ReverseEndianess(webcilHeader.version_minor);
webcilHeader.coff_sections=BinaryPrimitives.ReverseEndianess(webcilHeader.coff_sections);
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_rva);
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_size);
webcilHeader.pe_debug_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_rva);
webcilHeader.pe_debug_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_size);

Comment on lines 72 to 78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
header.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_major));
header.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_minor));
header.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.coff_sections));
header.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_rva));
header.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_size));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_rva));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_size));
header.version_major=BinaryPrimitives.ReverseEndianess(header.version_major);
header.version_minor=BinaryPrimitives.ReverseEndianess(header.version_minor);
header.coff_sections=BinaryPrimitives.ReverseEndianess(header.coff_sections);
header.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_rva);
header.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_size);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_rva);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_size);

Comment on lines 375 to 378

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(secheader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(secheader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(secheader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(secheader.PointerToRawData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Can you try removing the #if if it compiles? This function might be available on all frameworks (unless I'm missing something).

@saitama951saitama951Sep 19, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I have done the same for the WriteStructure as well and hence removing the endian checks for net472 as well. making it generic :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little more subtle than just having the API on net472. Because this code is used in an MSBuild task that might be called from a build in Visual Studio, the API has to be in whatever old version of the out of band assembly is already loaded in to the MSBuild process that invokes the task.

@saitama951 Hmm... actually we might also need to add this:

 <PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

to src/tasks/Microsoft.NET.WebAssembly.Webcil/Microsoft.NET.WebAssembly.Webcil.csproj

@ghost

Copy link
Copy Markdown

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes #91893
cc: @omajid@iii-i@uweigand

Author:saitama951
Assignees:-
Labels:

arch-wasm, area-Build-mono, community-contribution

Milestone:-

@marafmaraf added this to the 9.0.0 milestone Sep 20, 2023

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@saitama951

Copy link
Copy Markdown
ContributorAuthor

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@lambdageek I have done the suggested changes, Thank you for the input.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

Comment threadsrc/tasks/Microsoft.NET.WebAssembly.Webcil/WebcilConverter.cs Outdated
'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception
The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.
We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.
@saitama951

Copy link
Copy Markdown
ContributorAuthor

@lambdageek@radical@teo-tsirpanis can this be merged?

@radical

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

Copy link
Copy Markdown
Member

WasmBuildTests has relevant failures, like:

 [] Stack overflow.
[] Repeat 74781 times:
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteStructure(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteHeader(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteConversionTo(System.IO.Stream, System.IO.FileStream, PEFileInfo, WCFileInfo)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilder.ExecuteInternal()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilderBaseTask.Execute()
[] at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteInstantiatedTask>d__25 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<InitializeAndExecuteTask>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder.InitializeAndExecuteTask(Microsoft.Build.BackEnd.Logging.TaskLoggingContext, Microsoft.Build.BackEnd.ItemBucket, System.Collections.Generic.IDictionary`2<System.String,System.String>, Microsoft.Build.BackEnd.TaskHost, Microsoft.Build.BackEnd.TaskExecutionMode)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteBucket>d__19 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__18 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__13 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessBucket>d__51 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTarget>d__44 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget(Microsoft.Build.BackEnd.ITaskBuilder, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.ProcessTargetStack(Microsoft.Build.BackEnd.ITaskBuilder)
[] at Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.BuildTargets(Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.IRequestBuilderCallback, System.String[], Microsoft.Build.BackEnd.Lookup, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
[] at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
[] at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[] at System.Threading.Tasks.Task.ExecuteEntry()
[] at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@lambdageek

Copy link
Copy Markdown
Member

Moved the endianess fixup for WebcilHeader to WriteHeader instead of an overload of WriteStructure /cc @saitama951

@lambdageek

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@uweigand

Copy link
Copy Markdown
Contributor

This is failing in .NET 8 - can this be merged there?

@lambdageek

Copy link
Copy Markdown
Member

/backport to release/8.0

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6278015202

@ghostghost locked as resolved and limited conversation to collaborators Oct 22, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasmWebAssembly architecturearea-Build-monocommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The "ConvertDllsToWebCil" task failed: System.NotImplementedException on s390x

6 participants

@saitama951@radical@lambdageek@uweigand@teo-tsirpanis@maraf
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
[wasm] Endian fix for Webcil by saitama951 · Pull Request #92274 · dotnet/runtime · GitHub
Skip to content

[wasm] Endian fix for Webcil - #92274

Merged
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil
Sep 22, 2023
Merged

[wasm] Endian fix for Webcil#92274
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil

Conversation

@saitama951

@saitama951saitama951 commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes#91893
cc: @omajid@iii-i@uweigand

@ghostghost added area-Build-mono community-contribution Indicates that the PR has been added by a community member labels Sep 19, 2023
@saitama951
saitama951force-pushed the fix-endian-webcil branch 3 times, most recently from 2671ac0 to ab6a917CompareSeptember 19, 2023 16:00
Comment on lines 198 to 201

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.PointerToRawData)

Comment on lines 230 to 236

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
webcilHeader.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_major));
webcilHeader.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_minor));
webcilHeader.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.coff_sections));
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_rva));
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_size));
webcilHeader.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_rva));
webcilHeader.pe_debug_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_size));
webcilHeader.version_major=BinaryPrimitives.ReverseEndianess(webcilHeader.version_major);
webcilHeader.version_minor=BinaryPrimitives.ReverseEndianess(webcilHeader.version_minor);
webcilHeader.coff_sections=BinaryPrimitives.ReverseEndianess(webcilHeader.coff_sections);
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_rva);
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_size);
webcilHeader.pe_debug_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_rva);
webcilHeader.pe_debug_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_size);

Comment on lines 72 to 78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
header.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_major));
header.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_minor));
header.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.coff_sections));
header.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_rva));
header.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_size));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_rva));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_size));
header.version_major=BinaryPrimitives.ReverseEndianess(header.version_major);
header.version_minor=BinaryPrimitives.ReverseEndianess(header.version_minor);
header.coff_sections=BinaryPrimitives.ReverseEndianess(header.coff_sections);
header.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_rva);
header.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_size);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_rva);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_size);

Comment on lines 375 to 378

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(secheader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(secheader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(secheader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(secheader.PointerToRawData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Can you try removing the #if if it compiles? This function might be available on all frameworks (unless I'm missing something).

@saitama951saitama951Sep 19, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I have done the same for the WriteStructure as well and hence removing the endian checks for net472 as well. making it generic :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little more subtle than just having the API on net472. Because this code is used in an MSBuild task that might be called from a build in Visual Studio, the API has to be in whatever old version of the out of band assembly is already loaded in to the MSBuild process that invokes the task.

@saitama951 Hmm... actually we might also need to add this:

 <PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

to src/tasks/Microsoft.NET.WebAssembly.Webcil/Microsoft.NET.WebAssembly.Webcil.csproj

@ghost

Copy link
Copy Markdown

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes #91893
cc: @omajid@iii-i@uweigand

Author:saitama951
Assignees:-
Labels:

arch-wasm, area-Build-mono, community-contribution

Milestone:-

@marafmaraf added this to the 9.0.0 milestone Sep 20, 2023

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@saitama951

Copy link
Copy Markdown
ContributorAuthor

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@lambdageek I have done the suggested changes, Thank you for the input.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

Comment threadsrc/tasks/Microsoft.NET.WebAssembly.Webcil/WebcilConverter.cs Outdated
'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception
The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.
We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.
@saitama951

Copy link
Copy Markdown
ContributorAuthor

@lambdageek@radical@teo-tsirpanis can this be merged?

@radical

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

Copy link
Copy Markdown
Member

WasmBuildTests has relevant failures, like:

 [] Stack overflow.
[] Repeat 74781 times:
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteStructure(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteHeader(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteConversionTo(System.IO.Stream, System.IO.FileStream, PEFileInfo, WCFileInfo)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilder.ExecuteInternal()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilderBaseTask.Execute()
[] at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteInstantiatedTask>d__25 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<InitializeAndExecuteTask>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder.InitializeAndExecuteTask(Microsoft.Build.BackEnd.Logging.TaskLoggingContext, Microsoft.Build.BackEnd.ItemBucket, System.Collections.Generic.IDictionary`2<System.String,System.String>, Microsoft.Build.BackEnd.TaskHost, Microsoft.Build.BackEnd.TaskExecutionMode)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteBucket>d__19 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__18 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__13 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessBucket>d__51 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTarget>d__44 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget(Microsoft.Build.BackEnd.ITaskBuilder, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.ProcessTargetStack(Microsoft.Build.BackEnd.ITaskBuilder)
[] at Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.BuildTargets(Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.IRequestBuilderCallback, System.String[], Microsoft.Build.BackEnd.Lookup, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
[] at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
[] at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[] at System.Threading.Tasks.Task.ExecuteEntry()
[] at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@lambdageek

Copy link
Copy Markdown
Member

Moved the endianess fixup for WebcilHeader to WriteHeader instead of an overload of WriteStructure /cc @saitama951

@lambdageek

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@uweigand

Copy link
Copy Markdown
Contributor

This is failing in .NET 8 - can this be merged there?

@lambdageek

Copy link
Copy Markdown
Member

/backport to release/8.0

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6278015202

@ghostghost locked as resolved and limited conversation to collaborators Oct 22, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasmWebAssembly architecturearea-Build-monocommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The "ConvertDllsToWebCil" task failed: System.NotImplementedException on s390x

6 participants

@saitama951@radical@lambdageek@uweigand@teo-tsirpanis@maraf
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' [wasm] Endian fix for Webcil by saitama951 · Pull Request #92274 · dotnet/runtime · GitHub
Skip to content

[wasm] Endian fix for Webcil - #92274

Merged
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil
Sep 22, 2023
Merged

[wasm] Endian fix for Webcil#92274
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil

Conversation

@saitama951

@saitama951saitama951 commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes#91893
cc: @omajid@iii-i@uweigand

@ghostghost added area-Build-mono community-contribution Indicates that the PR has been added by a community member labels Sep 19, 2023
@saitama951
saitama951force-pushed the fix-endian-webcil branch 3 times, most recently from 2671ac0 to ab6a917CompareSeptember 19, 2023 16:00
Comment on lines 198 to 201

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.PointerToRawData)

Comment on lines 230 to 236

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
webcilHeader.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_major));
webcilHeader.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_minor));
webcilHeader.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.coff_sections));
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_rva));
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_size));
webcilHeader.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_rva));
webcilHeader.pe_debug_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_size));
webcilHeader.version_major=BinaryPrimitives.ReverseEndianess(webcilHeader.version_major);
webcilHeader.version_minor=BinaryPrimitives.ReverseEndianess(webcilHeader.version_minor);
webcilHeader.coff_sections=BinaryPrimitives.ReverseEndianess(webcilHeader.coff_sections);
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_rva);
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_size);
webcilHeader.pe_debug_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_rva);
webcilHeader.pe_debug_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_size);

Comment on lines 72 to 78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
header.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_major));
header.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_minor));
header.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.coff_sections));
header.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_rva));
header.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_size));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_rva));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_size));
header.version_major=BinaryPrimitives.ReverseEndianess(header.version_major);
header.version_minor=BinaryPrimitives.ReverseEndianess(header.version_minor);
header.coff_sections=BinaryPrimitives.ReverseEndianess(header.coff_sections);
header.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_rva);
header.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_size);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_rva);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_size);

Comment on lines 375 to 378

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(secheader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(secheader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(secheader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(secheader.PointerToRawData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Can you try removing the #if if it compiles? This function might be available on all frameworks (unless I'm missing something).

@saitama951saitama951Sep 19, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I have done the same for the WriteStructure as well and hence removing the endian checks for net472 as well. making it generic :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little more subtle than just having the API on net472. Because this code is used in an MSBuild task that might be called from a build in Visual Studio, the API has to be in whatever old version of the out of band assembly is already loaded in to the MSBuild process that invokes the task.

@saitama951 Hmm... actually we might also need to add this:

 <PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

to src/tasks/Microsoft.NET.WebAssembly.Webcil/Microsoft.NET.WebAssembly.Webcil.csproj

@ghost

Copy link
Copy Markdown

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes #91893
cc: @omajid@iii-i@uweigand

Author:saitama951
Assignees:-
Labels:

arch-wasm, area-Build-mono, community-contribution

Milestone:-

@marafmaraf added this to the 9.0.0 milestone Sep 20, 2023

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@saitama951

Copy link
Copy Markdown
ContributorAuthor

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@lambdageek I have done the suggested changes, Thank you for the input.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

Comment threadsrc/tasks/Microsoft.NET.WebAssembly.Webcil/WebcilConverter.cs Outdated
'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception
The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.
We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.
@saitama951

Copy link
Copy Markdown
ContributorAuthor

@lambdageek@radical@teo-tsirpanis can this be merged?

@radical

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

Copy link
Copy Markdown
Member

WasmBuildTests has relevant failures, like:

 [] Stack overflow.
[] Repeat 74781 times:
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteStructure(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteHeader(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteConversionTo(System.IO.Stream, System.IO.FileStream, PEFileInfo, WCFileInfo)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilder.ExecuteInternal()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilderBaseTask.Execute()
[] at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteInstantiatedTask>d__25 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<InitializeAndExecuteTask>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder.InitializeAndExecuteTask(Microsoft.Build.BackEnd.Logging.TaskLoggingContext, Microsoft.Build.BackEnd.ItemBucket, System.Collections.Generic.IDictionary`2<System.String,System.String>, Microsoft.Build.BackEnd.TaskHost, Microsoft.Build.BackEnd.TaskExecutionMode)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteBucket>d__19 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__18 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__13 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessBucket>d__51 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTarget>d__44 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget(Microsoft.Build.BackEnd.ITaskBuilder, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.ProcessTargetStack(Microsoft.Build.BackEnd.ITaskBuilder)
[] at Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.BuildTargets(Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.IRequestBuilderCallback, System.String[], Microsoft.Build.BackEnd.Lookup, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
[] at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
[] at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[] at System.Threading.Tasks.Task.ExecuteEntry()
[] at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@lambdageek

Copy link
Copy Markdown
Member

Moved the endianess fixup for WebcilHeader to WriteHeader instead of an overload of WriteStructure /cc @saitama951

@lambdageek

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@uweigand

Copy link
Copy Markdown
Contributor

This is failing in .NET 8 - can this be merged there?

@lambdageek

Copy link
Copy Markdown
Member

/backport to release/8.0

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6278015202

@ghostghost locked as resolved and limited conversation to collaborators Oct 22, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasmWebAssembly architecturearea-Build-monocommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The "ConvertDllsToWebCil" task failed: System.NotImplementedException on s390x

6 participants

@saitama951@radical@lambdageek@uweigand@teo-tsirpanis@maraf
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' [wasm] Endian fix for Webcil by saitama951 · Pull Request #92274 · dotnet/runtime · GitHub
Skip to content

[wasm] Endian fix for Webcil - #92274

Merged
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil
Sep 22, 2023
Merged

[wasm] Endian fix for Webcil#92274
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil

Conversation

@saitama951

@saitama951saitama951 commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes#91893
cc: @omajid@iii-i@uweigand

@ghostghost added area-Build-mono community-contribution Indicates that the PR has been added by a community member labels Sep 19, 2023
@saitama951
saitama951force-pushed the fix-endian-webcil branch 3 times, most recently from 2671ac0 to ab6a917CompareSeptember 19, 2023 16:00
Comment on lines 198 to 201

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.PointerToRawData)

Comment on lines 230 to 236

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
webcilHeader.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_major));
webcilHeader.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_minor));
webcilHeader.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.coff_sections));
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_rva));
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_size));
webcilHeader.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_rva));
webcilHeader.pe_debug_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_size));
webcilHeader.version_major=BinaryPrimitives.ReverseEndianess(webcilHeader.version_major);
webcilHeader.version_minor=BinaryPrimitives.ReverseEndianess(webcilHeader.version_minor);
webcilHeader.coff_sections=BinaryPrimitives.ReverseEndianess(webcilHeader.coff_sections);
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_rva);
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_size);
webcilHeader.pe_debug_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_rva);
webcilHeader.pe_debug_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_size);

Comment on lines 72 to 78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
header.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_major));
header.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_minor));
header.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.coff_sections));
header.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_rva));
header.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_size));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_rva));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_size));
header.version_major=BinaryPrimitives.ReverseEndianess(header.version_major);
header.version_minor=BinaryPrimitives.ReverseEndianess(header.version_minor);
header.coff_sections=BinaryPrimitives.ReverseEndianess(header.coff_sections);
header.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_rva);
header.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_size);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_rva);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_size);

Comment on lines 375 to 378

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(secheader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(secheader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(secheader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(secheader.PointerToRawData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Can you try removing the #if if it compiles? This function might be available on all frameworks (unless I'm missing something).

@saitama951saitama951Sep 19, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I have done the same for the WriteStructure as well and hence removing the endian checks for net472 as well. making it generic :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little more subtle than just having the API on net472. Because this code is used in an MSBuild task that might be called from a build in Visual Studio, the API has to be in whatever old version of the out of band assembly is already loaded in to the MSBuild process that invokes the task.

@saitama951 Hmm... actually we might also need to add this:

 <PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

to src/tasks/Microsoft.NET.WebAssembly.Webcil/Microsoft.NET.WebAssembly.Webcil.csproj

@ghost

Copy link
Copy Markdown

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes #91893
cc: @omajid@iii-i@uweigand

Author:saitama951
Assignees:-
Labels:

arch-wasm, area-Build-mono, community-contribution

Milestone:-

@marafmaraf added this to the 9.0.0 milestone Sep 20, 2023

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@saitama951

Copy link
Copy Markdown
ContributorAuthor

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@lambdageek I have done the suggested changes, Thank you for the input.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

Comment threadsrc/tasks/Microsoft.NET.WebAssembly.Webcil/WebcilConverter.cs Outdated
'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception
The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.
We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.
@saitama951

Copy link
Copy Markdown
ContributorAuthor

@lambdageek@radical@teo-tsirpanis can this be merged?

@radical

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

Copy link
Copy Markdown
Member

WasmBuildTests has relevant failures, like:

 [] Stack overflow.
[] Repeat 74781 times:
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteStructure(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteHeader(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteConversionTo(System.IO.Stream, System.IO.FileStream, PEFileInfo, WCFileInfo)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilder.ExecuteInternal()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilderBaseTask.Execute()
[] at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteInstantiatedTask>d__25 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<InitializeAndExecuteTask>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder.InitializeAndExecuteTask(Microsoft.Build.BackEnd.Logging.TaskLoggingContext, Microsoft.Build.BackEnd.ItemBucket, System.Collections.Generic.IDictionary`2<System.String,System.String>, Microsoft.Build.BackEnd.TaskHost, Microsoft.Build.BackEnd.TaskExecutionMode)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteBucket>d__19 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__18 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__13 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessBucket>d__51 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTarget>d__44 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget(Microsoft.Build.BackEnd.ITaskBuilder, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.ProcessTargetStack(Microsoft.Build.BackEnd.ITaskBuilder)
[] at Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.BuildTargets(Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.IRequestBuilderCallback, System.String[], Microsoft.Build.BackEnd.Lookup, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
[] at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
[] at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[] at System.Threading.Tasks.Task.ExecuteEntry()
[] at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@lambdageek

Copy link
Copy Markdown
Member

Moved the endianess fixup for WebcilHeader to WriteHeader instead of an overload of WriteStructure /cc @saitama951

@lambdageek

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@uweigand

Copy link
Copy Markdown
Contributor

This is failing in .NET 8 - can this be merged there?

@lambdageek

Copy link
Copy Markdown
Member

/backport to release/8.0

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6278015202

@ghostghost locked as resolved and limited conversation to collaborators Oct 22, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasmWebAssembly architecturearea-Build-monocommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The "ConvertDllsToWebCil" task failed: System.NotImplementedException on s390x

6 participants

@saitama951@radical@lambdageek@uweigand@teo-tsirpanis@maraf
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' [wasm] Endian fix for Webcil by saitama951 · Pull Request #92274 · dotnet/runtime · GitHub
Skip to content

[wasm] Endian fix for Webcil - #92274

Merged
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil
Sep 22, 2023
Merged

[wasm] Endian fix for Webcil#92274
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil

Conversation

@saitama951

@saitama951saitama951 commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes#91893
cc: @omajid@iii-i@uweigand

@ghostghost added area-Build-mono community-contribution Indicates that the PR has been added by a community member labels Sep 19, 2023
@saitama951
saitama951force-pushed the fix-endian-webcil branch 3 times, most recently from 2671ac0 to ab6a917CompareSeptember 19, 2023 16:00
Comment on lines 198 to 201

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.PointerToRawData)

Comment on lines 230 to 236

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
webcilHeader.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_major));
webcilHeader.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_minor));
webcilHeader.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.coff_sections));
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_rva));
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_size));
webcilHeader.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_rva));
webcilHeader.pe_debug_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_size));
webcilHeader.version_major=BinaryPrimitives.ReverseEndianess(webcilHeader.version_major);
webcilHeader.version_minor=BinaryPrimitives.ReverseEndianess(webcilHeader.version_minor);
webcilHeader.coff_sections=BinaryPrimitives.ReverseEndianess(webcilHeader.coff_sections);
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_rva);
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_size);
webcilHeader.pe_debug_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_rva);
webcilHeader.pe_debug_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_size);

Comment on lines 72 to 78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
header.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_major));
header.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_minor));
header.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.coff_sections));
header.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_rva));
header.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_size));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_rva));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_size));
header.version_major=BinaryPrimitives.ReverseEndianess(header.version_major);
header.version_minor=BinaryPrimitives.ReverseEndianess(header.version_minor);
header.coff_sections=BinaryPrimitives.ReverseEndianess(header.coff_sections);
header.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_rva);
header.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_size);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_rva);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_size);

Comment on lines 375 to 378

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(secheader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(secheader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(secheader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(secheader.PointerToRawData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Can you try removing the #if if it compiles? This function might be available on all frameworks (unless I'm missing something).

@saitama951saitama951Sep 19, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I have done the same for the WriteStructure as well and hence removing the endian checks for net472 as well. making it generic :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little more subtle than just having the API on net472. Because this code is used in an MSBuild task that might be called from a build in Visual Studio, the API has to be in whatever old version of the out of band assembly is already loaded in to the MSBuild process that invokes the task.

@saitama951 Hmm... actually we might also need to add this:

 <PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

to src/tasks/Microsoft.NET.WebAssembly.Webcil/Microsoft.NET.WebAssembly.Webcil.csproj

@ghost

Copy link
Copy Markdown

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes #91893
cc: @omajid@iii-i@uweigand

Author:saitama951
Assignees:-
Labels:

arch-wasm, area-Build-mono, community-contribution

Milestone:-

@marafmaraf added this to the 9.0.0 milestone Sep 20, 2023

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@saitama951

Copy link
Copy Markdown
ContributorAuthor

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@lambdageek I have done the suggested changes, Thank you for the input.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

Comment threadsrc/tasks/Microsoft.NET.WebAssembly.Webcil/WebcilConverter.cs Outdated
'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception
The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.
We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.
@saitama951

Copy link
Copy Markdown
ContributorAuthor

@lambdageek@radical@teo-tsirpanis can this be merged?

@radical

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

Copy link
Copy Markdown
Member

WasmBuildTests has relevant failures, like:

 [] Stack overflow.
[] Repeat 74781 times:
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteStructure(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteHeader(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteConversionTo(System.IO.Stream, System.IO.FileStream, PEFileInfo, WCFileInfo)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilder.ExecuteInternal()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilderBaseTask.Execute()
[] at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteInstantiatedTask>d__25 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<InitializeAndExecuteTask>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder.InitializeAndExecuteTask(Microsoft.Build.BackEnd.Logging.TaskLoggingContext, Microsoft.Build.BackEnd.ItemBucket, System.Collections.Generic.IDictionary`2<System.String,System.String>, Microsoft.Build.BackEnd.TaskHost, Microsoft.Build.BackEnd.TaskExecutionMode)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteBucket>d__19 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__18 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__13 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessBucket>d__51 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTarget>d__44 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget(Microsoft.Build.BackEnd.ITaskBuilder, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.ProcessTargetStack(Microsoft.Build.BackEnd.ITaskBuilder)
[] at Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.BuildTargets(Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.IRequestBuilderCallback, System.String[], Microsoft.Build.BackEnd.Lookup, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
[] at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
[] at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[] at System.Threading.Tasks.Task.ExecuteEntry()
[] at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@lambdageek

Copy link
Copy Markdown
Member

Moved the endianess fixup for WebcilHeader to WriteHeader instead of an overload of WriteStructure /cc @saitama951

@lambdageek

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@uweigand

Copy link
Copy Markdown
Contributor

This is failing in .NET 8 - can this be merged there?

@lambdageek

Copy link
Copy Markdown
Member

/backport to release/8.0

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6278015202

@ghostghost locked as resolved and limited conversation to collaborators Oct 22, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasmWebAssembly architecturearea-Build-monocommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The "ConvertDllsToWebCil" task failed: System.NotImplementedException on s390x

6 participants

@saitama951@radical@lambdageek@uweigand@teo-tsirpanis@maraf
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' [wasm] Endian fix for Webcil by saitama951 · Pull Request #92274 · dotnet/runtime · GitHub
Skip to content

[wasm] Endian fix for Webcil - #92274

Merged
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil
Sep 22, 2023
Merged

[wasm] Endian fix for Webcil#92274
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil

Conversation

@saitama951

@saitama951saitama951 commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes#91893
cc: @omajid@iii-i@uweigand

@ghostghost added area-Build-mono community-contribution Indicates that the PR has been added by a community member labels Sep 19, 2023
@saitama951
saitama951force-pushed the fix-endian-webcil branch 3 times, most recently from 2671ac0 to ab6a917CompareSeptember 19, 2023 16:00
Comment on lines 198 to 201

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.PointerToRawData)

Comment on lines 230 to 236

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
webcilHeader.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_major));
webcilHeader.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_minor));
webcilHeader.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.coff_sections));
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_rva));
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_size));
webcilHeader.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_rva));
webcilHeader.pe_debug_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_size));
webcilHeader.version_major=BinaryPrimitives.ReverseEndianess(webcilHeader.version_major);
webcilHeader.version_minor=BinaryPrimitives.ReverseEndianess(webcilHeader.version_minor);
webcilHeader.coff_sections=BinaryPrimitives.ReverseEndianess(webcilHeader.coff_sections);
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_rva);
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_size);
webcilHeader.pe_debug_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_rva);
webcilHeader.pe_debug_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_size);

Comment on lines 72 to 78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
header.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_major));
header.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_minor));
header.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.coff_sections));
header.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_rva));
header.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_size));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_rva));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_size));
header.version_major=BinaryPrimitives.ReverseEndianess(header.version_major);
header.version_minor=BinaryPrimitives.ReverseEndianess(header.version_minor);
header.coff_sections=BinaryPrimitives.ReverseEndianess(header.coff_sections);
header.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_rva);
header.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_size);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_rva);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_size);

Comment on lines 375 to 378

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(secheader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(secheader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(secheader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(secheader.PointerToRawData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Can you try removing the #if if it compiles? This function might be available on all frameworks (unless I'm missing something).

@saitama951saitama951Sep 19, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I have done the same for the WriteStructure as well and hence removing the endian checks for net472 as well. making it generic :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little more subtle than just having the API on net472. Because this code is used in an MSBuild task that might be called from a build in Visual Studio, the API has to be in whatever old version of the out of band assembly is already loaded in to the MSBuild process that invokes the task.

@saitama951 Hmm... actually we might also need to add this:

 <PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

to src/tasks/Microsoft.NET.WebAssembly.Webcil/Microsoft.NET.WebAssembly.Webcil.csproj

@ghost

Copy link
Copy Markdown

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes #91893
cc: @omajid@iii-i@uweigand

Author:saitama951
Assignees:-
Labels:

arch-wasm, area-Build-mono, community-contribution

Milestone:-

@marafmaraf added this to the 9.0.0 milestone Sep 20, 2023

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@saitama951

Copy link
Copy Markdown
ContributorAuthor

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@lambdageek I have done the suggested changes, Thank you for the input.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

Comment threadsrc/tasks/Microsoft.NET.WebAssembly.Webcil/WebcilConverter.cs Outdated
'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception
The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.
We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.
@saitama951

Copy link
Copy Markdown
ContributorAuthor

@lambdageek@radical@teo-tsirpanis can this be merged?

@radical

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

Copy link
Copy Markdown
Member

WasmBuildTests has relevant failures, like:

 [] Stack overflow.
[] Repeat 74781 times:
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteStructure(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteHeader(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteConversionTo(System.IO.Stream, System.IO.FileStream, PEFileInfo, WCFileInfo)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilder.ExecuteInternal()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilderBaseTask.Execute()
[] at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteInstantiatedTask>d__25 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<InitializeAndExecuteTask>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder.InitializeAndExecuteTask(Microsoft.Build.BackEnd.Logging.TaskLoggingContext, Microsoft.Build.BackEnd.ItemBucket, System.Collections.Generic.IDictionary`2<System.String,System.String>, Microsoft.Build.BackEnd.TaskHost, Microsoft.Build.BackEnd.TaskExecutionMode)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteBucket>d__19 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__18 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__13 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessBucket>d__51 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTarget>d__44 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget(Microsoft.Build.BackEnd.ITaskBuilder, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.ProcessTargetStack(Microsoft.Build.BackEnd.ITaskBuilder)
[] at Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.BuildTargets(Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.IRequestBuilderCallback, System.String[], Microsoft.Build.BackEnd.Lookup, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
[] at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
[] at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[] at System.Threading.Tasks.Task.ExecuteEntry()
[] at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@lambdageek

Copy link
Copy Markdown
Member

Moved the endianess fixup for WebcilHeader to WriteHeader instead of an overload of WriteStructure /cc @saitama951

@lambdageek

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@uweigand

Copy link
Copy Markdown
Contributor

This is failing in .NET 8 - can this be merged there?

@lambdageek

Copy link
Copy Markdown
Member

/backport to release/8.0

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6278015202

@ghostghost locked as resolved and limited conversation to collaborators Oct 22, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasmWebAssembly architecturearea-Build-monocommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The "ConvertDllsToWebCil" task failed: System.NotImplementedException on s390x

6 participants

@saitama951@radical@lambdageek@uweigand@teo-tsirpanis@maraf
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' [wasm] Endian fix for Webcil by saitama951 · Pull Request #92274 · dotnet/runtime · GitHub
Skip to content

[wasm] Endian fix for Webcil - #92274

Merged
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil
Sep 22, 2023
Merged

[wasm] Endian fix for Webcil#92274
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil

Conversation

@saitama951

@saitama951saitama951 commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes#91893
cc: @omajid@iii-i@uweigand

@ghostghost added area-Build-mono community-contribution Indicates that the PR has been added by a community member labels Sep 19, 2023
@saitama951
saitama951force-pushed the fix-endian-webcil branch 3 times, most recently from 2671ac0 to ab6a917CompareSeptember 19, 2023 16:00
Comment on lines 198 to 201

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.PointerToRawData)

Comment on lines 230 to 236

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
webcilHeader.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_major));
webcilHeader.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_minor));
webcilHeader.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.coff_sections));
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_rva));
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_size));
webcilHeader.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_rva));
webcilHeader.pe_debug_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_size));
webcilHeader.version_major=BinaryPrimitives.ReverseEndianess(webcilHeader.version_major);
webcilHeader.version_minor=BinaryPrimitives.ReverseEndianess(webcilHeader.version_minor);
webcilHeader.coff_sections=BinaryPrimitives.ReverseEndianess(webcilHeader.coff_sections);
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_rva);
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_size);
webcilHeader.pe_debug_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_rva);
webcilHeader.pe_debug_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_size);

Comment on lines 72 to 78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
header.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_major));
header.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_minor));
header.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.coff_sections));
header.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_rva));
header.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_size));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_rva));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_size));
header.version_major=BinaryPrimitives.ReverseEndianess(header.version_major);
header.version_minor=BinaryPrimitives.ReverseEndianess(header.version_minor);
header.coff_sections=BinaryPrimitives.ReverseEndianess(header.coff_sections);
header.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_rva);
header.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_size);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_rva);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_size);

Comment on lines 375 to 378

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(secheader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(secheader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(secheader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(secheader.PointerToRawData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Can you try removing the #if if it compiles? This function might be available on all frameworks (unless I'm missing something).

@saitama951saitama951Sep 19, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I have done the same for the WriteStructure as well and hence removing the endian checks for net472 as well. making it generic :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little more subtle than just having the API on net472. Because this code is used in an MSBuild task that might be called from a build in Visual Studio, the API has to be in whatever old version of the out of band assembly is already loaded in to the MSBuild process that invokes the task.

@saitama951 Hmm... actually we might also need to add this:

 <PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

to src/tasks/Microsoft.NET.WebAssembly.Webcil/Microsoft.NET.WebAssembly.Webcil.csproj

@ghost

Copy link
Copy Markdown

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes #91893
cc: @omajid@iii-i@uweigand

Author:saitama951
Assignees:-
Labels:

arch-wasm, area-Build-mono, community-contribution

Milestone:-

@marafmaraf added this to the 9.0.0 milestone Sep 20, 2023

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@saitama951

Copy link
Copy Markdown
ContributorAuthor

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@lambdageek I have done the suggested changes, Thank you for the input.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

Comment threadsrc/tasks/Microsoft.NET.WebAssembly.Webcil/WebcilConverter.cs Outdated
'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception
The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.
We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.
@saitama951

Copy link
Copy Markdown
ContributorAuthor

@lambdageek@radical@teo-tsirpanis can this be merged?

@radical

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

Copy link
Copy Markdown
Member

WasmBuildTests has relevant failures, like:

 [] Stack overflow.
[] Repeat 74781 times:
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteStructure(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteHeader(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteConversionTo(System.IO.Stream, System.IO.FileStream, PEFileInfo, WCFileInfo)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilder.ExecuteInternal()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilderBaseTask.Execute()
[] at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteInstantiatedTask>d__25 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<InitializeAndExecuteTask>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder.InitializeAndExecuteTask(Microsoft.Build.BackEnd.Logging.TaskLoggingContext, Microsoft.Build.BackEnd.ItemBucket, System.Collections.Generic.IDictionary`2<System.String,System.String>, Microsoft.Build.BackEnd.TaskHost, Microsoft.Build.BackEnd.TaskExecutionMode)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteBucket>d__19 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__18 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__13 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessBucket>d__51 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTarget>d__44 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget(Microsoft.Build.BackEnd.ITaskBuilder, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.ProcessTargetStack(Microsoft.Build.BackEnd.ITaskBuilder)
[] at Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.BuildTargets(Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.IRequestBuilderCallback, System.String[], Microsoft.Build.BackEnd.Lookup, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
[] at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
[] at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[] at System.Threading.Tasks.Task.ExecuteEntry()
[] at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@lambdageek

Copy link
Copy Markdown
Member

Moved the endianess fixup for WebcilHeader to WriteHeader instead of an overload of WriteStructure /cc @saitama951

@lambdageek

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@uweigand

Copy link
Copy Markdown
Contributor

This is failing in .NET 8 - can this be merged there?

@lambdageek

Copy link
Copy Markdown
Member

/backport to release/8.0

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6278015202

@ghostghost locked as resolved and limited conversation to collaborators Oct 22, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasmWebAssembly architecturearea-Build-monocommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The "ConvertDllsToWebCil" task failed: System.NotImplementedException on s390x

6 participants

@saitama951@radical@lambdageek@uweigand@teo-tsirpanis@maraf
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); [wasm] Endian fix for Webcil by saitama951 · Pull Request #92274 · dotnet/runtime · GitHub
Skip to content

[wasm] Endian fix for Webcil - #92274

Merged
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil
Sep 22, 2023
Merged

[wasm] Endian fix for Webcil#92274
lambdageek merged 3 commits into
dotnet:mainfrom
saitama951:fix-endian-webcil

Conversation

@saitama951

@saitama951saitama951 commented Sep 19, 2023

Copy link
Copy Markdown
Contributor

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes#91893
cc: @omajid@iii-i@uweigand

@ghostghost added area-Build-mono community-contribution Indicates that the PR has been added by a community member labels Sep 19, 2023
@saitama951
saitama951force-pushed the fix-endian-webcil branch 3 times, most recently from 2671ac0 to ab6a917CompareSeptember 19, 2023 16:00
Comment on lines 198 to 201

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(sectionHeader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(sectionHeader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(sectionHeader.PointerToRawData)

Comment on lines 230 to 236

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
webcilHeader.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_major));
webcilHeader.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.version_minor));
webcilHeader.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(webcilHeader.coff_sections));
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_rva));
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_cli_header_size));
webcilHeader.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_rva));
webcilHeader.pe_debug_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(webcilHeader.pe_debug_size));
webcilHeader.version_major=BinaryPrimitives.ReverseEndianess(webcilHeader.version_major);
webcilHeader.version_minor=BinaryPrimitives.ReverseEndianess(webcilHeader.version_minor);
webcilHeader.coff_sections=BinaryPrimitives.ReverseEndianess(webcilHeader.coff_sections);
webcilHeader.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_rva);
webcilHeader.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_cli_header_size);
webcilHeader.pe_debug_rva=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_rva);
webcilHeader.pe_debug_size=BinaryPrimitives.ReverseEndianess(webcilHeader.pe_debug_size);

Comment on lines 72 to 78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
header.version_major=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_major));
header.version_minor=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.version_minor));
header.coff_sections=BinaryPrimitives.ReadUInt16LittleEndian(BitConverter.GetBytes(header.coff_sections));
header.pe_cli_header_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_rva));
header.pe_cli_header_size=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_cli_header_size));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_rva));
header.pe_debug_rva=BinaryPrimitives.ReadUInt32LittleEndian(BitConverter.GetBytes(header.pe_debug_size));
header.version_major=BinaryPrimitives.ReverseEndianess(header.version_major);
header.version_minor=BinaryPrimitives.ReverseEndianess(header.version_minor);
header.coff_sections=BinaryPrimitives.ReverseEndianess(header.coff_sections);
header.pe_cli_header_rva=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_rva);
header.pe_cli_header_size=BinaryPrimitives.ReverseEndianess(header.pe_cli_header_size);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_rva);
header.pe_debug_rva=BinaryPrimitives.ReverseEndianess(header.pe_debug_size);

Comment on lines 375 to 378

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
virtualSize:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualSize)),
virtualAddress:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.VirtualAddress)),
sizeOfRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.SizeOfRawData)),
pointerToRawData:BinaryPrimitives.ReadInt32LittleEndian(BitConverter.GetBytes(secheader.PointerToRawData))
virtualSize:BinaryPrimitives.ReverseEndianess(secheader.VirtualSize),
virtualAddress:BinaryPrimitives.ReverseEndianess(secheader.VirtualAddress),
sizeOfRawData:BinaryPrimitives.ReverseEndianess(secheader.SizeOfRawData),
pointerToRawData:BinaryPrimitives.ReverseEndianess(secheader.PointerToRawData)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Can you try removing the #if if it compiles? This function might be available on all frameworks (unless I'm missing something).

@saitama951saitama951Sep 19, 2023

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out. I have done the same for the WriteStructure as well and hence removing the endian checks for net472 as well. making it generic :)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a little more subtle than just having the API on net472. Because this code is used in an MSBuild task that might be called from a build in Visual Studio, the API has to be in whatever old version of the out of band assembly is already loaded in to the MSBuild process that invokes the task.

@saitama951 Hmm... actually we might also need to add this:

 <PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

to src/tasks/Microsoft.NET.WebAssembly.Webcil/Microsoft.NET.WebAssembly.Webcil.csproj

@ghost

Copy link
Copy Markdown

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception

The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.

We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.

fixes #91893
cc: @omajid@iii-i@uweigand

Author:saitama951
Assignees:-
Labels:

arch-wasm, area-Build-mono, community-contribution

Milestone:-

@marafmaraf added this to the 9.0.0 milestone Sep 20, 2023

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@saitama951

Copy link
Copy Markdown
ContributorAuthor

thanks!

Update Please add the PackageReference to the csproj (see #92274 (comment))

@lambdageek I have done the suggested changes, Thank you for the input.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />
<PackageReferenceInclude="System.Memory"Version="$(SystemMemoryVersion)" />

Comment threadsrc/tasks/Microsoft.NET.WebAssembly.Webcil/WebcilConverter.cs Outdated
'dotnet new blazorwasm' command failed on s390x and was throwing a not implemented exception
The issue was with with the WebCil writer and reader, specific endianness conversions relating to the webcil payload were not implemented for big endian machines.
We considered fixing the generic implementation, but there were only two structures in use: WebcilHeader and WebcilSectionHeader, so it was easier to handle them explicitly.
@saitama951

Copy link
Copy Markdown
ContributorAuthor

@lambdageek@radical@teo-tsirpanis can this be merged?

@radical

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@radical

Copy link
Copy Markdown
Member

WasmBuildTests has relevant failures, like:

 [] Stack overflow.
[] Repeat 74781 times:
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteStructure(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] --------------------------------
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteHeader(System.IO.Stream, Microsoft.NET.WebAssembly.Webcil.WebcilHeader)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.WriteConversionTo(System.IO.Stream, System.IO.FileStream, PEFileInfo, WCFileInfo)
[] at Microsoft.NET.WebAssembly.Webcil.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WebcilConverter.ConvertToWebcil()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilder.ExecuteInternal()
[] at Microsoft.WebAssembly.Build.Tasks.WasmAppBuilderBaseTask.Execute()
[] at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteInstantiatedTask>d__25, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteInstantiatedTask>d__25 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<InitializeAndExecuteTask>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<InitializeAndExecuteTask>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder.InitializeAndExecuteTask(Microsoft.Build.BackEnd.Logging.TaskLoggingContext, Microsoft.Build.BackEnd.ItemBucket, System.Collections.Generic.IDictionary`2<System.String,System.String>, Microsoft.Build.BackEnd.TaskHost, Microsoft.Build.BackEnd.TaskExecutionMode)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteBucket>d__19, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteBucket>d__19 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__18, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__18 ByRef)
[] at Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TaskBuilder+<ExecuteTask>d__13, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTask>d__13 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ProcessBucket>d__51, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessBucket>d__51 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetEntry+<ExecuteTarget>d__44, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ExecuteTarget>d__44 ByRef)
[] at Microsoft.Build.BackEnd.TargetEntry.ExecuteTarget(Microsoft.Build.BackEnd.ITaskBuilder, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.TargetBuilder+<ProcessTargetStack>d__23, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ProcessTargetStack>d__23 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.ProcessTargetStack(Microsoft.Build.BackEnd.ITaskBuilder)
[] at Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.TargetBuilder+<BuildTargets>d__10, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildTargets>d__10 ByRef)
[] at Microsoft.Build.BackEnd.TargetBuilder.BuildTargets(Microsoft.Build.BackEnd.Logging.ProjectLoggingContext, Microsoft.Build.BackEnd.BuildRequestEntry, Microsoft.Build.BackEnd.IRequestBuilderCallback, System.String[], Microsoft.Build.BackEnd.Lookup, System.Threading.CancellationToken)
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildProject>d__68, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildProject>d__68 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildProject()
[] at Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<BuildAndReport>d__59, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<BuildAndReport>d__59 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.BuildAndReport()
[] at Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58.MoveNext()
[] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Microsoft.Build.BackEnd.RequestBuilder+<RequestThreadProc>d__58, Microsoft.Build, Version=15.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<RequestThreadProc>d__58 ByRef)
[] at Microsoft.Build.BackEnd.RequestBuilder.RequestThreadProc(Boolean)
[] at System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].InnerInvoke()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
[] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
[] at System.Threading.Tasks.Task.ExecuteEntry()
[] at Microsoft.Build.BackEnd.RequestBuilder+DedicatedThreadsTaskScheduler.<InjectThread>b__6_0()
[] at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)

@lambdageeklambdageek left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops

@lambdageek

Copy link
Copy Markdown
Member

Moved the endianess fixup for WebcilHeader to WriteHeader instead of an overload of WriteStructure /cc @saitama951

@lambdageek

Copy link
Copy Markdown
Member

/azp run runtime-wasm

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@uweigand

Copy link
Copy Markdown
Contributor

This is failing in .NET 8 - can this be merged there?

@lambdageek

Copy link
Copy Markdown
Member

/backport to release/8.0

@github-actions

Copy link
Copy Markdown
Contributor

Started backporting to release/8.0: https://github.com/dotnet/runtime/actions/runs/6278015202

@ghostghost locked as resolved and limited conversation to collaborators Oct 22, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasmWebAssembly architecturearea-Build-monocommunity-contributionIndicates that the PR has been added by a community member

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The "ConvertDllsToWebCil" task failed: System.NotImplementedException on s390x

6 participants

@saitama951@radical@lambdageek@uweigand@teo-tsirpanis@maraf