Skip to content

[release/7.0] Properly handle filesizes larger than 8 Gb - #76756

Merged
carlossanlop merged 4 commits into
release/7.0from
backport/pr-76707-to-release/7.0
Oct 10, 2022
Merged

[release/7.0] Properly handle filesizes larger than 8 Gb#76756
carlossanlop merged 4 commits into
release/7.0from
backport/pr-76707-to-release/7.0

Conversation

@github-actions

@github-actionsgithub-actionsBot commented Oct 7, 2022

Copy link
Copy Markdown
Contributor

Backport of #76707 to release/7.0

/cc @jozkee

Customer Impact

  • Right now, with any format, an entry greater than 2GB will cause an overflow exception to be thrown
    • The fix addresses this and we can now handle sizes up to what the spec allows, which is 11 octal digits
  • Right now, with V7 and Ustar, an entry greater than 8GB will be silently truncated
    • The fix will make these scenarios throw exceptions since those formats don't support >8GB entries
  • Right now, with other formats, an entry greater than 8GB will also be silently truncated
    • The fix will make these entries get written correctly with "unlimited" sizes
  • PAX is the only entry format that allows entry sizes greater than 8GB (working around the metadata limit)
    • PAX is our default entry format
    • The fix ensures PAX entries can exceed that 8GB limit
  • Impact to dotnet/sdk-container-builds
    • These builds are currently using the GNU format (Microsoft.NET.Build.Containers/Layer.cs) which is subject to the 8GB limit (and the current 2GB size parsing bug)
    • With this fix, dotnet/sdk-container-builds would only be subject to the 8GB limit
    • With this fix, dotnet/sdk-container-builds could switch to PAX and support entries greater than 8GB

Testing

  • Tested roundtripping with 8 Gb files (i.e. tar field size = 8,589,934,591) on all formats.
  • Tested with size 8 Gb + 1 and confirmed formats V7, Ustar, Gnu correctly throw since that wouldn't fit in the 11 octal bytes of the tar field size.
  • Tested roundtripping with 8 Gb + 1 on Pax format, since pax allows to write sizes past the limit as extended properties.

Risk

New feature, no regression involved. There's risk on the newly added tests since those create 8 Gb files, but they are marked as outerloop and parallelization is disabled.

IMPORTANT: Is this backport for a servicing release? If so and this change touches code that ships in a NuGet package, please make certain that you have added any necessary package authoring and gotten it explicitly reviewed.

@jozkeejozkee added this to the 7.0.0 milestone Oct 7, 2022
@jozkeejozkee added the Servicing-consider Issue for next servicing release review label Oct 7, 2022
@ghost

ghost commented Oct 7, 2022

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

Backport of #76707 to release/7.0

/cc @jozkee

Customer Impact

Testing

Risk

IMPORTANT: Is this backport for a servicing release? If so and this change touches code that ships in a NuGet package, please make certain that you have added any necessary package authoring and gotten it explicitly reviewed.

Author:github-actions[bot]
Assignees:-
Labels:

area-System.IO

Milestone:-

@carlossanlopcarlossanlop added Servicing-approved Approved for servicing release and removed Servicing-consider Issue for next servicing release review labels Oct 7, 2022
@carlossanlop

Copy link
Copy Markdown
Contributor

Approved by Tactics via email.

@jozkee

This comment was marked as off-topic.

@github-actions

This comment was marked as off-topic.

@jozkee

Copy link
Copy Markdown
Member

/azp run runtime-libraries-coreclr outerloop

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines successfully started running 1 pipeline(s).

@carlossanlop

Copy link
Copy Markdown
Contributor

CI failures unrelated. Approved and signed off. Ready to merge. :shipit:

@carlossanlop
carlossanlop merged commit de6b9a3 into release/7.0Oct 10, 2022
@carlossanlop
carlossanlop deleted the backport/pr-76707-to-release/7.0 branch October 10, 2022 15:58
@ghostghost locked as resolved and limited conversation to collaborators Nov 9, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

area-System.Formats.TarServicing-approvedApproved for servicing release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@carlossanlop@jozkee