Skip to content

Tar: Improve unseekable stream handling - #84279

Merged
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream
Jun 1, 2023
Merged

Tar: Improve unseekable stream handling#84279
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
Contributor

Fixes#76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A new path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.

@ghost

ghost commented Apr 4, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A newnew path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.
Author:carlossanlop
Assignees:carlossanlop
Labels:

area-System.Formats.Tar

Milestone:-

@carlossanlop

carlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
ContributorAuthor

I'll remove the new file TarFile.CreateFromDirectoryAsync.File.Roundtrip.cs and will submit it in a separate PR. It isn't related to this change.
Edit: Merged in separate PR: #84303

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it LGTM, but it would be nice if we could reduce the code duplication before merging.

Thank you @carlossanlop !

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@adamsitnikadamsitnik added this to the 8.0.0 milestone May 17, 2023

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply my suggestions before merging.

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@carlossanlop

Copy link
Copy Markdown
ContributorAuthor

Failures are unrelated.

@carlossanlop
carlossanlop merged commit 84a7be7 into dotnet:mainJun 1, 2023
@carlossanlop
carlossanlop deleted the TarUnseekableDataStream branch June 1, 2023 03:35
@ghostghost locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling when setting an unseekable stream as a tar entry's DataStream

3 participants

@carlossanlop@adamsitnik@ViktorHofer
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Tar: Improve unseekable stream handling by carlossanlop · Pull Request #84279 · dotnet/runtime · GitHub
Skip to content

Tar: Improve unseekable stream handling - #84279

Merged
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream
Jun 1, 2023
Merged

Tar: Improve unseekable stream handling#84279
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
Contributor

Fixes#76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A new path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.

@ghost

ghost commented Apr 4, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A newnew path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.
Author:carlossanlop
Assignees:carlossanlop
Labels:

area-System.Formats.Tar

Milestone:-

@carlossanlop

carlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
ContributorAuthor

I'll remove the new file TarFile.CreateFromDirectoryAsync.File.Roundtrip.cs and will submit it in a separate PR. It isn't related to this change.
Edit: Merged in separate PR: #84303

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it LGTM, but it would be nice if we could reduce the code duplication before merging.

Thank you @carlossanlop !

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@adamsitnikadamsitnik added this to the 8.0.0 milestone May 17, 2023

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply my suggestions before merging.

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@carlossanlop

Copy link
Copy Markdown
ContributorAuthor

Failures are unrelated.

@carlossanlop
carlossanlop merged commit 84a7be7 into dotnet:mainJun 1, 2023
@carlossanlop
carlossanlop deleted the TarUnseekableDataStream branch June 1, 2023 03:35
@ghostghost locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling when setting an unseekable stream as a tar entry's DataStream

3 participants

@carlossanlop@adamsitnik@ViktorHofer
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Tar: Improve unseekable stream handling by carlossanlop · Pull Request #84279 · dotnet/runtime · GitHub
Skip to content

Tar: Improve unseekable stream handling - #84279

Merged
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream
Jun 1, 2023
Merged

Tar: Improve unseekable stream handling#84279
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
Contributor

Fixes#76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A new path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.

@ghost

ghost commented Apr 4, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A newnew path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.
Author:carlossanlop
Assignees:carlossanlop
Labels:

area-System.Formats.Tar

Milestone:-

@carlossanlop

carlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
ContributorAuthor

I'll remove the new file TarFile.CreateFromDirectoryAsync.File.Roundtrip.cs and will submit it in a separate PR. It isn't related to this change.
Edit: Merged in separate PR: #84303

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it LGTM, but it would be nice if we could reduce the code duplication before merging.

Thank you @carlossanlop !

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@adamsitnikadamsitnik added this to the 8.0.0 milestone May 17, 2023

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply my suggestions before merging.

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@carlossanlop

Copy link
Copy Markdown
ContributorAuthor

Failures are unrelated.

@carlossanlop
carlossanlop merged commit 84a7be7 into dotnet:mainJun 1, 2023
@carlossanlop
carlossanlop deleted the TarUnseekableDataStream branch June 1, 2023 03:35
@ghostghost locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling when setting an unseekable stream as a tar entry's DataStream

3 participants

@carlossanlop@adamsitnik@ViktorHofer
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Tar: Improve unseekable stream handling by carlossanlop · Pull Request #84279 · dotnet/runtime · GitHub
Skip to content

Tar: Improve unseekable stream handling - #84279

Merged
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream
Jun 1, 2023
Merged

Tar: Improve unseekable stream handling#84279
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
Contributor

Fixes#76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A new path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.

@ghost

ghost commented Apr 4, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A newnew path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.
Author:carlossanlop
Assignees:carlossanlop
Labels:

area-System.Formats.Tar

Milestone:-

@carlossanlop

carlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
ContributorAuthor

I'll remove the new file TarFile.CreateFromDirectoryAsync.File.Roundtrip.cs and will submit it in a separate PR. It isn't related to this change.
Edit: Merged in separate PR: #84303

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it LGTM, but it would be nice if we could reduce the code duplication before merging.

Thank you @carlossanlop !

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@adamsitnikadamsitnik added this to the 8.0.0 milestone May 17, 2023

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply my suggestions before merging.

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@carlossanlop

Copy link
Copy Markdown
ContributorAuthor

Failures are unrelated.

@carlossanlop
carlossanlop merged commit 84a7be7 into dotnet:mainJun 1, 2023
@carlossanlop
carlossanlop deleted the TarUnseekableDataStream branch June 1, 2023 03:35
@ghostghost locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling when setting an unseekable stream as a tar entry's DataStream

3 participants

@carlossanlop@adamsitnik@ViktorHofer
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Tar: Improve unseekable stream handling by carlossanlop · Pull Request #84279 · dotnet/runtime · GitHub
Skip to content

Tar: Improve unseekable stream handling - #84279

Merged
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream
Jun 1, 2023
Merged

Tar: Improve unseekable stream handling#84279
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
Contributor

Fixes#76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A new path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.

@ghost

ghost commented Apr 4, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A newnew path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.
Author:carlossanlop
Assignees:carlossanlop
Labels:

area-System.Formats.Tar

Milestone:-

@carlossanlop

carlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
ContributorAuthor

I'll remove the new file TarFile.CreateFromDirectoryAsync.File.Roundtrip.cs and will submit it in a separate PR. It isn't related to this change.
Edit: Merged in separate PR: #84303

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it LGTM, but it would be nice if we could reduce the code duplication before merging.

Thank you @carlossanlop !

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@adamsitnikadamsitnik added this to the 8.0.0 milestone May 17, 2023

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply my suggestions before merging.

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@carlossanlop

Copy link
Copy Markdown
ContributorAuthor

Failures are unrelated.

@carlossanlop
carlossanlop merged commit 84a7be7 into dotnet:mainJun 1, 2023
@carlossanlop
carlossanlop deleted the TarUnseekableDataStream branch June 1, 2023 03:35
@ghostghost locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling when setting an unseekable stream as a tar entry's DataStream

3 participants

@carlossanlop@adamsitnik@ViktorHofer
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Tar: Improve unseekable stream handling by carlossanlop · Pull Request #84279 · dotnet/runtime · GitHub
Skip to content

Tar: Improve unseekable stream handling - #84279

Merged
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream
Jun 1, 2023
Merged

Tar: Improve unseekable stream handling#84279
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
Contributor

Fixes#76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A new path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.

@ghost

ghost commented Apr 4, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A newnew path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.
Author:carlossanlop
Assignees:carlossanlop
Labels:

area-System.Formats.Tar

Milestone:-

@carlossanlop

carlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
ContributorAuthor

I'll remove the new file TarFile.CreateFromDirectoryAsync.File.Roundtrip.cs and will submit it in a separate PR. It isn't related to this change.
Edit: Merged in separate PR: #84303

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it LGTM, but it would be nice if we could reduce the code duplication before merging.

Thank you @carlossanlop !

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@adamsitnikadamsitnik added this to the 8.0.0 milestone May 17, 2023

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply my suggestions before merging.

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@carlossanlop

Copy link
Copy Markdown
ContributorAuthor

Failures are unrelated.

@carlossanlop
carlossanlop merged commit 84a7be7 into dotnet:mainJun 1, 2023
@carlossanlop
carlossanlop deleted the TarUnseekableDataStream branch June 1, 2023 03:35
@ghostghost locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling when setting an unseekable stream as a tar entry's DataStream

3 participants

@carlossanlop@adamsitnik@ViktorHofer
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Tar: Improve unseekable stream handling by carlossanlop · Pull Request #84279 · dotnet/runtime · GitHub
Skip to content

Tar: Improve unseekable stream handling - #84279

Merged
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream
Jun 1, 2023
Merged

Tar: Improve unseekable stream handling#84279
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
Contributor

Fixes#76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A new path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.

@ghost

ghost commented Apr 4, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A newnew path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.
Author:carlossanlop
Assignees:carlossanlop
Labels:

area-System.Formats.Tar

Milestone:-

@carlossanlop

carlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
ContributorAuthor

I'll remove the new file TarFile.CreateFromDirectoryAsync.File.Roundtrip.cs and will submit it in a separate PR. It isn't related to this change.
Edit: Merged in separate PR: #84303

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it LGTM, but it would be nice if we could reduce the code duplication before merging.

Thank you @carlossanlop !

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@adamsitnikadamsitnik added this to the 8.0.0 milestone May 17, 2023

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply my suggestions before merging.

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@carlossanlop

Copy link
Copy Markdown
ContributorAuthor

Failures are unrelated.

@carlossanlop
carlossanlop merged commit 84a7be7 into dotnet:mainJun 1, 2023
@carlossanlop
carlossanlop deleted the TarUnseekableDataStream branch June 1, 2023 03:35
@ghostghost locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling when setting an unseekable stream as a tar entry's DataStream

3 participants

@carlossanlop@adamsitnik@ViktorHofer
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Tar: Improve unseekable stream handling by carlossanlop · Pull Request #84279 · dotnet/runtime · GitHub
Skip to content

Tar: Improve unseekable stream handling - #84279

Merged
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream
Jun 1, 2023
Merged

Tar: Improve unseekable stream handling#84279
carlossanlop merged 22 commits into
dotnet:mainfrom
carlossanlop:TarUnseekableDataStream

Conversation

@carlossanlop

@carlossanlopcarlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
Contributor

Fixes#76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A new path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.

@ghost

ghost commented Apr 4, 2023

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-formats-tar
See info in area-owners.md if you want to be subscribed.

Issue Details

Fixes #76690

Addresses the issue reported in dotnet/sdk-container-builds#192 (comment) by @rainersigwald

Before this change, it wasn't possible to write into an archive a file entry that contained an unseekable data stream. The undelying cause was that we were calling Length to try to get the total number of bytes of the data stream, but it is unsupported when the stream's CanSeek is false.

The way I fixed this was by separating the code that writes an entry into two possible paths:

  • One path for entries with seekable data streams, which remains mostly unmodified.

  • A newnew path for entries with unseekable streams, which changes the order in which we collect the entry information into the buffer:

    • First, write into the buffer all the metadata except for the size field.
    • Then, write the data stream into the archive (this is possible because we know the exact location where the data should start), counting the number of bytes that are written.
    • Afterwards, write that collected length value in the location for the size field in the buffer.
    • Finally, calculate the checksum, then write the buffer into the archive.
Author:carlossanlop
Assignees:carlossanlop
Labels:

area-System.Formats.Tar

Milestone:-

@carlossanlop

carlossanlop commented Apr 4, 2023

Copy link
Copy Markdown
ContributorAuthor

I'll remove the new file TarFile.CreateFromDirectoryAsync.File.Roundtrip.cs and will submit it in a separate PR. It isn't related to this change.
Edit: Merged in separate PR: #84303

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it LGTM, but it would be nice if we could reduce the code duplication before merging.

Thank you @carlossanlop !

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@adamsitnikadamsitnik added this to the 8.0.0 milestone May 17, 2023

@adamsitnikadamsitnik left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please apply my suggestions before merging.

Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
Comment threadsrc/libraries/System.Formats.Tar/src/System/Formats/Tar/TarHeader.Write.cs Outdated
@carlossanlop

Copy link
Copy Markdown
ContributorAuthor

Failures are unrelated.

@carlossanlop
carlossanlop merged commit 84a7be7 into dotnet:mainJun 1, 2023
@carlossanlop
carlossanlop deleted the TarUnseekableDataStream branch June 1, 2023 03:35
@ghostghost locked as resolved and limited conversation to collaborators Jul 1, 2023
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve handling when setting an unseekable stream as a tar entry's DataStream

3 participants

@carlossanlop@adamsitnik@ViktorHofer