Skip to content

feat: Copy Directory - #563

Closed
simone-panico wants to merge 5 commits into
apple:mainfrom
simone-panico:copy-dir
Closed

feat: Copy Directory#563
simone-panico wants to merge 5 commits into
apple:mainfrom
simone-panico:copy-dir

Conversation

@simone-panico

@simone-panicosimone-panico commented Feb 27, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

This pull request introduces functionality to support copying directories between the host and container #543.

See #1190 for more context.

dcantahand others added 3 commits January 12, 2026 21:44
@dcantah
_ContainerizationTar: Add new native tar reader/writer
7156492
Swift does not have a native tar reader/writer, and we'd really
like to avoid linking more (libarchive) libraries to vminitd if we can get
away with it. For copying directories into/out of containers tar
is fairly nice as it's a simple way to preserve everything you need
to be able to reassemble the directory on the receiving end. Due to this,
I decided to write a somewhat simple tar reader/writer solely for this
purpose. Unfortunately, there's quite a lot of work to get vminitd/ to
compile on macOS without the static linux SDK, so it was making unit testing
these additions quite a pain, so for now the new work lives in _ContainerizationTar
(where the underscore is trying to denote that this is rather experimental..).
This change aims to add a simple tar reader and writer with support pax extended headers
(for long file names and > 8 GiB files). Because its intended purpose is in a
scenario where we own both the creator and ingestor, the reader does NOT handle every
case, but it is good at unarchiving the archives the library has made :)
@simone-panico
Merge branch 'pr-474' into copy-dir
0794a3c
@simone-panico
Add CopyDirIn and CopyDirOut support with _ContainerizationTar
1445e67
@dcantah

dcantah commented Feb 27, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

I don't think we need a new set of rpcs. Presumably we can reuse the copyIn/copyOut and if you need extra info just add it to the proto message

@jglogan

jglogan commented Mar 1, 2026

Copy link
Copy Markdown
Contributor

@simone-panico@dcantah One thing to keep an eye on is when this lands in the swift SDK: swiftlang/swift-docker@ea31a33.

At that point we might be able to replace any homegrown archiver we use with libarchive.

@simone-panico

simone-panico commented Mar 2, 2026

Copy link
Copy Markdown
ContributorAuthor

libarchive

So that means ContainerizationTar is not sure to be merged?

jglogan
jglogan reviewed Mar 2, 2026
View reviewed changes
Comment threadSources/Containerization/Vminitd.swift Outdated
Comment threadSources/Containerization/Vminitd.swift Outdated
@jglogan

jglogan commented Mar 2, 2026

Copy link
Copy Markdown
Contributor

libarchive

So that means ContainerizationTar is not sure to be merged?

Hi Simone! I'm not sure when the updated Swift SDK is going to be available. The plumbing for a tar-based implementation will be the same regardless of whether we have a homegrown tar or libarchive, so I'm fine with merging everything we need for cp support using tar archives.

The main concern I have is that if we implement tar ourselves, it needs to be secure.

jglogan
jglogan requested changes Mar 2, 2026
View reviewed changes
Comment threadSources/Containerization/Vminitd.swift
}

/// Extract a tar archive to a destination directory.
private static func extractTar(reader: TarReader, to destURL: URL) throws {

@jgloganjgloganMar 2, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@simone-panico Have a look at https://github.com/apple/containerization/blob/main/Sources/ContainerizationArchive/ArchiveReader.swift#L304. I had to do similar fixes there back in January 😄.

You're skipping entries containing .., but the implementation can currently write through symlinks there's a TOCTOU race between creating the parent directory for a regular file extraction, and writing the regular file.

The tests for exercising safe extraction are at: https://github.com/simone-panico/containerization/blob/main/Tests/ContainerizationArchiveTests/ArchiveReaderTests.swift

@jglogan

jglogan commented Mar 4, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Dev snapshots are out now:

swift sdk install \
https://download.swift.org/swift-6.3-branch/static-sdk/swift-6.3-DEVELOPMENT-SNAPSHOT-2026-02-27-a/swift-6.3-DEVELOPMENT-SNAPSHOT-2026-02-27-a_static-linux-0.1.0.artifactbundle.tar.gz \
--checksum 7dc6f58d1300438ebcf2e998de7c69d8da0ec42102460bd8e7f91c668e8b30c6

@simone-panico

simone-panico commented Mar 5, 2026

Copy link
Copy Markdown
ContributorAuthor

Dev snapshots are out now:

swift sdk install \
https://download.swift.org/swift-6.3-branch/static-sdk/swift-6.3-DEVELOPMENT-SNAPSHOT-2026-02-27-a/swift-6.3-DEVELOPMENT-SNAPSHOT-2026-02-27-a_static-linux-0.1.0.artifactbundle.tar.gz \
--checksum 7dc6f58d1300438ebcf2e998de7c69d8da0ec42102460bd8e7f91c668e8b30c6

Nice, I will try to switch out the ContainerizatonTar with Libarchive

@dcantah

dcantah commented Mar 5, 2026

Copy link
Copy Markdown
Contributor

@simone-panico Do you mind if we try and handle the Containerization side and you can plumb this through in container? There's quite a few pitfalls I keep thinking of in my head for this that are not simple to solve.

  1. There's not a great holding spot to write the archives to today as the VMs rootfs is read only. We can write to /tmp but this is a tmpfs, and if the archive was decently large we can run the risk of OOM.
  2. libarchive will be synchronous blocking IO, and we need to find a decent approach to run this without stalling the guest (as a user can choose a 1 core guest if they really want). Likely farm the work off to a thread and bridge with a continuation, but the devil is in the details.

There's a lot of solutions to these, but they all have drawbacks to me..

@simone-panico

simone-panico commented Mar 5, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@simone-panico Do you mind if we try and handle the Containerization side and you can plumb this through in container? There's quite a few pitfalls I keep thinking of in my head for this that are not simple to solve.

  1. There's not a great holding spot to write the archives to today as the VMs rootfs is read only. We can write to /tmp but this is a tmpfs, and if the archive was decently large we can run the risk of OOM.
  2. libarchive will be synchronous blocking IO, and we need to find a decent approach to run this without stalling the guest (as a user can choose a 1 core guest if they really want). Likely farm the work off to a thread and bridge with a continuation, but the devil is in the details.

There's a lot of solutions to these, but they all have drawbacks to me..

Yeah sure, go for it. The main goal in the first place for me was to finally merge the container side 😄

@dcantahdcantah closed this Mar 6, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jgloganjgloganjglogan requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@simone-panico@dcantah@jglogan@katiewasnothere