Uh oh!
There was an error while loading. Please reload this page.
FileSystem.Unix: improve CopyFile. - #59695
Conversation
ghost
commented
Sep 28, 2021
Tagging subscribers to this area: @dotnet/area-system-io Issue DetailsLike the upcoming version of GNU coreutils 'cp' prefer a copy-on-write clone. Eliminate a 'stat' call that is always performed for checking if the target is a directory Eliminate a 'stat' call for retrieving the file size of the source by passing through Create the destination with file permissions that match the source. When performing a manual copy, limit the allocated buffer for small files.
|
adamsitnik
commented
Oct 15, 2021
Hi @tmds Could you please provide some benchmark results for small, medium and big files? The change is non-trivial, it would be good to make sure it's worth the complexity. Thanks! |
For small files, we see an improvement due to eliminating syscalls. For large files, this doesn't weigh in, and we see a huge gain on the filesystem that support CoW (my home partition uses Btrfs which supports it, while
Benchmark: usingSystem;usingSystem.Collections.Generic;usingSystem.IO;usingBenchmarkDotNet;usingBenchmarkDotNet.Attributes;namespaceFileCopyBenchmark{publicclassBenchmarks{privateconstintkB=1<<10;privateconstintMB=kB<<10;[Params(0,1,100,512,1*kB,4*kB,1*MB,10*MB)]publicintSourceSize;[ParamsSource(nameof(ValuesForBaseDir))]publicstringBaseDir{get;set;}publicIEnumerable<string>ValuesForBaseDir=>new[]{Path.GetTempPath(),Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments)};privatestringSourceFileName;privatestringDestinationFileName;[GlobalSetup]publicvoidGlobalSetup(){DestinationFileName=Path.Combine(BaseDir,Guid.NewGuid().ToString());SourceFileName=Path.Combine(BaseDir,Guid.NewGuid().ToString());usingFileStreamrandom=File.OpenRead("/dev/random");usingFileStreamsourceFile=File.OpenWrite(SourceFileName);Span<byte>buffer=stackallocbyte[4*kB];intlength=SourceSize;while(length>0){intread=random.Read(buffer.Slice(0,Math.Min(length,buffer.Length)));sourceFile.Write(buffer.Slice(0,read));length-=read;}}[GlobalCleanup]publicvoidGlobalCleanup(){File.Delete(SourceFileName);File.Delete(DestinationFileName);}[Benchmark]publicvoidFileCopy(){for(inti=0;i<100;i++){File.Delete(DestinationFileName);File.Copy(SourceFileName,DestinationFileName);}}}} |
stephentoub
commented
Oct 19, 2021
What is the impact on subsequent writes to the file? |
tmds
commented
Oct 19, 2021
The copies get avoided by the CoW, but then when you write to the file, you are making those copies again. Updating the benchmark to overwrite half of the file after [Benchmark]publicvoidFileCopy(){Span<byte>buffer=stackallocbyte[4*kB];for(inti=0;i<100;i++){File.Delete(DestinationFileName);File.Copy(SourceFileName,DestinationFileName);usingSafeFileHandlewriteHandle=File.OpenHandle(DestinationFileName,FileMode.Open,FileAccess.Write,FileShare.Write);for(intoffset=SourceSize/2;offset<SourceSize;){RandomAccess.Write(writeHandle,buffer,offset);offset+=buffer.Length;}}}
|
Like the upcoming version of GNU coreutils 'cp' prefer a copy-on-write clone. This shares the physical storage between files, which means no data needs to copied. CoW-clones are supported by a number of Linux file systems, like Btrfs, XFS, and overlayfs. Eliminate a 'stat' call that is always performed for checking if the target is a directory by only performing the check when the 'open' syscall reports an error. Eliminate a 'stat' call for retrieving the file size of the source by passing through the length that was retrieved when checking the opened file is not a directory. Create the destination with file permissions that match the source. We still need to fchmod due to umask being applied to the open mode. When performing a manual copy, limit the allocated buffer for small files. And, avoid the last 'read' call by checking when we've copied the expected nr of bytes.
I had messed up the PR by resolving conflicts in the GitHub UI. |
tmds
commented
Oct 20, 2021
I ran the benchmark once more to ensure there are no regressions. I've included some more sizes.
|
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
tmds
commented
Oct 28, 2021
@stephentoub all feedback is addressed and CI is happy. This is good to merge. |
deeprobin
left a comment
There was a problem hiding this comment.
A few small suggestions about coding style, but nothing important.
LGTM
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
tmds
commented
Nov 4, 2021
@stephentoub @dotnet/area-system-io can you merge this? |
jeffhandley
commented
Nov 15, 2021
Thanks as always, @tmds! |
| Interop.Sys.Permissions filePermissions; | ||
| using SafeFileHandle src = SafeFileHandle.OpenReadOnly(sourceFullPath, FileOptions.None, out fileLength, out filePermissions); | ||
| using SafeFileHandle dst = SafeFileHandle.Open(destFullPath, overwrite ? FileMode.Create : FileMode.CreateNew, | ||
| FileAccess.ReadWrite, FileShare.None, FileOptions.None, preallocationSize: 0, openPermissions: filePermissions, |
There was a problem hiding this comment.
I wonder whether it would be beneficial to provide preallocationSize: fileLength here. @tmds what do you think?
Like the upcoming version of GNU coreutils 'cp' prefer a copy-on-write clone.
This shares the physical storage between files, which means no data needs to copied.
CoW-clones are supported by a number of Linux file systems, like Btrfs, XFS, and overlayfs.
Eliminate a 'stat' call that is always performed for checking if the target is a directory
by only performing the check when the 'open' syscall reports an error.
Eliminate a 'stat' call for retrieving the file size of the source by passing through
the length that was retrieved when checking the opened file is not a directory.
Create the destination with file permissions that match the source.
We still need to fchmod due to umask being applied to the open mode.
When performing a manual copy, limit the allocated buffer for small files.
And, avoid the last 'read' call by checking when we've copied the expected nr of bytes.