Skip to content

File.CopyTo: try to pre-allocate disk space for large destination files - #61676

Closed
adamsitnik wants to merge 3 commits into
dotnet:mainfrom
adamsitnik:fileCopyUsePreallocationSize
Closed

File.CopyTo: try to pre-allocate disk space for large destination files#61676
adamsitnik wants to merge 3 commits into
dotnet:mainfrom
adamsitnik:fileCopyUsePreallocationSize

Conversation

@adamsitnik

@adamsitnikadamsitnik commented Nov 16, 2021•
edited
Loading

Copy link
Copy Markdown
Member

Using the dotnet/performance#2134 benchmarks I got the following results:

BenchmarkDotNet=v0.13.1.1616-nightly, OS=ubuntu 18.04
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
MethodToolchainsizeMeanRatio
CopyTo/main/corerun51233.81 us1.00
CopyTo/after/corerun51233.91 us1.00
CopyToOverwrite/main/corerun51237.15 us1.00
CopyToOverwrite/after/corerun51237.36 us1.01
CopyTo/main/corerun409634.06 us1.00
CopyTo/after/corerun409633.46 us0.98
CopyToOverwrite/main/corerun409637.17 us1.00
CopyToOverwrite/after/corerun409637.68 us1.01
CopyTo/main/corerun1048576499.15 us1.00
CopyTo/after/corerun1048576489.84 us0.98
CopyToOverwrite/main/corerun10485761,064.98 us1.00
CopyToOverwrite/after/corerun1048576473.70 us0.45
CopyTo/main/corerun10485760058,511.27 us1.00
CopyTo/after/corerun10485760055,992.31 us0.96
CopyToOverwrite/main/corerun104857600102,380.19 us1.00
CopyToOverwrite/after/corerun10485760055,508.35 us0.54

As we can see there is no regression for small files. For other files we can see:

  • 2-5% improvement for File.CopyTo where we write to a new file.
  • 45-55% improvement for File.CopyTo(overwrite: true) where we overwrite an existing file. To be honest I am surprised that the gain is so huge.

@adamsitnikadamsitnik added area-System.IO os-linux Linux OS (any supported distro) tenet-performance Performance related issue labels Nov 16, 2021
@adamsitnikadamsitnik added this to the 7.0.0 milestone Nov 16, 2021
@ghost

ghost commented Nov 16, 2021

Copy link
Copy Markdown

Tagging subscribers to this area: @dotnet/area-system-io
See info in area-owners.md if you want to be subscribed.

Issue Details

I am currently re-running all the dotnet/performance#2134 benchmarks and will post the results soon. From initial quick run I was able to get up to x2 for large files.

Author:adamsitnik
Assignees:-
Labels:

area-System.IO, os-linux, tenet-performance

Milestone:7.0.0

@adamsitnik
adamsitnikforce-pushed the fileCopyUsePreallocationSize branch from 15922c7 to 3a27e9fCompareNovember 16, 2021 18:43
@adamsitnik
adamsitnik requested a review from tmdsNovember 16, 2021 19:10
@tmds

tmds commented Nov 16, 2021

Copy link
Copy Markdown
Member

Using the dotnet/performance#2134 benchmarks I got the following results:

I see these benchmarks do a single copy.
When I ran a similar benchmarks, BenchmarkDotNet told me to add a loop to reduce the variance.

What file system is used in on the benchmark machine? ext4?

where we overwrite an existing file. To be honest I am surprised that the gain is so huge.

Looking at the absolute numbers: CopyToOverwrite was twice as slow as CopyTo. That is the real surprise (to me).

@adamsitnik

adamsitnik commented Nov 17, 2021

Copy link
Copy Markdown
MemberAuthor

I see these benchmarks do a single copy.
When I ran a similar benchmarks, BenchmarkDotNet told me to add a loop to reduce the variance.

Is there any chance that your benchmarks were using [IterationSetup] attribute? (doc)

Or performing an operation that runs very slow at first execution but performs much better for every next call? (in such case BDN could under estimate the number of invocations per iteration).

Do you have the source code by any chance?

@adamsitnik

adamsitnik commented Nov 17, 2021•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

ext4?

Yes, ext4.

That is the real surprise (to me).

To me as well. @tmds Is there any chance you could run them on your machine and share the results?

git clone https://github.com/dotnet/performance.git
python3 ./performance/scripts/benchmarks_ci.py -f net7.0 --filter '*File.CopyTo*'

@adamsitnik

adamsitnik commented Nov 17, 2021

Copy link
Copy Markdown
MemberAuthor

My macOS and Ubuntu 18.04 results:

BenchmarkDotNet=v0.13.1.1616-nightly, OS=macOS Big Sur 11.4 (20F71) [Darwin 20.5.0]
Intel Core i7-5557U CPU 3.10GHz (Broadwell), 1 CPU, 4 logical and 2 physical cores
.NET SDK=7.0.100-alpha.1.21566.20
[Host] : .NET 7.0.0 (7.0.21.56201), X64 RyuJIT
Job-VREZES : .NET 7.0.0 (7.0.21.56201), X64 RyuJIT
MethodsizeMeanErrorStdDevMedianMinMaxAllocated
CopyTo512559.5 us32.22 us95.49 us535.2 us496.7 us1,092.8 us130 B
CopyToOverwrite512353.0 us3.66 us10.78 us350.8 us344.2 us435.5 us129 B
CopyTo4096530.8 us8.70 us25.64 us530.5 us488.9 us688.9 us130 B
CopyToOverwrite4096340.7 us4.49 us13.23 us338.0 us332.6 us460.3 us129 B
CopyTo10485761,828.3 us17.39 us51.28 us1,812.8 us1,780.4 us2,164.7 us134 B
CopyToOverwrite10485761,689.8 us42.78 us126.82 us1,643.4 us1,602.5 us2,181.6 us140 B
CopyTo104857600155,437.3 us11,214.75 us33,242.03 us152,217.0 us94,939.3 us252,142.1 us872 B
CopyToOverwrite104857600176,698.7 us10,668.87 us31,623.98 us172,378.7 us125,061.9 us259,941.0 us872 B
BenchmarkDotNet=v0.13.1.1616-nightly, OS=ubuntu 18.04
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=7.0.100-alpha.1.21566.20
[Host] : .NET 7.0.0 (7.0.21.56201), X64 RyuJIT
Job-GTINUV : .NET 7.0.0 (7.0.21.56201), X64 RyuJIT
MethodsizeMeanErrorStdDevMedianMinMaxAllocated
CopyTo51233.57 us0.958 us1.025 us33.34 us32.19 us36.04 us128 B
CopyToOverwrite51237.21 us1.421 us1.637 us36.63 us35.51 us40.92 us128 B
CopyTo409634.78 us1.372 us1.580 us34.04 us33.08 us37.77 us128 B
CopyToOverwrite409638.26 us1.711 us1.971 us37.71 us35.78 us41.32 us128 B
CopyTo1048576493.21 us3.632 us3.033 us492.28 us490.76 us500.19 us130 B
CopyToOverwrite10485761,069.77 us9.813 us9.179 us1,072.09 us1,049.49 us1,083.27 us133 B
CopyTo10485760056,528.31 us139.447 us130.439 us56,504.66 us56,347.31 us56,767.74 us538 B
CopyToOverwrite104857600108,496.90 us2,088.556 us1,851.450 us108,550.01 us104,995.72 us112,027.34 us580 B

@tmds

tmds commented Nov 17, 2021

Copy link
Copy Markdown
Member

On my machine, which has Fedora 34 with btrfs CopyTo and CopyToOverwrite perform similar.

Adding the fallocate will probably regress things a little, but not for copies between same partition which will use FICLONE.

MethodsizeMeanErrorStdDevMedianMinMaxAllocated
CopyTo51216.72 us0.312 us0.277 us16.69 us16.30 us17.34 us128 B
CopyToOverwrite51210.82 us0.143 us0.134 us10.81 us10.62 us11.10 us128 B
CopyTo409616.93 us0.202 us0.189 us16.86 us16.71 us17.24 us128 B
CopyToOverwrite409610.84 us0.076 us0.067 us10.82 us10.71 us10.98 us128 B
CopyTo1048576322.75 us6.931 us7.704 us322.59 us309.68 us336.30 us129 B
CopyToOverwrite1048576320.77 us9.420 us10.849 us323.43 us297.78 us334.26 us129 B
CopyTo10485760041,254.24 us695.573 us616.608 us41,229.77 us40,074.01 us42,357.61 us264 B
CopyToOverwrite10485760041,350.87 us798.138 us746.579 us41,123.53 us40,089.23 us42,685.22 us264 B

I see the same thing when using tmpfs. tmpfs doesn't support FICLONE, so a small regression is expected from adding fallocate.

MethodsizeMeanErrorStdDevMedianMinMaxAllocated
CopyTo51216.86 us0.238 us0.222 us16.83 us16.43 us17.22 us128 B
CopyToOverwrite51210.70 us0.182 us0.162 us10.67 us10.50 us11.07 us128 B
CopyTo409617.23 us0.215 us0.201 us17.25 us16.90 us17.59 us128 B
CopyToOverwrite409610.91 us0.198 us0.185 us10.87 us10.72 us11.27 us128 B
CopyTo1048576325.38 us7.286 us8.391 us324.18 us306.64 us341.47 us129 B
CopyToOverwrite1048576321.10 us6.859 us7.899 us321.20 us304.62 us333.13 us129 B
CopyTo10485760046,025.87 us690.920 us646.287 us45,981.90 us45,194.21 us47,076.94 us332 B
CopyToOverwrite10485760044,983.72 us598.061 us559.427 us44,930.80 us44,086.34 us45,915.32 us264 B

The performance improvement is ext4 specific. From the benchmark results, it seems it is faster to delete and create a new file than it is to overwrite a file. It is surprising.

adamsitnik
adamsitnik commented Nov 17, 2021
View reviewed changes
Comment threadsrc/libraries/Native/Unix/System.Native/pal_io.c Outdated
@adamsitnik

adamsitnik commented Nov 17, 2021

Copy link
Copy Markdown
MemberAuthor

Adding the fallocate will probably regress things a little, but not for copies between same partition which will use FICLONE.

The code is placed after the usage of FICLONE, I've also added a check to perform it only if ioctl(FICLONE) has failed to copy the files. So it should definitely not regress the FICLONE.

I'll run the benchmarks against tmpfs and see if I can get the btrfs numbers as well.

@tmds

tmds commented Nov 17, 2021•
edited
Loading

Copy link
Copy Markdown
Member

it seems it is faster to delete and create a new file than it is to overwrite a file.

It would be interesting to see a flame graph that shows where this unexpected slowness comes from.

If I'd make a guess, the ftruncate has a cost on ext4 that depends on the file size that gets truncated. And the fallocate added in this PR undoes that cost.

runtime/src/libraries/System.Private.CoreLib/src/Microsoft/Win32/SafeHandles/SafeFileHandle.Unix.cs

Line 408 in 10e107d

if(Interop.Sys.FTruncate(this,0)<0)

@tmds

tmds commented Nov 18, 2021•
edited
Loading

Copy link
Copy Markdown
Member

If we add fallocate, probably we should handle the errno that indicate there is no space?

The tricky part is that fallocate is file system dependent.
So we're assuming this to be true for any file system:
If the size to be copied is large enough, fallocate will be a gain, or its cost is negligible compared to copying the data.
Maybe it is safe to assume that if fallocate would be really costly, a file system implementation would not implement it and return EOPNOTSUPP.

For ext4 with a target file that gets truncated, the gain is significant.
I think the ext4 behavior may be related to the auto_da_alloc option (https://man7.org/linux/man-pages/man5/ext4.5.html).

@GSPP

GSPP commented Dec 30, 2021

Copy link
Copy Markdown

Regardless of the performance delta, preallocation can be highly advantageous on NTFS to reduce fragmentation. NTFS can fragment absolutely hideously, like placing each cluster in a separate fragment (without necessity). I had good experiences with preallocating.

@tmds

tmds commented Jan 4, 2022

Copy link
Copy Markdown
Member

preallocation can be highly advantageous on NTFS

Preallocation performance benefits are file system type dependent.

afaik, for 'Linux filesystems' improvements have only been observed with ext4.

@adamsitnikadamsitnik self-assigned this Jan 31, 2022
@adamsitnik
Merge remote-tracking branch 'upstream/main' into fileCopyUsePrealloc…
2d6f1a0
…ationSize
# Conflicts:
#	src/native/libs/System.Native/pal_io.c
This was referenced Jan 31, 2022
Merged
Open
@adamsitnik

adamsitnik commented Jan 31, 2022

Copy link
Copy Markdown
MemberAuthor

I can't finish this experiment as I simply have a lot of other more important things to deliver, so I am going to close the PR.

To make sure this opportunity is not lost I've created a new up-for-grabs issue: #64539

@adamsitnik
adamsitnik deleted the fileCopyUsePreallocationSize branch January 31, 2022 15:16
@ghostghost locked as resolved and limited conversation to collaborators Mar 2, 2022
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@tmdstmdsAwaiting requested review from tmds

Assignees

@adamsitnikadamsitnik

Labels

area-System.IOos-linuxLinux OS (any supported distro)tenet-performancePerformance related issue

Projects

None yet

Milestone

7.0.0

Development

Successfully merging this pull request may close these issues.

3 participants

@adamsitnik@tmds@GSPP