fix: reduce memory consumption of scans

[solracsf]

• Resolves: [Bug]: occ files:scan-app-data consumes too much memory #40919

Summary

array_filter is pretty memory intensive, specially in the context of appdata_ folder where it can have thousands of folders. Construct the array in a loop instead.

The array_filter creates a new array and populates it with elements that meet the filter condition. This means it creates a new array in memory to hold the filtered elements, which can be memory-intensive, especially if the original array $children is large.

On the other hand, a foreach loop initializes an empty array $childFolders and then iterates through the $children array, only adding elements to $childFolders if they meet the filtering condition. This approach doesn't create an additional array for the filtered elements, so it consumes less memory.

Before

After

Test system

+---------+--------+--------------+
| Folders | Files | Elapsed time |
+---------+--------+--------------+
| 217688 | 155452 | 00:09:46 |
+---------+--------+--------------+

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)

[kesselb]

To benchmark it myself, I need to upload a good number of images, pre generate all previews, delete the previews again and run occ files:scan-app-data?

[come-nc]

Summary

array_filter is pretty memory intensive, specially in the context of appdata_ folder where it can have thousands of folders. Construct the array in a loop instead.

The array_filter creates a new array and populates it with elements that meet the filter condition. This means it creates a new array in memory to hold the filtered elements, which can be memory-intensive, especially if the original array $children is large.

On the other hand, a foreach loop initializes an empty array $childFolders and then iterates through the $children array, only adding elements to $childFolders if they meet the filtering condition. This approach doesn't create an additional array for the filtered elements, so it consumes less memory.

I do not understand the difference, you wrote twice the same thing from what I understand.

If array_filter is worse than a loop doing the same thing an issue should be opened with the PHP project.
Did you test on several PHP version for a difference?
Also, maybe it uses more memory because it prioritizes speed?

[come-nc]

https://externals.io/message/100452
Interesting read, 6 years old so not sure if all still apply but it seems a callback function is always expensive speedwise.

[solracsf]

On my tests, on different php projects, the foreach loop was always less memory intensive (i'm not analysing speed/performance here) once the array is big (>hundreds). In this specific case, I've tested with v8.1 only.

[solracsf]

/backport to stable28

[solracsf]

/backport to stable27

[solracsf]

/backport to stable26

[blizzz]

foreach() has a lower memory footprint the most of the array_* functions, so when you can expect to get a few more entries in your arrays, use foreach.

[solracsf]

/backport to stable28

[solracsf]

/skjnldsv-backport to stable28

[solracsf]

/skjnldsv-backport to stable27

[mrtumnus]

Ok, but this made it nearly 2x as slow, so was this really a good tradeoff?
(stumbled across this as I wait for my files:scan-app-data to complete 😆 )