Improve iterators

[Altahrim]

See:

• Updater is too slow #593
• Updater recurses into datadirectory even though it's later excluded in createBackup() #507
• Updater's should stop iterating through ALL files in deleteOldFiles() #397

[blizzz]

Please don't forget to set the milestone when opening a PR 😃

[tovine]

What's the outlook on the timeline for this one? I just had an update basically time out and break itself when I decided to look at the updater code - only to find out that it recurses through all files (including user data and old backups) before doing the filtering.
My server is basically stuck at creating backup for an hour, with the apache process disk-limited at 2% CPU usage and about ~250kB/s read. This pull request looks like it should fix all that and make updating Nextcloud bearable again... 🙏

[Altahrim]

No timeline as I don't find time to work on this…
As a workaround you can do a backup manually and use the --no-backup option in the updater

[doits]

As a workaround you can do a backup manually and use the --no-backup option in the updater

Even with this flag it still hangs/iterates at "Delete old files ..." unfortunately, so it is only a partial solution. (My updates take 1+ hour, too.)

[tovine]

No timeline as I don't find time to work on this… As a workaround you can do a backup manually and use the --no-backup option in the updater

Understandable.
Is there much left to be done or is it just a matter of testing?

[doits]

My last update from 29.0.1 to 29.0.2 was much faster though (still with --no-backup)! Only took very few minutes for the "Delete old files ..." step (didn't pay attention to exactly how long because I thought it will run for 1+ hour but it finished much much quicker). Coincidence or was something merged between those releases that made this much faster? I'm puzzled.

[tovine]

Is there a way I can test this fix on my own instance? Looks like the change should work and be safe enough so I'd like to give it a try if possible

[Altahrim]

Just let me know how to proceed (and what you would like me to look for) and I can try.

@tovine That's not an easy one to test since it should fix several issues.

• Updater fail to scan a large data directory: it may be tested with a large (many files) data directory. With this PR, it should be much faster
• Updater fail to remove some files. It was well described here: Update leads to error 500 on step 11 (continue with web based installer) #509 (comment)
  The difficult part is to have the right underlying filesystem. @caco3, if you are still having this issue, maybe you can give a try?

[tovine]

Just let me know how to proceed (and what you would like me to look for) and I can try.

@tovine That's not an easy one to test since it should fix several issues.

• Updater fail to scan a large data directory: it may be tested with a large (many files) data directory. With this PR, it should be much faster
• Updater fail to remove some files. It was well described here: Update leads to error 500 on step 11 (continue with web based installer) #509 (comment)
  The difficult part is to have the right underlying filesystem. @caco3, if you are still having this issue, maybe you can give a try?

My install is at least large enough that each of the backup files and delete old files steps are painfully slow (we're talking hours), so I should definitely notice if this works in that regard.

[marcelstoer]

AFAICS this will also solve #158 and supersede #594.

[tovine]

Awsome to see that this has been merged now - is there any way to download the newest updater so that it will be the one used for my next update?

Is it enough to just grab the latest updater.phar from this repository and run that from the command line, or do I just have to wait and see until I get a future Nextcloud version bundling this new updater?

[caco3]

@Altahrim and others: Thanks for the progress!

I just tested it on hostpoint (with the with the replaced updater.phar and index.php from master) and it updated without major issues (29.0.4 -> 29.0.11).
There was one issue, but I guess it is not related:
It failed during the step Enable maintenance mode: Uncaught TypeError: s is undefined.

Manually enabling the maintenance mode in the config file and pressing retry did not help. How ever after I went back to the admin page and started the updater once more it continued without issues!

Afterwards I tested the update to 30.0.5 (again with the replaced updater.phar and index.php).
This time the Download step failed 😒 . But again it continued when I restarted the updater.

@tovine
Usually it is enough to grab the new version of index.php and updater.phar from the master branch and place it on your nextcloud instance in the folder updater (replace the existing files). Afterwards head to the the Nextcloud Web UI and run the update.

[come-nc]

It failed during the step Enable maintenance mode: Uncaught TypeError: s is undefined.

Is that a PHP error or a JS one from the UI?

[come-nc]

It failed during the step Enable maintenance mode: Uncaught TypeError: s is undefined.

Is that a PHP error or a JS one from the UI?

I think it’s a JS error and could be fixed by #591
But most likely there was a backend error that got hidden by this frontend issue.

[caco3]

It failed during the step Enable maintenance mode: Uncaught TypeError: s is undefined.

Is that a PHP error or a JS one from the UI?

JS

[tovine]

@tovine Usually it is enough to grab the new version of index.php and updater.phar from the master branch and place it on your nextcloud instance in the folder updater (replace the existing files). Afterwards head to the the Nextcloud Web UI and run the update.

I just tried now (downloaded latest master version of index.php and updater.phar) and it worked like a charm - the complete update process (including waiting for me who was slow to press enter) took just over 2.5 minutes, whereas previously it would typically be stuck at the backup and delete old files steps for a few hours each.
As far as I can tell everything is working as it should after the NC update as well.

Thanks a lot for fixing this, @Altahrim and @come-nc! ❤️

PS: FYI this update was run in the command line using updater.phar

Nextcloud Updater - version: v31.0.0rc3-33-g422fe88 dirty

Current version is 30.0.5.

Update to Nextcloud 30.0.6 available. (channel: "stable")
[The rest of the update proceeeded as usual]

[doits]

Big 👍 from me too, upgrade speed went from 30+ minutes to less than 30 seconds! thank you very much!

[skjnldsv]

/backport e23bd54 to stable31

[skjnldsv]

/backport e23bd54 to stable30