fix(caldav): automatically delete outdated scheduling objects by miaulalala · Pull Request #45235 · nextcloud/server

miaulalala · 2024-05-08T18:42:15Z

Resolves: Remove old scheduling objects from INBOX and oc_schedulingobjects via cron #43621

Summary

oc_schedulingobjects currently grows without ever deleting outdated objects. This PR a repair step that is declared as expensive so admins can decide to run them at their convenience for the initial delete.

The delete is chunked to 50k rows on each transaction so the database isn't locked for a long time (especially in clustered setups this could cause issues). MySQL needs special treatment as it doesn't support LIMITs on DELETE queries, so it does a SELECT on the ids to delete, and then runs the delete on those.

After the repair step has run, a regular cron job is added to the Jobs List that runs every hour to get rid of scheduling objects that are older than an hour. We don't really need them and could theoretically delete them as soon as they're processed by the ITip\Broker but as rooms and resources are also run in a cron job, keeping them until the principal room and resources are added is probably a good idea as I can't exclude unwanted side effects. I also updated the runtime for rooms and resources to run every half hour for that reason.

Checklist

Code is properly formatted
Sign-off message is added to all commits
Tests (unit, integration, api and/or acceptance) are included
Documentation (manuals or wiki) has been updated or is not required
Backports requested where applicable (ex: critical bugfixes)

ChristophWurst

Let's get rid of the hourly full table scan

👍 otherwise

+ ->from('schedulingobjects')
+ ->where($query->expr()->lt('lastmodified', $query->createNamedParameter($modifiedBefore)))
+ ->setMaxResults($limit);
+ $result = $query->executeQuery();


ChristophWurst

Code looks good

ChristophWurst · 2024-05-28T18:08:51Z

+ ITimeFactory $timeFactory,
+	) {
+ parent::__construct($timeFactory);
+ $this->setInterval(23 * 60 * 60);


Why not 24? 😆

@nickvergessen can explain 😉

it would move back all the time, as the run time is not 0 seconds and it can be started with up to 14 minutes delay.
That would mean it would fall further back and back in the "maintenance time window" and eventually skip some day as current time + 24h is outside of maintenance window.
With 23h it will just be saver to run every day.

miaulalala · 2024-05-28T19:18:41Z

/backport to stable29

miaulalala · 2024-05-28T19:18:51Z

/backport to stable28

miaulalala · 2024-05-28T19:19:01Z

/backport to stable27

Signed-off-by: Anna Larch <anna@nextcloud.com>

miaulalala self-assigned this May 8, 2024

miaulalala added 2. developing Work in progress performance 🚀 feature: caldav Related to CalDAV internals labels May 8, 2024

miaulalala added this to the Nextcloud 30 milestone May 8, 2024

github-advanced-securityAI found potential problems May 8, 2024
View reviewed changes

Comment threadapps/dav/lib/Migration/DeleteSchedulingObjectsJobRegistrar.php Fixed

This comment was marked as outdated.
Sign in to view

tcitworld reviewed May 14, 2024
View reviewed changes

Comment threadapps/dav/lib/BackgroundJob/DeleteOutdatedSchedulingObjects.php Outdated

This comment was marked as outdated.
Sign in to view

github-advanced-securityAI found potential problems May 16, 2024
View reviewed changes

Comment threadapps/dav/lib/CalDAV/CalDavBackend.php Fixed
Comment threadlib/private/Log.php Fixed
Comment threadapps/dav/lib/Migration/DeleteSchedulingObjects.php Fixed

github-advanced-securityAI found potential problems May 16, 2024
View reviewed changes

Comment threadapps/dav/lib/AppInfo/Application.php Fixed

miaulalala force-pushed the fix/remove-old-scheduling-objects branch from cbcae93 to f2d7e9fCompare May 16, 2024 18:22

miaulalala marked this pull request as ready for review May 16, 2024 18:23

miaulalala requested a review from ChristophWurst as a code owner May 16, 2024 18:23

miaulalala requested review from a team, Altahrim, nickvergessen, sorbaugh and yemkareems and removed request for a teamMay 16, 2024 18:23

miaulalala marked this pull request as draft May 16, 2024 18:34

ChristophWurst reviewed May 22, 2024
View reviewed changes

github-advanced-securityAI found potential problems May 23, 2024
View reviewed changes

Comment threadapps/settings/lib/SetupChecks/SchedulingTableSize.php Fixed

ChristophWurst reviewed May 27, 2024
View reviewed changes

Comment threadapps/dav/lib/Migration/Version1031Date20240523183435.php Outdated

github-advanced-securityAI found potential problems May 28, 2024
View reviewed changes

miaulalala marked this pull request as ready for review May 28, 2024 15:55

miaulalala added 3. to review Waiting for reviews and removed 2. developing Work in progress labels May 28, 2024

kesselb reviewed May 28, 2024
View reviewed changes

Comment threadapps/dav/lib/CalDAV/CalDavBackend.php

kesselb approved these changes May 28, 2024
View reviewed changes

ChristophWurst approved these changes May 28, 2024
View reviewed changes

miaulalala force-pushed the fix/remove-old-scheduling-objects branch from 1a53766 to 6ba43b3Compare May 28, 2024 19:18

backportbotBot added the backport-request label May 28, 2024

kesselb reviewed May 28, 2024
View reviewed changes

Comment threadapps/dav/lib/BackgroundJob/DeleteOutdatedSchedulingObjects.php
Comment threadapps/dav/lib/Migration/DeleteSchedulingObjects.php
Comment threadapps/settings/lib/SetupChecks/SchedulingTableSize.php

fix(caldav): automatically delete outdated scheduling objects
ad78f7e
Signed-off-by: Anna Larch <anna@nextcloud.com>

miaulalala force-pushed the fix/remove-old-scheduling-objects branch from 6ba43b3 to ad78f7eCompare May 31, 2024 11:14

miaulalala enabled auto-merge May 31, 2024 13:29

miaulalala merged commit a28a970 into masterJun 3, 2024

miaulalala deleted the fix/remove-old-scheduling-objects branch June 3, 2024 11:06

This was referenced Jun 3, 2024
[stable27] fix(caldav): automatically delete outdated scheduling objects #45639
Merged
[stable28] fix(caldav): automatically delete outdated scheduling objects #45640
Merged
[stable29] fix(caldav): automatically delete outdated scheduling objects #45641
Merged

backportbotBot removed the backport-request label Jun 3, 2024

blizzz mentioned this pull request Jul 24, 2024
30.0.0 beta 1 #46713
Merged

joshtrichards mentioned this pull request Jul 31, 2024
[Bug]: php-fpm eats all cpus executing select from oc_schedulingobjects #39758
Closed
8 tasks

Uh oh!

Uh oh!

Conversation

miaulalala commented May 8, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChristophWurst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check notice

Uh oh!

ChristophWurst left a comment

Choose a reason for hiding this comment

Uh oh!

ChristophWurstMay 28, 2024

Choose a reason for hiding this comment

Uh oh!

miaulalalaMay 28, 2024

Choose a reason for hiding this comment

Uh oh!

nickvergessenJun 6, 2024

Choose a reason for hiding this comment

Uh oh!

miaulalala commented May 28, 2024

Uh oh!

miaulalala commented May 28, 2024

Uh oh!

miaulalala commented May 28, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

miaulalala commented May 8, 2024•
edited
Loading