cache webcal calendars on server by georgehrke · Pull Request #10059 · nextcloud/server

georgehrke · 2018-06-29T17:12:10Z

ToDos:

adapt refresh cycle to refresh rate of subscription
- this could actually be tricky. the interval set in the constructor is called before run, but we only know the corresponding subscription when we know the arguments for run
- we can obviously keep track of the last actual update ourself and just run the job once an hour, but I'd prefer if IJobList could just handle that.
update subscription in database when remote returns 301
unit tests

rullzer · 2018-07-02T07:41:22Z

@georgehrke I'd say for now just write a small abstraction class yourself to solve the extra check issue.
Then if it turns out to work corectly etc we can always merge it back to the server part

MorrisJobke · 2018-07-20T09:58:16Z

Most likely nothing for 14 -> moving to 15.

georgehrke · 2018-10-31T11:29:47Z

+ if ($calendarObjectsTable->hasIndex('calobjects_index')) {
+ $calendarObjectsTable->dropIndex('calobjects_index');
+ }
+ $calendarObjectsTable->addUniqueIndex(['calendarid', 'calendartype', 'uri'], 'calobjects_index');


@nickvergessen Will building new indices be an issue? Should it be outsourced to a migration step?

I think for this it is fine. Should be relatively fast.
However, a step first to see if there are duplicated might be needed.
As else it will just explode

It removes the old index consisting of calendarid, uri and creates a new index calendarid, calendartype, uri. And calendartype is the all new column that was introduced by the same migration and set to 0 for all existing entries. So there should be no duplicates.

Ah right. :) Carry on then

@rullzer I was generally not sure what's the cleanest / best approach here.
We have calendars and subscriptions in different tables: calendars respectively calendarsubscriptions. This was no issue so far since we didn't store objects related to subscriptions.
So basically there are two options
Have duplicate tables of calendarchanges, calendarobjects and calendarobjects_props for subscriptions
Introduce calendartype that indicates whether calendarid refers to the calendars or the subscriptions table.

I'm fine with both ways - even if the second one would be cleaner you need to update all the existing queries, right?

Yes, i guess the big question here is whether introducing the new column calendartype is reducing performance of day-to-day queries.

When it is used in queries just add an index to it and it should be fine.

georgehrke · 2018-10-31T11:35:40Z

@tcitworld Can you please test? Are there any other remarks? :)

georgehrke · 2018-11-07T10:20:20Z

The acceptance test results are unrelated.

The phan thing seems to be a faulty php-doc in Sabre/Dav.
https://github.com/nextcloud/3rdparty/blob/214c4155f587f5178d792fe4a839044bdc9982f1/sabre/vobject/lib/Splitter/ICalendar.php#L87

georgehrke · 2018-11-07T10:21:12Z

Should I just use @phan-suppress-next-line?

georgehrke · 2018-11-07T10:27:17Z

Just wanted to send a pull-request, but it's already fixed upstream: https://github.com/georgehrke/vobject/blob/master/lib/Splitter/ICalendar.php#L86

blizzz · 2018-11-07T10:29:29Z

@georgehrke is the upstream released and reasonable to get this in? Otherwise, let's suppress it with a note/issue to remove the suppression once we pull again.

georgehrke · 2018-11-07T10:31:15Z

The fix was not released yet, adding @phan-suppress-next-line ...

georgehrke · 2018-11-07T10:52:20Z

@MorrisJobke How to update phan?
We are running phan 0.11.1, but I need 0.12.9 to be able to use @phan-suppress-next-line

blizzz · 2018-11-07T11:03:40Z

@georgehrke try to push https://github.com/nextcloud/server/blob/master/.drone.yml#L104

kesselb · 2018-11-07T11:03:52Z

+	/**
+ * @param $subscriptionId
+ */
+	public function getSubscriptionById($subscriptionId) {


When int $subscriptionId works for bigint colums i would use it here.

Signed-off-by: Georg Ehrke <developer@georgehrke.com>

MorrisJobke · 2018-11-07T15:01:21Z

Please update the 3rdparty repo due to
nextcloud/3rdparty#185

Signed-off-by: Georg Ehrke <developer@georgehrke.com>

georgehrke · 2018-11-07T16:06:13Z

Failing tests seem unrelated to me

rullzer

Code looks good to me. Lets do this.

ChristophWurst · 2018-11-12T11:39:56Z

FYI: This caused a regression: #12410

BernieO · 2019-08-22T04:53:00Z

I accidentally stumpled accross this when debugging an abnormality with my script calcardbackup and noticed that this is not in the changelog for Nextcloud 15

This commit adds events with the same calendarid belonging to different calendars ("regular" calendars and "subscribed" calendars). To determine to which calender the events belong there is a new database field calendartype added.

All this are quite big changes which are definitely worth to be mentioned in the changelog. Could this new featur please be added to the changelog?

georgehrke added the 2. developing Work in progress label Jun 29, 2018

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 4 times, most recently from 9d83b9f to 2cba151Compare June 30, 2018 12:38

georgehrke added this to the Nextcloud 14 milestone Jun 30, 2018

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 3 times, most recently from 88c8f76 to 864e40fCompare June 30, 2018 20:58

georgehrke mentioned this pull request Jul 1, 2018
Allow (internal) sharing of subscribed webcal calendars #1600
Open

MorrisJobke modified the milestones: Nextcloud 14, Nextcloud 15 Jul 20, 2018

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 4 times, most recently from b072922 to c0868f8Compare October 26, 2018 12:25

tcitworld reviewed Oct 26, 2018
View reviewed changes

Comment threadapps/dav/lib/CalDAV/WebcalCaching/Plugin.php
Comment threadapps/dav/lib/BackgroundJob/RefreshWebcalJob.php

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 4 times, most recently from fb7c1f4 to 1b17e2fCompare October 31, 2018 11:28

georgehrke requested a review from nickvergessen October 31, 2018 11:28

georgehrke commented Oct 31, 2018
View reviewed changes

georgehrke added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Oct 31, 2018

georgehrke requested a review from rullzer October 31, 2018 12:20

tcitworld previously approved these changes Oct 31, 2018
View reviewed changes

Comment threadapps/dav/lib/BackgroundJob/RefreshWebcalJob.php

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch from 31ac4b6 to cc66f65Compare November 7, 2018 11:06

kesselb reviewed Nov 7, 2018
View reviewed changes

MorrisJobke mentioned this pull request Nov 7, 2018
15.0.0 beta 1 #12310
Merged
29 tasks

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch 3 times, most recently from 730c48e to 8967e85Compare November 7, 2018 12:08

cache webcal calendars on server
30d13bb
Signed-off-by: Georg Ehrke <developer@georgehrke.com>

georgehrke mentioned this pull request Nov 7, 2018
manually patch PHP-Doc of VObject Splitter nextcloud/3rdparty#185
Merged

update 3rdparty to fix phan error message
712b79e
Signed-off-by: Georg Ehrke <developer@georgehrke.com>

georgehrke force-pushed the feature/1497/cache_webcal_calendars branch from 8967e85 to 712b79eCompare November 7, 2018 15:11

MorrisJobke added the enhancement label Nov 7, 2018

rullzer approved these changes Nov 8, 2018
View reviewed changes

MorrisJobke merged commit 0fb74af into masterNov 8, 2018

MorrisJobke deleted the feature/1497/cache_webcal_calendars branch November 8, 2018 09:55

ChristophWurst mentioned this pull request Nov 12, 2018
Unknown column 'calendartype' (table calendarobjects) #12410
Closed

nickvergessen mentioned this pull request Nov 14, 2018
Index identifier calendarid_calendartype_synctoken is too long #12446
Closed

BernieO mentioned this pull request Aug 22, 2019
remove orphaned events also from deleted webcal calendars #16833
Closed

tcitworld mentioned this pull request Aug 23, 2019
Remove orphaned calendar data from deleted subscriptions #16835
Merged

redtux mentioned this pull request Sep 14, 2019
Expose subscribed webcal calendars via CalDAV #1497
Closed

Uh oh!

Uh oh!

Conversation

georgehrke commented Jun 29, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rullzer commented Jul 2, 2018

Uh oh!

MorrisJobke commented Jul 20, 2018

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

georgehrke commented Oct 31, 2018

Uh oh!

Uh oh!

georgehrke commented Nov 7, 2018• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

georgehrke commented Nov 7, 2018

Uh oh!

georgehrke commented Nov 7, 2018

Uh oh!

blizzz commented Nov 7, 2018

Uh oh!

georgehrke commented Nov 7, 2018

Uh oh!

georgehrke commented Nov 7, 2018

Uh oh!

blizzz commented Nov 7, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MorrisJobke commented Nov 7, 2018

Uh oh!

georgehrke commented Nov 7, 2018

Uh oh!

rullzer left a comment

Choose a reason for hiding this comment

Uh oh!

ChristophWurst commented Nov 12, 2018

Uh oh!

BernieO commented Aug 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

georgehrke commented Jun 29, 2018•
edited
Loading

georgehrke commented Nov 7, 2018•
edited
Loading