Adding delayed calculation of annotation values for AdminUsers by jredrejo · Pull Request #2359 · learningequality/studio

jredrejo · 2020-10-02T17:47:49Z

Description

As done in #2333 , here we're caching and delaying to be asynchronously calculated some of the annotated fields for the AdminUserViewSet

Issue Addressed (if applicable)

Slow or impossible view with thousands of users in our Studio database

Implementation Notes (optional)

Added also an api for the frontend to fetch delayed data. Example of use:
http://localhost:8080/api/admin-users/deferred_data?id__in=1,2,3,90

One task to do all the users calculation is implemented (a single user task is not needed by now):
cache_multiple_users_metadata_task

At a high level, how did you implement this?

Values are stored in the cache and recalculated when they are requested if one hour has passed since its last calculation. To avoid cache stampede the expiration time is calculated with the @cache_stampede decorator that uses a randomized gaussian function so not all the calculations are done simultaneously

General view: Calculating annotated fields for the ViewSet in an asynchronous task

when (not) to use this:

Using this approach delays the calculation of the annotated fields. If they are required in real time, don't use it
Using this approach you don't have data in the first request of the field (user, channel, etc.), instead a DEFERRED_FLAG is returned. After the task is run there's always a returned value, but it might have been calculated some time ago, depending on the expire time used in the @cache_stampede decorator
Using this approach you can't filter by the annotated fields

when to use this:

When using this approach the annotated fields are removed from the main queryset, making it much simpler, thus faster. It usually also removes group_by and order_by commands that add more searchs in the database
When using this approach, there is always (excepting the first request) an usable value in the redis cache that can be accessed easily
The implemented stampede method guaranties the value is requested in the expire interval while ensuring not all the requests are done at the same time, to distribute the database work
there is an api for the frontend to fetch easily the cached values
Tasks can be set to calculate the value for a group of items in one single task (to be shown in a table, for example), or to calculate one per task

… for users

codecov · 2020-10-02T18:07:44Z

Codecov Report

Merging #2359 into develop will decrease coverage by 0.23%.
The diff coverage is 28.37%.

@@ Coverage Diff @@## develop #2359 +/- ##
===========================================
- Coverage 80.57% 80.33% -0.24% 
===========================================
Files 299 300 +1 Lines 15168 15238 +70 ===========================================
+ Hits 12221 12241 +20 - Misses 2947 2997 +50

Impacted Files	Coverage Δ
...ontentcuration/contentcuration/viewsets/channel.py	`73.33% <0.00%> (ø)`
contentcuration/contentcuration/viewsets/user.py	`56.28% <25.58%> (-7.87%)`	⬇️
contentcuration/contentcuration/utils/user.py	`29.62% <29.62%> (ø)`
contentcuration/contentcuration/tasks.py	`72.72% <66.66%> (-0.16%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74754d6...723e3b8. Read the comment docs.

rtibbles · 2020-10-05T23:07:15Z

Could we add the PR comment to the dev docs and then merge?

jredrejo · 2020-10-02T17:47:49Z

Description

As done in #2333 , here we're caching and delaying to be asynchronously calculated some of the annotated fields for the AdminUserViewSet

Issue Addressed (if applicable)

Slow or impossible view with thousands of users in our Studio database

Implementation Notes (optional)

Added also an api for the frontend to fetch delayed data. Example of use:
http://localhost:8080/api/admin-users/deferred_data?id__in=1,2,3,90

One task to do all the users calculation is implemented (a single user task is not needed by now):
cache_multiple_users_metadata_task

At a high level, how did you implement this?

Values are stored in the cache and recalculated when they are requested if one hour has passed since its last calculation. To avoid cache stampede the expiration time is calculated with the @cache_stampede decorator that uses a randomized gaussian function so not all the calculations are done simultaneously

General view: Calculating annotated fields for the ViewSet in an asynchronous task

when (not) to use this:

Using this approach delays the calculation of the annotated fields. If they are required in real time, don't use it
Using this approach you don't have data in the first request of the field (user, channel, etc.), instead a DEFERRED_FLAG is returned. After the task is run there's always a returned value, but it might have been calculated some time ago, depending on the expire time used in the @cache_stampede decorator
Using this approach you can't filter by the annotated fields

when to use this:

When using this approach the annotated fields are removed from the main queryset, making it much simpler, thus faster. It usually also removes group_by and order_by commands that add more searchs in the database
When using this approach, there is always (excepting the first request) an usable value in the redis cache that can be accessed easily
The implemented stampede method guaranties the value is requested in the expire interval while ensuring not all the requests are done at the same time, to distribute the database work
there is an api for the frontend to fetch easily the cached values
Tasks can be set to calculate the value for a group of items in one single task (to be shown in a table, for example), or to calculate one per task

… for users

codecov · 2020-10-02T18:07:44Z

Codecov Report

Merging #2359 into develop will decrease coverage by 0.23%.
The diff coverage is 28.37%.

@@ Coverage Diff @@## develop #2359 +/- ##
===========================================
- Coverage 80.57% 80.33% -0.24% 
===========================================
Files 299 300 +1 Lines 15168 15238 +70 ===========================================
+ Hits 12221 12241 +20 - Misses 2947 2997 +50

Impacted Files	Coverage Δ
...ontentcuration/contentcuration/viewsets/channel.py	`73.33% <0.00%> (ø)`
contentcuration/contentcuration/viewsets/user.py	`56.28% <25.58%> (-7.87%)`	⬇️
contentcuration/contentcuration/utils/user.py	`29.62% <29.62%> (ø)`
contentcuration/contentcuration/tasks.py	`72.72% <66.66%> (-0.16%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74754d6...723e3b8. Read the comment docs.

rtibbles · 2020-10-05T23:07:15Z

Could we add the PR comment to the dev docs and then merge?

jredrejo · 2020-10-02T17:47:49Z

Description

As done in #2333 , here we're caching and delaying to be asynchronously calculated some of the annotated fields for the AdminUserViewSet

Issue Addressed (if applicable)

Slow or impossible view with thousands of users in our Studio database

Implementation Notes (optional)

Added also an api for the frontend to fetch delayed data. Example of use:
http://localhost:8080/api/admin-users/deferred_data?id__in=1,2,3,90

One task to do all the users calculation is implemented (a single user task is not needed by now):
cache_multiple_users_metadata_task

At a high level, how did you implement this?

Values are stored in the cache and recalculated when they are requested if one hour has passed since its last calculation. To avoid cache stampede the expiration time is calculated with the @cache_stampede decorator that uses a randomized gaussian function so not all the calculations are done simultaneously

General view: Calculating annotated fields for the ViewSet in an asynchronous task

when (not) to use this:

Using this approach delays the calculation of the annotated fields. If they are required in real time, don't use it
Using this approach you don't have data in the first request of the field (user, channel, etc.), instead a DEFERRED_FLAG is returned. After the task is run there's always a returned value, but it might have been calculated some time ago, depending on the expire time used in the @cache_stampede decorator
Using this approach you can't filter by the annotated fields

when to use this:

When using this approach the annotated fields are removed from the main queryset, making it much simpler, thus faster. It usually also removes group_by and order_by commands that add more searchs in the database
When using this approach, there is always (excepting the first request) an usable value in the redis cache that can be accessed easily
The implemented stampede method guaranties the value is requested in the expire interval while ensuring not all the requests are done at the same time, to distribute the database work
there is an api for the frontend to fetch easily the cached values
Tasks can be set to calculate the value for a group of items in one single task (to be shown in a table, for example), or to calculate one per task

… for users

codecov · 2020-10-02T18:07:44Z

Codecov Report

Merging #2359 into develop will decrease coverage by 0.23%.
The diff coverage is 28.37%.

@@ Coverage Diff @@## develop #2359 +/- ##
===========================================
- Coverage 80.57% 80.33% -0.24% 
===========================================
Files 299 300 +1 Lines 15168 15238 +70 ===========================================
+ Hits 12221 12241 +20 - Misses 2947 2997 +50

Impacted Files	Coverage Δ
...ontentcuration/contentcuration/viewsets/channel.py	`73.33% <0.00%> (ø)`
contentcuration/contentcuration/viewsets/user.py	`56.28% <25.58%> (-7.87%)`	⬇️
contentcuration/contentcuration/utils/user.py	`29.62% <29.62%> (ø)`
contentcuration/contentcuration/tasks.py	`72.72% <66.66%> (-0.16%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74754d6...723e3b8. Read the comment docs.

rtibbles · 2020-10-05T23:07:15Z

Could we add the PR comment to the dev docs and then merge?

jredrejo · 2020-10-02T17:47:49Z

Description

As done in #2333 , here we're caching and delaying to be asynchronously calculated some of the annotated fields for the AdminUserViewSet

Issue Addressed (if applicable)

Slow or impossible view with thousands of users in our Studio database

Implementation Notes (optional)

Added also an api for the frontend to fetch delayed data. Example of use:
http://localhost:8080/api/admin-users/deferred_data?id__in=1,2,3,90

One task to do all the users calculation is implemented (a single user task is not needed by now):
cache_multiple_users_metadata_task

At a high level, how did you implement this?

Values are stored in the cache and recalculated when they are requested if one hour has passed since its last calculation. To avoid cache stampede the expiration time is calculated with the @cache_stampede decorator that uses a randomized gaussian function so not all the calculations are done simultaneously

General view: Calculating annotated fields for the ViewSet in an asynchronous task

when (not) to use this:

Using this approach delays the calculation of the annotated fields. If they are required in real time, don't use it
Using this approach you don't have data in the first request of the field (user, channel, etc.), instead a DEFERRED_FLAG is returned. After the task is run there's always a returned value, but it might have been calculated some time ago, depending on the expire time used in the @cache_stampede decorator
Using this approach you can't filter by the annotated fields

when to use this:

When using this approach the annotated fields are removed from the main queryset, making it much simpler, thus faster. It usually also removes group_by and order_by commands that add more searchs in the database
When using this approach, there is always (excepting the first request) an usable value in the redis cache that can be accessed easily
The implemented stampede method guaranties the value is requested in the expire interval while ensuring not all the requests are done at the same time, to distribute the database work
there is an api for the frontend to fetch easily the cached values
Tasks can be set to calculate the value for a group of items in one single task (to be shown in a table, for example), or to calculate one per task

… for users

codecov · 2020-10-02T18:07:44Z

Codecov Report

Merging #2359 into develop will decrease coverage by 0.23%.
The diff coverage is 28.37%.

@@ Coverage Diff @@## develop #2359 +/- ##
===========================================
- Coverage 80.57% 80.33% -0.24% 
===========================================
Files 299 300 +1 Lines 15168 15238 +70 ===========================================
+ Hits 12221 12241 +20 - Misses 2947 2997 +50

Impacted Files	Coverage Δ
...ontentcuration/contentcuration/viewsets/channel.py	`73.33% <0.00%> (ø)`
contentcuration/contentcuration/viewsets/user.py	`56.28% <25.58%> (-7.87%)`	⬇️
contentcuration/contentcuration/utils/user.py	`29.62% <29.62%> (ø)`
contentcuration/contentcuration/tasks.py	`72.72% <66.66%> (-0.16%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74754d6...723e3b8. Read the comment docs.

rtibbles · 2020-10-05T23:07:15Z

Could we add the PR comment to the dev docs and then merge?

jredrejo · 2020-10-02T17:47:49Z

Description

As done in #2333 , here we're caching and delaying to be asynchronously calculated some of the annotated fields for the AdminUserViewSet

Issue Addressed (if applicable)

Slow or impossible view with thousands of users in our Studio database

Implementation Notes (optional)

Added also an api for the frontend to fetch delayed data. Example of use:
http://localhost:8080/api/admin-users/deferred_data?id__in=1,2,3,90

One task to do all the users calculation is implemented (a single user task is not needed by now):
cache_multiple_users_metadata_task

At a high level, how did you implement this?

Values are stored in the cache and recalculated when they are requested if one hour has passed since its last calculation. To avoid cache stampede the expiration time is calculated with the @cache_stampede decorator that uses a randomized gaussian function so not all the calculations are done simultaneously

General view: Calculating annotated fields for the ViewSet in an asynchronous task

when (not) to use this:

Using this approach delays the calculation of the annotated fields. If they are required in real time, don't use it
Using this approach you don't have data in the first request of the field (user, channel, etc.), instead a DEFERRED_FLAG is returned. After the task is run there's always a returned value, but it might have been calculated some time ago, depending on the expire time used in the @cache_stampede decorator
Using this approach you can't filter by the annotated fields

when to use this:

When using this approach the annotated fields are removed from the main queryset, making it much simpler, thus faster. It usually also removes group_by and order_by commands that add more searchs in the database
When using this approach, there is always (excepting the first request) an usable value in the redis cache that can be accessed easily
The implemented stampede method guaranties the value is requested in the expire interval while ensuring not all the requests are done at the same time, to distribute the database work
there is an api for the frontend to fetch easily the cached values
Tasks can be set to calculate the value for a group of items in one single task (to be shown in a table, for example), or to calculate one per task

… for users

codecov · 2020-10-02T18:07:44Z

Codecov Report

Merging #2359 into develop will decrease coverage by 0.23%.
The diff coverage is 28.37%.

@@ Coverage Diff @@## develop #2359 +/- ##
===========================================
- Coverage 80.57% 80.33% -0.24% 
===========================================
Files 299 300 +1 Lines 15168 15238 +70 ===========================================
+ Hits 12221 12241 +20 - Misses 2947 2997 +50

Impacted Files	Coverage Δ
...ontentcuration/contentcuration/viewsets/channel.py	`73.33% <0.00%> (ø)`
contentcuration/contentcuration/viewsets/user.py	`56.28% <25.58%> (-7.87%)`	⬇️
contentcuration/contentcuration/utils/user.py	`29.62% <29.62%> (ø)`
contentcuration/contentcuration/tasks.py	`72.72% <66.66%> (-0.16%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74754d6...723e3b8. Read the comment docs.

rtibbles · 2020-10-05T23:07:15Z

Could we add the PR comment to the dev docs and then merge?

jredrejo · 2020-10-02T17:47:49Z

Description

As done in #2333 , here we're caching and delaying to be asynchronously calculated some of the annotated fields for the AdminUserViewSet

Issue Addressed (if applicable)

Slow or impossible view with thousands of users in our Studio database

Implementation Notes (optional)

Added also an api for the frontend to fetch delayed data. Example of use:
http://localhost:8080/api/admin-users/deferred_data?id__in=1,2,3,90

One task to do all the users calculation is implemented (a single user task is not needed by now):
cache_multiple_users_metadata_task

At a high level, how did you implement this?

Values are stored in the cache and recalculated when they are requested if one hour has passed since its last calculation. To avoid cache stampede the expiration time is calculated with the @cache_stampede decorator that uses a randomized gaussian function so not all the calculations are done simultaneously

General view: Calculating annotated fields for the ViewSet in an asynchronous task

when (not) to use this:

Using this approach delays the calculation of the annotated fields. If they are required in real time, don't use it
Using this approach you don't have data in the first request of the field (user, channel, etc.), instead a DEFERRED_FLAG is returned. After the task is run there's always a returned value, but it might have been calculated some time ago, depending on the expire time used in the @cache_stampede decorator
Using this approach you can't filter by the annotated fields

when to use this:

When using this approach the annotated fields are removed from the main queryset, making it much simpler, thus faster. It usually also removes group_by and order_by commands that add more searchs in the database
When using this approach, there is always (excepting the first request) an usable value in the redis cache that can be accessed easily
The implemented stampede method guaranties the value is requested in the expire interval while ensuring not all the requests are done at the same time, to distribute the database work
there is an api for the frontend to fetch easily the cached values
Tasks can be set to calculate the value for a group of items in one single task (to be shown in a table, for example), or to calculate one per task

… for users

codecov · 2020-10-02T18:07:44Z

Codecov Report

Merging #2359 into develop will decrease coverage by 0.23%.
The diff coverage is 28.37%.

@@ Coverage Diff @@## develop #2359 +/- ##
===========================================
- Coverage 80.57% 80.33% -0.24% 
===========================================
Files 299 300 +1 Lines 15168 15238 +70 ===========================================
+ Hits 12221 12241 +20 - Misses 2947 2997 +50

Impacted Files	Coverage Δ
...ontentcuration/contentcuration/viewsets/channel.py	`73.33% <0.00%> (ø)`
contentcuration/contentcuration/viewsets/user.py	`56.28% <25.58%> (-7.87%)`	⬇️
contentcuration/contentcuration/utils/user.py	`29.62% <29.62%> (ø)`
contentcuration/contentcuration/tasks.py	`72.72% <66.66%> (-0.16%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74754d6...723e3b8. Read the comment docs.

rtibbles · 2020-10-05T23:07:15Z

Could we add the PR comment to the dev docs and then merge?

jredrejo · 2020-10-02T17:47:49Z

Description

As done in #2333 , here we're caching and delaying to be asynchronously calculated some of the annotated fields for the AdminUserViewSet

Issue Addressed (if applicable)

Slow or impossible view with thousands of users in our Studio database

Implementation Notes (optional)

Added also an api for the frontend to fetch delayed data. Example of use:
http://localhost:8080/api/admin-users/deferred_data?id__in=1,2,3,90

One task to do all the users calculation is implemented (a single user task is not needed by now):
cache_multiple_users_metadata_task

At a high level, how did you implement this?

Values are stored in the cache and recalculated when they are requested if one hour has passed since its last calculation. To avoid cache stampede the expiration time is calculated with the @cache_stampede decorator that uses a randomized gaussian function so not all the calculations are done simultaneously

General view: Calculating annotated fields for the ViewSet in an asynchronous task

when (not) to use this:

Using this approach delays the calculation of the annotated fields. If they are required in real time, don't use it
Using this approach you don't have data in the first request of the field (user, channel, etc.), instead a DEFERRED_FLAG is returned. After the task is run there's always a returned value, but it might have been calculated some time ago, depending on the expire time used in the @cache_stampede decorator
Using this approach you can't filter by the annotated fields

when to use this:

When using this approach the annotated fields are removed from the main queryset, making it much simpler, thus faster. It usually also removes group_by and order_by commands that add more searchs in the database
When using this approach, there is always (excepting the first request) an usable value in the redis cache that can be accessed easily
The implemented stampede method guaranties the value is requested in the expire interval while ensuring not all the requests are done at the same time, to distribute the database work
there is an api for the frontend to fetch easily the cached values
Tasks can be set to calculate the value for a group of items in one single task (to be shown in a table, for example), or to calculate one per task

… for users

codecov · 2020-10-02T18:07:44Z

Codecov Report

Merging #2359 into develop will decrease coverage by 0.23%.
The diff coverage is 28.37%.

@@ Coverage Diff @@## develop #2359 +/- ##
===========================================
- Coverage 80.57% 80.33% -0.24% 
===========================================
Files 299 300 +1 Lines 15168 15238 +70 ===========================================
+ Hits 12221 12241 +20 - Misses 2947 2997 +50

Impacted Files	Coverage Δ
...ontentcuration/contentcuration/viewsets/channel.py	`73.33% <0.00%> (ø)`
contentcuration/contentcuration/viewsets/user.py	`56.28% <25.58%> (-7.87%)`	⬇️
contentcuration/contentcuration/utils/user.py	`29.62% <29.62%> (ø)`
contentcuration/contentcuration/tasks.py	`72.72% <66.66%> (-0.16%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74754d6...723e3b8. Read the comment docs.

rtibbles · 2020-10-05T23:07:15Z

Could we add the PR comment to the dev docs and then merge?

jredrejo · 2020-10-02T17:47:49Z

Description

As done in #2333 , here we're caching and delaying to be asynchronously calculated some of the annotated fields for the AdminUserViewSet

Issue Addressed (if applicable)

Slow or impossible view with thousands of users in our Studio database

Implementation Notes (optional)

Added also an api for the frontend to fetch delayed data. Example of use:
http://localhost:8080/api/admin-users/deferred_data?id__in=1,2,3,90

One task to do all the users calculation is implemented (a single user task is not needed by now):
cache_multiple_users_metadata_task

At a high level, how did you implement this?

Values are stored in the cache and recalculated when they are requested if one hour has passed since its last calculation. To avoid cache stampede the expiration time is calculated with the @cache_stampede decorator that uses a randomized gaussian function so not all the calculations are done simultaneously

General view: Calculating annotated fields for the ViewSet in an asynchronous task

when (not) to use this:

Using this approach delays the calculation of the annotated fields. If they are required in real time, don't use it
Using this approach you don't have data in the first request of the field (user, channel, etc.), instead a DEFERRED_FLAG is returned. After the task is run there's always a returned value, but it might have been calculated some time ago, depending on the expire time used in the @cache_stampede decorator
Using this approach you can't filter by the annotated fields

when to use this:

When using this approach the annotated fields are removed from the main queryset, making it much simpler, thus faster. It usually also removes group_by and order_by commands that add more searchs in the database
When using this approach, there is always (excepting the first request) an usable value in the redis cache that can be accessed easily
The implemented stampede method guaranties the value is requested in the expire interval while ensuring not all the requests are done at the same time, to distribute the database work
there is an api for the frontend to fetch easily the cached values
Tasks can be set to calculate the value for a group of items in one single task (to be shown in a table, for example), or to calculate one per task

… for users

codecov · 2020-10-02T18:07:44Z

Codecov Report

Merging #2359 into develop will decrease coverage by 0.23%.
The diff coverage is 28.37%.

@@ Coverage Diff @@## develop #2359 +/- ##
===========================================
- Coverage 80.57% 80.33% -0.24% 
===========================================
Files 299 300 +1 Lines 15168 15238 +70 ===========================================
+ Hits 12221 12241 +20 - Misses 2947 2997 +50

Impacted Files	Coverage Δ
...ontentcuration/contentcuration/viewsets/channel.py	`73.33% <0.00%> (ø)`
contentcuration/contentcuration/viewsets/user.py	`56.28% <25.58%> (-7.87%)`	⬇️
contentcuration/contentcuration/utils/user.py	`29.62% <29.62%> (ø)`
contentcuration/contentcuration/tasks.py	`72.72% <66.66%> (-0.16%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 74754d6...723e3b8. Read the comment docs.

rtibbles · 2020-10-05T23:07:15Z

Could we add the PR comment to the dev docs and then merge?

Adding delayed calculation of count of editable and viewable channels…
723e3b8
… for users

jredrejo added the needs review label Oct 2, 2020

jredrejo requested a review from rtibbles October 2, 2020 17:47

rtibbles linked an issue Oct 2, 2020 that may be closed by this pull request
Optimize admin page #2202
Closed

rtibbles approved these changes Oct 5, 2020
View reviewed changes

jredrejo mentioned this pull request Oct 6, 2020
Admin page: __DEFERRED shows up while channel editor / viewer information is being determined #2396
Closed

jredrejo merged commit f66847c into learningequality:developOct 6, 2020

jredrejo deleted the delay_adminuser_annotations branch October 6, 2020 17:00

jredrejo mentioned this pull request Dec 1, 2020
annotate after paginate Admin channels and users #2607
Merged