Add created_at indexes for faster get_feed.

[stereosteve]

Also convert some existing indexes to partial indexes.

Description

get_feed does a lot of order by created_at desc limit 20 type of queries, but there's no index on created_at which results in a materialization and sort of the full intermediate result set.

Before:

explain analyze select * from tracks where is_current and not is_delete and owner_id in (
select followee_user_id as user_id from follows
where is_current and not is_delete and follower_user_id = 1
) order by created_at desc limit 100;
QUERY PLAN -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=48071.94..48072.19 rows=100 width=1213) (actual time=8822.507..8822.544 rows=100 loops=1)
-> Sort (cost=48071.94..48117.88 rows=18375 width=1213) (actual time=8822.505..8822.527 rows=100 loops=1)
Sort Key: tracks.created_at DESC
Sort Method: top-N heapsort Memory: 242kB
-> Nested Loop (cost=961.78..47369.66 rows=18375 width=1213) (actual time=4320.045..8645.083 rows=25348 loops=1)
-> HashAggregate (cost=961.35..966.19 rows=484 width=4) (actual time=4307.701..4311.156 rows=1207 loops=1)
Group Key: follows.followee_user_id
-> Index Scan using ix_follows_follower_user_id on follows (cost=0.43..960.13 rows=489 width=4) (actual time=6.045..4183.419 rows=1207 loops=1)
Index Cond: (follower_user_id = 1)
Filter: (is_current AND (NOT is_delete))
Rows Removed by Filter: 181
-> Index Scan using track_owner_id_idx on tracks (cost=0.43..95.55 rows=32 width=1213) (actual time=3.193..3.567 rows=21 loops=1207)
Index Cond: (owner_id = follows.followee_user_id)
Filter: (is_current AND (NOT is_delete))
Rows Removed by Filter: 22
Planning Time: 242.960 ms
Execution Time: 9030.741 ms

explain analyze select * from tracks where is_current and not is_delete and owner_id in (
select followee_user_id as user_id from follows
where is_current and not is_delete and follower_user_id = 1
) order by created_at desc limit 100;
QUERY PLAN -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Limit (cost=1001.13..9426.05 rows=100 width=1186) (actual time=28.008..143.348 rows=100 loops=1)
-> Gather Merge (cost=1001.13..1521108.40 rows=18043 width=1186) (actual time=28.006..143.322 rows=100 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Nested Loop Semi Join (cost=1.11..1518025.76 rows=7518 width=1186) (actual time=20.752..123.292 rows=36 loops=3)
-> Parallel Index Scan Backward using track_created_at_idx on tracks (cost=0.42..158319.35 rows=299819 width=1186) (actual time=7.994..58.432 rows=15063 loops=3)
-> Index Scan using follows_pkey on follows (cost=0.69..4.53 rows=1 width=4) (actual time=0.004..0.004 rows=0 loops=45189)
Index Cond: ((is_current = true) AND (follower_user_id = 1) AND (followee_user_id = tracks.owner_id))
Filter: (is_current AND (NOT is_delete))
Rows Removed by Filter: 0
Planning Time: 5.835 ms
Execution Time: 143.472 ms

Tests

Stand by for some more concrete before and after numbers.

How will this change be monitored? Are there sufficient logs?

GCP SQL panel

[raymondjacobson]

Maybe worth separating the PRs into the ones that are additive and the ones that are mutations

[raymondjacobson]

DROP INDEX IF EXISTS makes sense to me just as a guard rail, but otherwise, PR looks good to me!

[stereosteve]

After looking more at https://github.com/AudiusProject/audius-protocol/blob/master/discovery-provider/src/tasks/index_aggregate_user.py#L20

I agree with Ray's suggestion from Slack about not omitting is_delete rows from index.

We could maybe consider a compound index like (owner_id, is_delete) if we still want to have index only scans for some queries... but I will save that for another day