Add new job to save token insights to the database by rickyrombo · Pull Request #352 · AudiusProject/api

rickyrombo · 2025-09-03T02:28:01Z

Adds new jobs package to hold "jobs"
Adds new artist_coin_stats table to hold token insights
Creates new job for saving token insights
Runs the artist coin stats job in Solana Indexer

Also:

Moves the Birdeye client to its own package
Moves scheduling of retries to own function in the solana indexer
Replaces With(zap.String("service", "SolanaIndexer") with Named("SolanaIndexer")
Makes previous migrations idempotent
Moves DbPool interface to database package to be shared

rickyrombo · 2025-09-03T02:29:26Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


Config for this interval? TBH not sure if there's a benefit. Deploying a new config is much like deploying a code change 🤷

I 100000% agree with this

In late to say at the very least these should be named constants defined in a centralized config object instead of inline in a function. But that's a nitpick.

I did say nitpick, but I think a config object (a constant, not env variables) is easier for somebody who didn't write the code to come in and adjust the value if needed than having to dig through to find where it's passed. Just a stylistic preference.

I think if the initialization of jobs are all short one liners and listed together that's probably good enough. But if there initialized all over the place or are more complicated to init it may get harder to grok

raymondjacobson

nice looks great!

raymondjacobson · 2025-09-03T03:47:59Z

@@ -1,2 +1,2 @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT '';
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT '';


raymondjacobson · 2025-09-03T03:49:09Z

+	}
+
+	for offset := 0; offset < count; offset += tokenPageSize {
+ batch, err := j.getTokenBatch(ctx, tokenPageSize, offset)


I think batch is kinda overkill, could just paginate 1 by 1 and fetch i think. but this works

raymondjacobson · 2025-09-03T03:49:56Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


I 100000% agree with this

rickyrombo · 2025-09-03T02:28:01Z

Adds new jobs package to hold "jobs"
Adds new artist_coin_stats table to hold token insights
Creates new job for saving token insights
Runs the artist coin stats job in Solana Indexer

Also:

Moves the Birdeye client to its own package
Moves scheduling of retries to own function in the solana indexer
Replaces With(zap.String("service", "SolanaIndexer") with Named("SolanaIndexer")
Makes previous migrations idempotent
Moves DbPool interface to database package to be shared

rickyrombo · 2025-09-03T02:29:26Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


Config for this interval? TBH not sure if there's a benefit. Deploying a new config is much like deploying a code change 🤷

I 100000% agree with this

In late to say at the very least these should be named constants defined in a centralized config object instead of inline in a function. But that's a nitpick.

I did say nitpick, but I think a config object (a constant, not env variables) is easier for somebody who didn't write the code to come in and adjust the value if needed than having to dig through to find where it's passed. Just a stylistic preference.

I think if the initialization of jobs are all short one liners and listed together that's probably good enough. But if there initialized all over the place or are more complicated to init it may get harder to grok

raymondjacobson

nice looks great!

raymondjacobson · 2025-09-03T03:47:59Z

@@ -1,2 +1,2 @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT '';
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT '';


raymondjacobson · 2025-09-03T03:49:09Z

+	}
+
+	for offset := 0; offset < count; offset += tokenPageSize {
+ batch, err := j.getTokenBatch(ctx, tokenPageSize, offset)


I think batch is kinda overkill, could just paginate 1 by 1 and fetch i think. but this works

raymondjacobson · 2025-09-03T03:49:56Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


I 100000% agree with this

rickyrombo · 2025-09-03T02:28:01Z

Adds new jobs package to hold "jobs"
Adds new artist_coin_stats table to hold token insights
Creates new job for saving token insights
Runs the artist coin stats job in Solana Indexer

Also:

Moves the Birdeye client to its own package
Moves scheduling of retries to own function in the solana indexer
Replaces With(zap.String("service", "SolanaIndexer") with Named("SolanaIndexer")
Makes previous migrations idempotent
Moves DbPool interface to database package to be shared

rickyrombo · 2025-09-03T02:29:26Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


Config for this interval? TBH not sure if there's a benefit. Deploying a new config is much like deploying a code change 🤷

I 100000% agree with this

In late to say at the very least these should be named constants defined in a centralized config object instead of inline in a function. But that's a nitpick.

I did say nitpick, but I think a config object (a constant, not env variables) is easier for somebody who didn't write the code to come in and adjust the value if needed than having to dig through to find where it's passed. Just a stylistic preference.

I think if the initialization of jobs are all short one liners and listed together that's probably good enough. But if there initialized all over the place or are more complicated to init it may get harder to grok

raymondjacobson

nice looks great!

raymondjacobson · 2025-09-03T03:47:59Z

@@ -1,2 +1,2 @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT '';
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT '';


raymondjacobson · 2025-09-03T03:49:09Z

+	}
+
+	for offset := 0; offset < count; offset += tokenPageSize {
+ batch, err := j.getTokenBatch(ctx, tokenPageSize, offset)


I think batch is kinda overkill, could just paginate 1 by 1 and fetch i think. but this works

raymondjacobson · 2025-09-03T03:49:56Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


I 100000% agree with this

rickyrombo · 2025-09-03T02:28:01Z

Adds new jobs package to hold "jobs"
Adds new artist_coin_stats table to hold token insights
Creates new job for saving token insights
Runs the artist coin stats job in Solana Indexer

Also:

Moves the Birdeye client to its own package
Moves scheduling of retries to own function in the solana indexer
Replaces With(zap.String("service", "SolanaIndexer") with Named("SolanaIndexer")
Makes previous migrations idempotent
Moves DbPool interface to database package to be shared

rickyrombo · 2025-09-03T02:29:26Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


Config for this interval? TBH not sure if there's a benefit. Deploying a new config is much like deploying a code change 🤷

I 100000% agree with this

In late to say at the very least these should be named constants defined in a centralized config object instead of inline in a function. But that's a nitpick.

I did say nitpick, but I think a config object (a constant, not env variables) is easier for somebody who didn't write the code to come in and adjust the value if needed than having to dig through to find where it's passed. Just a stylistic preference.

I think if the initialization of jobs are all short one liners and listed together that's probably good enough. But if there initialized all over the place or are more complicated to init it may get harder to grok

raymondjacobson

nice looks great!

raymondjacobson · 2025-09-03T03:47:59Z

@@ -1,2 +1,2 @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT '';
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT '';


raymondjacobson · 2025-09-03T03:49:09Z

+	}
+
+	for offset := 0; offset < count; offset += tokenPageSize {
+ batch, err := j.getTokenBatch(ctx, tokenPageSize, offset)


I think batch is kinda overkill, could just paginate 1 by 1 and fetch i think. but this works

raymondjacobson · 2025-09-03T03:49:56Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


I 100000% agree with this

rickyrombo · 2025-09-03T02:28:01Z

Adds new jobs package to hold "jobs"
Adds new artist_coin_stats table to hold token insights
Creates new job for saving token insights
Runs the artist coin stats job in Solana Indexer

Also:

Moves the Birdeye client to its own package
Moves scheduling of retries to own function in the solana indexer
Replaces With(zap.String("service", "SolanaIndexer") with Named("SolanaIndexer")
Makes previous migrations idempotent
Moves DbPool interface to database package to be shared

rickyrombo · 2025-09-03T02:29:26Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


Config for this interval? TBH not sure if there's a benefit. Deploying a new config is much like deploying a code change 🤷

I 100000% agree with this

In late to say at the very least these should be named constants defined in a centralized config object instead of inline in a function. But that's a nitpick.

I did say nitpick, but I think a config object (a constant, not env variables) is easier for somebody who didn't write the code to come in and adjust the value if needed than having to dig through to find where it's passed. Just a stylistic preference.

I think if the initialization of jobs are all short one liners and listed together that's probably good enough. But if there initialized all over the place or are more complicated to init it may get harder to grok

raymondjacobson

nice looks great!

raymondjacobson · 2025-09-03T03:47:59Z

@@ -1,2 +1,2 @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT '';
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT '';


raymondjacobson · 2025-09-03T03:49:09Z

+	}
+
+	for offset := 0; offset < count; offset += tokenPageSize {
+ batch, err := j.getTokenBatch(ctx, tokenPageSize, offset)


I think batch is kinda overkill, could just paginate 1 by 1 and fetch i think. but this works

raymondjacobson · 2025-09-03T03:49:56Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


I 100000% agree with this

rickyrombo · 2025-09-03T02:28:01Z

Adds new jobs package to hold "jobs"
Adds new artist_coin_stats table to hold token insights
Creates new job for saving token insights
Runs the artist coin stats job in Solana Indexer

Also:

Moves the Birdeye client to its own package
Moves scheduling of retries to own function in the solana indexer
Replaces With(zap.String("service", "SolanaIndexer") with Named("SolanaIndexer")
Makes previous migrations idempotent
Moves DbPool interface to database package to be shared

rickyrombo · 2025-09-03T02:29:26Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


Config for this interval? TBH not sure if there's a benefit. Deploying a new config is much like deploying a code change 🤷

I 100000% agree with this

In late to say at the very least these should be named constants defined in a centralized config object instead of inline in a function. But that's a nitpick.

I did say nitpick, but I think a config object (a constant, not env variables) is easier for somebody who didn't write the code to come in and adjust the value if needed than having to dig through to find where it's passed. Just a stylistic preference.

I think if the initialization of jobs are all short one liners and listed together that's probably good enough. But if there initialized all over the place or are more complicated to init it may get harder to grok

raymondjacobson

nice looks great!

raymondjacobson · 2025-09-03T03:47:59Z

@@ -1,2 +1,2 @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT '';
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT '';


raymondjacobson · 2025-09-03T03:49:09Z

+	}
+
+	for offset := 0; offset < count; offset += tokenPageSize {
+ batch, err := j.getTokenBatch(ctx, tokenPageSize, offset)


I think batch is kinda overkill, could just paginate 1 by 1 and fetch i think. but this works

raymondjacobson · 2025-09-03T03:49:56Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


I 100000% agree with this

rickyrombo · 2025-09-03T02:28:01Z

Adds new jobs package to hold "jobs"
Adds new artist_coin_stats table to hold token insights
Creates new job for saving token insights
Runs the artist coin stats job in Solana Indexer

Also:

Moves the Birdeye client to its own package
Moves scheduling of retries to own function in the solana indexer
Replaces With(zap.String("service", "SolanaIndexer") with Named("SolanaIndexer")
Makes previous migrations idempotent
Moves DbPool interface to database package to be shared

rickyrombo · 2025-09-03T02:29:26Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


Config for this interval? TBH not sure if there's a benefit. Deploying a new config is much like deploying a code change 🤷

I 100000% agree with this

In late to say at the very least these should be named constants defined in a centralized config object instead of inline in a function. But that's a nitpick.

I did say nitpick, but I think a config object (a constant, not env variables) is easier for somebody who didn't write the code to come in and adjust the value if needed than having to dig through to find where it's passed. Just a stylistic preference.

I think if the initialization of jobs are all short one liners and listed together that's probably good enough. But if there initialized all over the place or are more complicated to init it may get harder to grok

raymondjacobson

nice looks great!

raymondjacobson · 2025-09-03T03:47:59Z

@@ -1,2 +1,2 @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT '';
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT '';


raymondjacobson · 2025-09-03T03:49:09Z

+	}
+
+	for offset := 0; offset < count; offset += tokenPageSize {
+ batch, err := j.getTokenBatch(ctx, tokenPageSize, offset)


I think batch is kinda overkill, could just paginate 1 by 1 and fetch i think. but this works

raymondjacobson · 2025-09-03T03:49:56Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


I 100000% agree with this

rickyrombo · 2025-09-03T02:28:01Z

Adds new jobs package to hold "jobs"
Adds new artist_coin_stats table to hold token insights
Creates new job for saving token insights
Runs the artist coin stats job in Solana Indexer

Also:

Moves the Birdeye client to its own package
Moves scheduling of retries to own function in the solana indexer
Replaces With(zap.String("service", "SolanaIndexer") with Named("SolanaIndexer")
Makes previous migrations idempotent
Moves DbPool interface to database package to be shared

rickyrombo · 2025-09-03T02:29:26Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


Config for this interval? TBH not sure if there's a benefit. Deploying a new config is much like deploying a code change 🤷

I 100000% agree with this

In late to say at the very least these should be named constants defined in a centralized config object instead of inline in a function. But that's a nitpick.

I did say nitpick, but I think a config object (a constant, not env variables) is easier for somebody who didn't write the code to come in and adjust the value if needed than having to dig through to find where it's passed. Just a stylistic preference.

I think if the initialization of jobs are all short one liners and listed together that's probably good enough. But if there initialized all over the place or are more complicated to init it may get harder to grok

raymondjacobson

nice looks great!

raymondjacobson · 2025-09-03T03:47:59Z

@@ -1,2 +1,2 @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT '';
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT '';


raymondjacobson · 2025-09-03T03:49:09Z

+	}
+
+	for offset := 0; offset < count; offset += tokenPageSize {
+ batch, err := j.getTokenBatch(ctx, tokenPageSize, offset)


I think batch is kinda overkill, could just paginate 1 by 1 and fetch i think. but this works

raymondjacobson · 2025-09-03T03:49:56Z

+	go s.ScheduleRetries(ctx, s.config.SolanaIndexerRetryInterval)
+
+	go jobs.NewCoinStatsJob(s.config, s.pool).
+ ScheduleEvery(ctx, 5*time.Minute)


I 100000% agree with this

Add new job to save token insights to the database
1d2db97

rickyrombo requested review from raymondjacobson and schottra September 3, 2025 02:28

rickyrombo commented Sep 3, 2025
View reviewed changes

raymondjacobson approved these changes Sep 3, 2025
View reviewed changes

rickyrombo merged commit 5f1d684 into mjp-dbc-pricesSep 3, 2025
4 checks passed

rickyrombo deleted the mjp-scheduled-insights branch September 3, 2025 16:40

Add new job to save token insights to the database
1d2db97

rickyrombo requested review from raymondjacobson and schottra September 3, 2025 02:28

rickyrombo commented Sep 3, 2025
View reviewed changes

raymondjacobson approved these changes Sep 3, 2025
View reviewed changes

rickyrombo merged commit 5f1d684 into mjp-dbc-pricesSep 3, 2025
4 checks passed

rickyrombo deleted the mjp-scheduled-insights branch September 3, 2025 16:40

Add new job to save token insights to the database
1d2db97

rickyrombo requested review from raymondjacobson and schottra September 3, 2025 02:28

rickyrombo commented Sep 3, 2025
View reviewed changes

raymondjacobson approved these changes Sep 3, 2025
View reviewed changes

rickyrombo merged commit 5f1d684 into mjp-dbc-pricesSep 3, 2025
4 checks passed

rickyrombo deleted the mjp-scheduled-insights branch September 3, 2025 16:40

Add new job to save token insights to the database
1d2db97

rickyrombo requested review from raymondjacobson and schottra September 3, 2025 02:28

rickyrombo commented Sep 3, 2025
View reviewed changes

raymondjacobson approved these changes Sep 3, 2025
View reviewed changes

rickyrombo merged commit 5f1d684 into mjp-dbc-pricesSep 3, 2025
4 checks passed

rickyrombo deleted the mjp-scheduled-insights branch September 3, 2025 16:40

Add new job to save token insights to the database
1d2db97

rickyrombo requested review from raymondjacobson and schottra September 3, 2025 02:28

rickyrombo commented Sep 3, 2025
View reviewed changes

raymondjacobson approved these changes Sep 3, 2025
View reviewed changes

rickyrombo merged commit 5f1d684 into mjp-dbc-pricesSep 3, 2025
4 checks passed

rickyrombo deleted the mjp-scheduled-insights branch September 3, 2025 16:40

Add new job to save token insights to the database
1d2db97

rickyrombo requested review from raymondjacobson and schottra September 3, 2025 02:28

rickyrombo commented Sep 3, 2025
View reviewed changes

raymondjacobson approved these changes Sep 3, 2025
View reviewed changes

rickyrombo merged commit 5f1d684 into mjp-dbc-pricesSep 3, 2025
4 checks passed

rickyrombo deleted the mjp-scheduled-insights branch September 3, 2025 16:40

Add new job to save token insights to the database
1d2db97

rickyrombo requested review from raymondjacobson and schottra September 3, 2025 02:28

rickyrombo commented Sep 3, 2025
View reviewed changes

raymondjacobson approved these changes Sep 3, 2025
View reviewed changes

rickyrombo merged commit 5f1d684 into mjp-dbc-pricesSep 3, 2025
4 checks passed

rickyrombo deleted the mjp-scheduled-insights branch September 3, 2025 16:40

Add new job to save token insights to the database
1d2db97

rickyrombo requested review from raymondjacobson and schottra September 3, 2025 02:28

rickyrombo commented Sep 3, 2025
View reviewed changes

raymondjacobson approved these changes Sep 3, 2025
View reviewed changes

rickyrombo merged commit 5f1d684 into mjp-dbc-pricesSep 3, 2025
4 checks passed

rickyrombo deleted the mjp-scheduled-insights branch September 3, 2025 16:40

@@ @@ -1,2 +1,2 @@ @@
 ALTER TABLE artist_coins
- ADD COLUMN name TEXT NOT NULL DEFAULT ''; No newline at end of file
+ ADD COLUMN IF NOT EXISTS name TEXT NOT NULL DEFAULT ''; No newline at end of file

Uh oh!

Conversation

rickyrombo commented Sep 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raymondjacobson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rickyrombo commented Sep 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raymondjacobson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

rickyrombo commented Sep 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!