[INF-76] New Local Deployment Setup with docker-compose by cheran-senthil · Pull Request #2988 · AudiusProject/apps

cheran-senthil · 2022-04-28T16:02:11Z

This PR removes the need for init-repos

Everything works solely through docker and the entire audius dev stack can be brought up with docker compose up

This is incompatible with service-commands and is a full refactor

Description

sudo and /etc/hosts modification no longer required
Makes Solana contract addresses deterministic
Block Explorer for Ganache instances
Reduces size of solana-programs and increases the build speed dramatically (13 GB to 2 GB)
Centralizes configuration to one file
Use pre-deployed images for Solana, ETH and POA -> this should bring down A up to < 1 minute
Add Discovery Provider and register it
Add Creator Node and register it
Creates wrapper script to imitate A

Follow Up PR

Rewrite all service-command commands and scripts in JavaScript for consistency
Collect and centralize scripts across audius

Tests

docker compose up
Health Checks on local

# Ubuntu 22.04 LTS | 12 vCPU / 24 GB RAM | 256 GB SSD
# initial setup
curl "https://raw.githubusercontent.com/AudiusProject/audius-protocol/master/dev-tools/setup.sh" | bash
# refresh terminal for docker
exit
# build and pull
audius-compose build # About 10 minutes
docker compose pull
audius-compose up # About 20 seconds

cheran-senthil · 2022-05-09T13:45:53Z

@hareeshnagaraj if you could take a look at the Solana changes, that would be great, thanks!

csjiang

Added some comments/qs! Looks great overall can't wait to switch to this new world :mochi_nani:

csjiang

nvm - ran this and doesn't work yet so making this a comment haha

csjiang

sorry, approval -> comment doesn't work so i'm doing approval -> request changes to remove the trigger-happy approval heh

raymondjacobson

This change looks good to me & rebasing+merging I think is the most sensible next steps. Client still has some work to get going off of this (proxy/registration stuff we chatted about), AAO and ES probably need implementation, and I think we should separate the containers some. I think we need to get through the long tail of those TODOs (and deleting the old A code) before we end up with two standards :)

raymondjacobson · 2022-07-12T19:16:21Z

+ETH_CONTRACTS_ABI_DIR = pathlib.Path(__file__).parent / "../build/eth-contracts"
+
+
+def health_check(discprov_url):


nit, preference for this to work based on status code alone. block diff should just be accounted for in health check long term imo

we may also want to not health check the node pre-registration per @hareeshnagaraj 's changes today.
content nodes need a discovery node to be online to be healthy
registering a content node will fail unless there are discovery nodes
discovery nodes will be unable to index the endpoints of content nodes if they never get registered
death

Identity times out the nodes that are not healthy though, I'm not sure how to get around this

Will add a follow up task

raymondjacobson · 2022-07-12T23:46:28Z

+mkdir -p "$HOME/.local/bin"
+
+ln -sf "$PROTOCOL_DIR/dev-tools/audius-compose" "$HOME/.local/bin/audius-compose"
+# ln -sf "$PROTOCOL_DIR/dev-tools/audius-cloud" "$HOME/.local/bin/audius-cloud"


remove maybe?

We should be adding audius-cloud soon, I think this is safe to keep?

raymondjacobson · 2022-07-12T23:48:57Z

 sudo -u postgres createdb audius_discovery
 else
- sudo -u postgres pg_ctl start -D /db
+ sudo -u postgres pg_ctl start -D /db -o "-c shared_buffers=512MB" -o "-c shared_preload_libraries=pg_stat_statements"


thanks, this is great! although long-term i still do believe in the db being similarly containerized being a helpful thing.
I know you were trying to save complexity in number of containers running, but maybe if we are better about port mgmt & naming of containers, we'll make life a lot easier
discovery_1_indexer
discovery_1_server
discovery_1_db
discovery_2_indexer
etc. etc.

Yes, this sounds good, I've made a follow up task for this, thanks!

raymondjacobson · 2022-07-12T23:50:10Z


-
 # start es-indexer
 if [[ "$audius_elasticsearch_url" ]] && [[ "$audius_elasticsearch_run_indexer" ]]; then


in this new world, i don't think there's ES implemented yet right? not sure what the es indexer will do.
also noticed in playing with client, we don't have AAO either.

The ES indexer should not start for now since audius_elasticsearch_url is not set
Will add ES and AAO in a follow up PR

## Changelog - #3463 by nicoback2 bump - #3420 by nicoback2 Use - #2988 by Cheran [INF-76] - #3458 by Johannes Naylor - #3456 by Sid Sethi - libs by Johannes Naylor - #3451 by Sebastian Klingler - #3373 by Sebastian Klingler - #3449 by Theo Ilie - #3408 by Theo Ilie - #3445 by Sebastian Klingler - #3435 by Reed Added - #3438 by Raymond Jacobson - #3434 by Hareesh Nagaraj - #3433 by Sebastian Klingler - #3406 by nicoback2 Make - #3402 by Hareesh Nagaraj - #3394 by Raymond Jacobson - #3392 by Raymond Jacobson - #3363 by Michael Piazza - #3366 by Joseph Lee - #3354 by Joseph Lee - #3362 by Michael Piazza - #3361 by Raymond Jacobson - #3360 by Raymond Jacobson - #3347 by Raymond Jacobson - #3345 by Raymond Jacobson - #3337 by Sebastian Klingler - #3336 by Sid Sethi

Deploy contracts via docker
2cbd49b

pull-request-sizeBot added the size/L label Apr 28, 2022

cheran-senthil added 6 commits May 5, 2022 20:32

Add identity
a814506

Merge branch 'master' into vss-new-local-deploy
543e5c7

Reduce solana-programs build size
167f2ca

Fix user replica set migration for audius dev
9cadb12

Speedup identity build
87e822f

Add docker-compose
4c05fc8

pull-request-sizeBot added size/XXL and removed size/L labels May 9, 2022

cheran-senthil mentioned this pull request May 9, 2022
Reduce docker image size #3041
Closed

cheran-senthil changed the title ~~Deploy contracts via docker~~New Local Deployment Setup with docker-composeMay 9, 2022

cheran-senthil added requires-special-attention This change is risky and/or critical and requires special attention in review service-commands Service Commands solana labels May 9, 2022

cheran-senthil requested a review from hareeshnagaraj May 9, 2022 13:40

cheran-senthil changed the title ~~New Local Deployment Setup with docker-compose~~[INF-76] New Local Deployment Setup with docker-composeMay 9, 2022

hareeshnagaraj reviewed May 9, 2022
View reviewed changes

Comment threaddocker-compose.yml Outdated

hareeshnagaraj reviewed May 9, 2022
View reviewed changes

Comment threadsolana-programs/Cargo.toml

Merge branch 'master' into vss-new-local-deploy
44cb721

cheran-senthil assigned jowlee and rickyrombo May 10, 2022

jowlee reviewed May 10, 2022
View reviewed changes

Comment threadcontracts/migrations/4_user_replica_set_migration.js Outdated
Comment threadcontracts/contract-config.js
Comment threadcontracts/contract-config.js
Comment threaddocker-compose.yml Outdated
Comment threadidentity-service/Dockerfile Outdated

csjiang approved these changes May 10, 2022
View reviewed changes

csjiang reviewed May 10, 2022
View reviewed changes

csjiang suggested changes May 10, 2022
View reviewed changes

cheran-senthil added 3 commits May 11, 2022 12:37

Add blockscout for POA
25f97d0

Create Dockerfile.dev for predeployed ganache
e227252

Similar predeployed ganache for eth contracts
c9ea09e

cheran-senthil added 3 commits July 11, 2022 23:38

Adjust postgres instead of sleep
861e7e0

Add quotes
e91fcc1

Some more fixes
f7e5d2d

raymondjacobson reviewed Jul 12, 2022
View reviewed changes

Comment threaddev/startup/proxy.sh Outdated

raymondjacobson mentioned this pull request Jul 12, 2022
[INF-180] Update client for new compose local dev AudiusProject/audius-client#1577
Merged

raymondjacobsonand others added 3 commits July 12, 2022 10:51

Fix typo (#3427)
da4842c

Add comments
d8c2d08

Add exec permissions globally
b4d79d0

cheran-senthil force-pushed the vss-new-local-deploy branch from 72183a0 to 3923bcbCompare July 12, 2022 17:34

Rename dev to dev-tools
fd9fabe

cheran-senthil force-pushed the vss-new-local-deploy branch 2 times, most recently from d70658a to fd9fabeCompare July 12, 2022 18:18

cheran-senthil added 3 commits July 12, 2022 14:26

Merge branch 'master' into vss-new-local-deploy
3d97361

Revert changes to build
b4228c1

Merge branch 'master' into vss-new-local-deploy
c5c9d26

raymondjacobson approved these changes Jul 12, 2022
View reviewed changes

Merge branch 'master' into vss-new-local-deploy
5c23246

cheran-senthil force-pushed the vss-new-local-deploy branch 3 times, most recently from 7bddebc to b9dbca8Compare July 14, 2022 16:02

Fix broken config
46781f0

cheran-senthil force-pushed the vss-new-local-deploy branch from b9dbca8 to 46781f0Compare July 14, 2022 16:06

Update resource class
0d729c7

cheran-senthil merged commit ba91046 into masterJul 14, 2022

cheran-senthil deleted the vss-new-local-deploy branch July 14, 2022 16:26

nicoback2 mentioned this pull request Jul 14, 2022
Bump SDK to version 0.0.16 #3463
Merged

AudiusProject deleted a comment from linearBotSep 11, 2023

		ETH_CONTRACTS_ABI_DIR = pathlib.Path(__file__).parent / "../build/eth-contracts"


		def health_check(discprov_url):



		# start es-indexer
		if [[ "$audius_elasticsearch_url" ]] && [[ "$audius_elasticsearch_run_indexer" ]]; then

Uh oh!

Conversation

cheran-senthil commented Apr 28, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Follow Up PR

Tests

Uh oh!

cheran-senthil commented May 9, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

csjiang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

csjiang left a comment

Choose a reason for hiding this comment

Uh oh!

csjiang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

raymondjacobson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

cheran-senthil commented Apr 28, 2022•
edited
Loading