User Service

CircleCI insights

dev

The User service block enables secure and highly extensible user management and organization management capabilities - User registration, Live and JWT basesd session management, Delete user, Forgot password with email notifications.

System Requirements

Operating System: Ubuntu 22
Node.js: v20
PostgreSQL: 16
Citus: 12.1
Apache Kafka: 3.5.0

Setup Options

Elevate user services can be set in local using two methods:

Dockerized service with local dependencies(Intermediate)

Expectation: Run single docker containerized service with existing local (in host) or remote dependencies.

Local Dependencies Steps

Update dependency (Mongo v4.1.4, Kafka etc) IP addresses in .env with "host.docker.internal".
Eg:
```
 #Kafka Host Server URL
KAFKA_URL = host.docker.external:9092
```
Find host.docker.internal IP address and added it to mongod.conf file in host.
Eg: If host.docker.internal is 172.17.0.1, mongod.conf:
```
# network interfaces
net:
port: 27017
bindIp: "127.0.0.1,172.17.0.1"
```
Note: Steps to find host.docker.internal IP address & location of mongod.conf is operating system specific. Refer this for more information.

Build the docker image.

/ELEVATE/user$ docker build -t elevate/user:1.0 .

Run the docker container.

For Mac & Windows with docker v18.03+:

$ docker run --name user elevate/user:1.0

For Linux:

$ docker run --name user --add-host=host.docker.internal:host-gateway elevate/user:1.0`

Refer this for more information.

Remote Dependencies Steps

Update dependency (Mongo v4.1.4, Kafka etc) Ip addresses in .env with respective remote server IPs.

Eg:

 #DB Connectivity Url
DATABASE_URL=postgres://postgres:postgres@localhost:5432/elevate-user
#Kafka Host Server URL
KAFKA_URL = 11.2.3.45:9092

Add Bind IP to mongod.conf in host:
Follow instructions given here.
Note: Instructions might differ based on MongoDB version and operating system.

Build the docker image.

/ELEVATE/user$ docker build -t elevate/user:1.0 .

Run the docker container.

$ docker run --name user elevate/user:1.0

Local Service with local dependencies(Hardest)

Expectation: Run single service with existing local dependencies in host (Non-Docker Implementation).

Installations

Install Node.js LTS

Refer to the NodeSource distributions installation scripts for Node.js installation.

$ curl -fsSL https://deb.nodesource.com/setup_lts.x | sudo -E bash - &&\
sudo apt-get install -y nodejs

Install Build Essential

$ sudo apt-get install build-essential

Install Kafka

Refer to Kafka Ubuntu 22.04 setup guide

Install OpenJDK 11:
```
$ sudo apt install openjdk-11-jdk
```

Download and extract Kafka:

$ sudo wget https://downloads.apache.org/kafka/3.5.0/kafka_2.12-3.5.0.tgz
$ sudo tar xzf kafka_2.12-3.5.0.tgz
$ sudo mv kafka_2.12-3.5.0 /opt/kafka

Configure Zookeeper:

$ sudo nano /etc/systemd/system/zookeeper.service

Paste the following lines into the zookeeper.service file:

/etc/systemd/system/zookeeper.service
[Unit]Description=Apache Zookeeper service
Documentation=http://zookeeper.apache.org
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]Type=simple
ExecStart=/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop=/opt/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]WantedBy=multi-user.target

Save and exit.

Reload systemd:
```
$ sudo systemctl daemon-reload
```

Configure Kafka:

$ sudo nano /etc/systemd/system/kafka.service

Paste the following lines into the kafka.service file:

[Unit]Description=Apache Kafka Service
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
[Service]Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"ExecStart=/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties
ExecStop=/opt/kafka/bin/kafka-server-stop.sh
[Install]WantedBy=multi-user.target

Save and exit.

Reload systemd:
```
$ sudo systemctl daemon-reload
```
Start Zookeeper:
```
$ sudo systemctl start zookeeper
```
Check status:
```
$ sudo systemctl status zookeeper
```
Zookeeper service status should be shown as active (running).
Start Kafka:
```
$ sudo systemctl start kafka
```
Check status:
```
$ sudo systemctl status kafka
```
Kafka status should be shown as active (running).

Install Redis

Refer to Redis Ubuntu 22.04 setup guide

Update the package list:
```
$ sudo apt update
```
Install Redis:
```
$ sudo apt install redis-server
```

Configure Redis for systemd:

$ sudo nano /etc/redis/redis.conf

Find the supervised directive and change it to "systemd" as follows:

. . .
# If you run Redis from upstart or systemd, Redis can interact with your# supervision tree. Options:# supervised no - no supervision interaction# supervised upstart - signal upstart by putting Redis into SIGSTOP mode# supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET# supervised auto - detect upstart or systemd method based on# UPSTART_JOB or NOTIFY_SOCKET environment variables# Note: these supervision methods only signal "process is ready."# They do not enable continuous liveness pings back to your supervisor.
supervised systemd
. . .

Save and exit.

Restart the Redis service:
```
$ sudo systemctl restart redis.service
```

Install Single-Node Citus (Distributed Database)

Refer to official Citus single-node setup

Download and install Citus:

$ curl https://install.citusdata.com/community/deb.sh | sudo bash
$ sudo apt-get -y install postgresql-16-citus-12.1

Switch to the PostgreSQL user:
```
$ sudo su - postgres
```
Set the PostgreSQL bin directory in the PATH and create a directory for Citus:
```
$ export PATH=$PATH:/usr/lib/postgresql/16/bin
$ cd~
$ mkdir citus
```
Initialize the Citus database:
```
$ initdb -D citus
```

Configure Citus in citus/postgresql.conf:

$ echo"shared_preload_libraries = 'citus'">> citus/postgresql.conf

Start the Citus server:

$ pg_ctl -D citus -o "-p 9700" -l citus_logfile start

Create the Citus extension:

$ psql -p 9700 -c "CREATE EXTENSION citus;"

Check the Citus version:

$ psql -p 9700 -c "select citus_version();"

You should see an output similar to the following, indicating that Citus is successfully installed:

postgres=# select citus_version();
citus_version
----------------------------------------------------------------------------------------------------
Citus 12.1.1on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, 64-bit
(1 row)

Install PM2

Refer to How To Set Up a Node.js Application for Production on Ubuntu 22.04.

Exit the postgres user account and run the following command

$ sudo npm install pm2@latest -g

Setting up Repository

Clone the user repository to /opt/backend directory

opt/backend$ git clone -b develop-2.5 --single-branch "https://github.com/ELEVATE-Project/user.git"

Install Npm packages from src directory

backend/user/src$ sudo npm i

Create .env file in src directory

user/src$ sudo nano .env

Copy-paste the following env variables to the .env file:

NB : Make sure to update the credentials according to your configurations.

ACCESS_TOKEN_EXPIRY=10ACCESS_TOKEN_SECRET=asadsd8as7df9as8df987asdfADMIN_INVITEE_UPLOAD_EMAIL_TEMPLATE_CODE=invitee_upload_statusADMIN_SECRET_CODE=Na7ad23ws5cm3kfmw24dmdsflaksdAPI_DOC_URL=/user/api-docAPPLICATION_ENV=developmentAPPLICATION_PORT=3001APP_NAME=MentorEDAWS_ACCESS_KEY_ID="adsfg98a7sdfg"AWS_BUCKET_ENDPOINT="s3.ap-south-1.amazonaws.com"AWS_BUCKET_REGION="ap-south-1"AWS_SECRET_ACCESS_KEY="asd9786fg9a8sd/asdfg9a8sd7fg"AZURE_ACCOUNT_KEY=asd897gfa09sd87f09as8dAZURE_ACCOUNT_NAME=mentoringCLEAR_INTERNAL_CACHE=userinternalCLOUD_STORAGE=GCPDEFAULT_AWS_BUCKET_NAME=mentoring-dev-storageDEFAULT_AZURE_CONTAINER_NAME=mentoring-imagesDEFAULT_GCP_BUCKET_NAME=mentoring-dev-storageDEFAULT_ORGANISATION_CODE=default_codeDEFAULT_ORG_ID=1DEFAULT_QUEUE=user-queueDEFAULT_ROLE=menteeDEV_DATABASE_URL=postgres://shikshalokam:slpassword123@localhost:9700/elevate_userDISABLE_LOG=falseEMAIL_ID_ENCRYPTION_ALGORITHM=aes-256-cbcEMAIL_ID_ENCRYPTION_IV=a19f1ewaqwei9e03edkc32eEMAIL_ID_ENCRYPTION_KEY=9bszawjkckw2e3dm35fcw27ws4ed5rftg6y6y7y7654tf4rwq5tr0ol2qa9owsieENABLE_EMAIL_OTP_VERIFICATION=trueENABLE_LOG=trueERROR_LOG_LEVEL=sillyEVENT_ENABLE_ORG_EVENTS=trueEVENT_ORG_LISTENER_URLS=http://localhost:3567/mentoring/v1/organization/eventListenerGCP_PATH=gcp.jsonGCP_PROJECT_ID=sl-dev-projectGENERIC_INVITATION_EMAIL_TEMPLATE_CODE=generic_inviteINTERNAL_ACCESS_TOKEN=FqdkfaswekdlweINTERNAL_CACHE_EXP_TIME=86400INVITEE_EMAIL_TEMPLATE_CODE=invite_userIV=LKYTTAqkajswiawqw/Z==KAFKA_GROUP_ID=dev.usersKAFKA_TOPIC=dev.topicKAFKA_URL=localhost:9092KEY=W/m2cr/aMswjrdsa23sgfy5e34d+bKcbAWZSLjJP2qY=MENTEE_INVITATION_EMAIL_TEMPLATE_CODE=invite_menteeMENTORING_SERVICE_URL=http://localhost:3000MENTOR_INVITATION_EMAIL_TEMPLATE_CODE=invite_mentorMENTOR_REQUEST_ACCEPTED_EMAIL_TEMPLATE_CODE=mentor_request_acceptedMENTOR_REQUEST_REJECTED_EMAIL_TEMPLATE_CODE=mentor_request_rejectedMENTOR_SECRET_CODE=4567NOTIFICATION_KAFKA_TOPIC=dev.notificationORG_ADMIN_INVITATION_EMAIL_TEMPLATE_CODE=invite_org_adminOTP_EMAIL_TEMPLATE_CODE=emailotpOTP_EXP_TIME=86400PORTAL_URL="https://dev.elevate-mentoring.shikshalokam.org/auth/login"RATING_KAFKA_TOPIC=dev.mentor_ratingREDIS_HOST=redis://localhost:6379REFRESH_TOKEN_EXPIRY=183REFRESH_TOKEN_SECRET=371hkjadidy2ashiKAkajshdkid23iuekw71yekiaskdvkvegxvy23t78veQwexqviveit6ttZyeeytx62tx236uvREFRESH_VIEW_INTERVAL=30000REGISTRATION_EMAIL_TEMPLATE_CODE=registrationREGISTRATION_OTP_EMAIL_TEMPLATE_CODE=registrationotpSALT_ROUNDS=10SAMPLE_CSV_FILE_PATH=sample/bulk_user_creation.csvSCHEDULER_SERVICE_BASE_URL=/scheduler/SCHEDULER_SERVICE_ERROR_REPORTING_EMAIL_ID=rakesh.k@pacewisdom.comSCHEDULER_SERVICE_HOST=http://localhost:3567SCHEDULER_SERVICE_URL=http://localhost:3567/jobs/scheduleJobcreated_time=2024-02-08T07:40:04.571464939Zcustom_metadata=nulldestroyed=falseversion=31

Save and exit.

Setting up Databases

Log into the postgres user

sudo su postgres

Log into psql

psql -p 9700

Create a database user/role:

CREATEUSERshikshalokam WITH ENCRYPTED PASSWORD 'slpassword';

Create the elevate_user database

CREATEDATABASEelevate_user;
GRANT ALL PRIVILEGES ON DATABASE elevate_user TO shikshalokam;
\c elevate_user
GRANT ALL ON SCHEMA public TO shikshalokam;

Running Migrations To Create Tables

Exit the postgres user account and install sequelize-cli globally

$ sudo npm i sequelize-cli -g

Navigate to the src folder of user service and run sequelize-cli migration command:

user/src$ npx sequelize-cli db:migrate

Now all the tables must be available in the Citus databases

Setting up Distribution Columns in Citus PostgreSQL Database

Refer Choosing Distribution Column for more information regarding Citus distribution columns.

Login into the postgres user

sudo su postgres

Login to psql

psql -p 9700

Login to the elevate_user database

\c elevate_user

Enable Citus for elevate_user

CREATE EXTENSION citus;

Within elevate_user, run the following queries:

SELECT create_distributed_table('entities', 'entity_type_id');
SELECT create_distributed_table('entity_types', 'organization_id');
SELECT create_distributed_table('file_uploads', 'organization_id');
SELECT create_distributed_table('forms', 'organization_id');
SELECT create_distributed_table('notification_templates', 'organization_id');
SELECT create_distributed_table('organizations', 'id');
SELECT create_distributed_table('organization_codes', 'code');
SELECT create_distributed_table('organization_domains', 'domain');
SELECT create_distributed_table('organization_role_requests','organization_id');
SELECT create_distributed_table('organization_user_invites','organization_id');
SELECT create_distributed_table('users_credentials','email');
SELECT create_distributed_table('users', 'organization_id');

Running Seeder to Populate the Tables with Seed Data

Exit the postgres user navigate to the script folder of the user service

Run the insertDefaultOrg.js script

src/scripts$ node insertDefaultOrg.js

Keep note of the default organization id generated by the script

Navigate to the src folder of the user service and update the .env file with these variables:

DEFAULT_ORG_ID=<id generated by the insertDefaultOrg script>DEFAULT_ORGANISATION_CODE=default_code

Run the seeder command

src$ npm run db:seed:all

Start the Service

Run pm2 start command:

user/src$ pm2 start app.js -i 2 --name elevate-user

Run pm2 ls command

$ pm2 ls

Output should look like this (Sample output, might slightly differ in your installation):

┌────┬─────────────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name │ namespace │ version │ mode │ pid │ uptime │ ↺ │ status │ cpu │ mem │ user │ watching │
├────┼─────────────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 1 │ elevate-user │ default │ 1.0.0 │ cluster │ 106976 │ 27h │ 0 │ online │ 0% │ 167.0mb │ jenkins │ disabled │
│ 2 │ elevate-user │ default │ 1.0.0 │ cluster │ 106986 │ 27h │ 0 │ online │ 0% │ 169.3mb │ jenkins │ disabled │
└────┴─────────────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

This concludes the services and dependency setup.

Postman Collections

User Service

Migrations Commands

Check migrations

npm run elevate-migrations s

Create migrations

npm run elevate-migrations create categoryEntity #Where categoryEntity is the file name.

20220726145008-categoryEntity.js

We have followed the following structure for migration files to reduce code duplication.

letcategories=[{value: 'sqaa',label: 'SQAA',image: 'entity/SQAA.jpg',},{value: 'communication',label: 'Communication',image: 'entity/Communication.png',},
...
]varmoment=require('moment')module.exports={asyncup(db){global.migrationMsg='Uploaded categories entity'letentityData=[]categories.forEach(asyncfunction(category){category['status']='ACTIVE'category['deleted']=falsecategory['type']='categories'category['updatedAt']=moment().format()category['createdAt']=moment().format()category['createdBy']='SYSTEM'category['updatedBy']='SYSTEM'entityData.push(category)})awaitdb.collection('entities').insertMany(entityData)},asyncdown(db){db.collection('entities').deleteMany({value: {$in: categories.map((category)=>category.value)},})},}

Run migrations

npm run elevate-migrations up

Down migrations

npm run elevate-migrations down

To know more about migrations refer project Wiki

Run tests

Integration tests

npm run test:integration

To know more about integration tests and their implementation refer to the project Wiki.

Unit tests

npm test

Used in

This project was built to be used with Mentoring Service, Project Service, Survey Service, User Service.

The frontend/mobile application for Mentoring repo and Projects and Survey repo

You can learn more about the full implementation of various capabilities of ELEVATE here .

Team

Open Source Dependencies

Several open-source dependencies have aided user service development:

Name		Name	Last commit message	Last commit date
Latest commit History 2,616 Commits
.circleci		.circleci
.github/workflows		.github/workflows
deployment		deployment
dev-ops		dev-ops
release-notes		release-notes
src		src
.coderabbit.yaml		.coderabbit.yaml
.dockerignore		.dockerignore
.gitignore		.gitignore
.prettierrc.json		.prettierrc.json
AGENTS.md		AGENTS.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
docker-compose.yml		docker-compose.yml
docker.readme.md		docker.readme.md
package.json		package.json
sonar-project.properties		sonar-project.properties

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

User Service

System Requirements

Setup Options

Local Dependencies Steps

Remote Dependencies Steps

Installations

Install Node.js LTS

Install Build Essential

Install Kafka

Install Redis

Install Single-Node Citus (Distributed Database)

Install PM2

Setting up Repository

Clone the user repository to /opt/backend directory

Install Npm packages from src directory

Create .env file in src directory

Setting up Databases

Running Migrations To Create Tables

Setting up Distribution Columns in Citus PostgreSQL Database

Running Seeder to Populate the Tables with Seed Data

Start the Service

Run pm2 ls command

Postman Collections

Migrations Commands

Check migrations

Create migrations

Run migrations

Down migrations

Run tests

Integration tests

Unit tests

Used in

Team

Open Source Dependencies

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages