Pogy.db

Pogy.db is a mongoose (v.6.5.2) based database which is used in Pogy. Our database makes it easy for migrations, complicated queries, supports in map caching, logging to know what is happening within the database and more.

Endured storage, your data doesn't disappear on restarts.
Supports both mongoose and redis TTL.
Migrations, to backup your data every once in a while.
Caching using Maps or Redis, to speed up your queries.
Logging, get real-time updates on what is happening within the database and redis.
Ping, check the execution time of your queries.
Fully customizable, you can customize your database, tables and queries.
Built in support for Redis, that means you move from Maps to Redis as your application grows.
Beginner Friendly, Dot notation support, Asynchronous, and easy to use.

Help

Click here to join our support server.

Download

npm install pogy.db --save

Example

constdatabase=require("pogy.db");//orimportdatabasefrom"pogy.db";asyncfunctionstart(){//start the connection to the databaseawaitdatabase.connect("mongodb://localhost:27017/test",{cache: {toggle: true,cacheOnly: false},logs: {file: "database.log",hidden: false},redis: {url: "redis://localhost:6379",}},{keepAlive: true,minPoolSize: 3,maxPoolSize: 10,serverSelectionTimeoutMS: 10000,socketTimeoutMS: 60000,});// Create a table or what you call as a collection// this will create a table called "users" if its not already created.// extended from Model<CollectionInterface<unknown>>constschema=awaitnewdatabase.table("users",{cacheLargeData: false,catchErrors: true});if(schema){awaitschema.set("discord",{id: "710465231779790849",username: "Peter_#4444",},{returnData: true,database: {ttl: 1},redis: {ttl: 1},cache: {toggle: true,cacheOnly: false,}});awaitschema.push("discord.badges","verified");awaitschema.pull("discord.badges","verified");awaitschema.add("discord.message_count",1);awaitschema.subtract("discord.message_count",1);awaitschema.get("discord.badges");awaitschema.has("discord.badges");// -> trueawaitschema.delete("discord.badges");awaitschema.shift("discord.badges");awaitschema.unshift("discord.badges","verified");awaitschema.stats();awaitschema.all({documentForm: false,cache: {cacheOnly: false;},limit: 10,sort: "id",filter: (doc)=>doc.id==="710465231779790849"});awaitschema.drop();}}start();

Test this Library

You can test all the functions when downloading the repo at once by typing:

node test --mongoURL --redisURL

but replace "mongoURL" with the url of your database and "redisURL" (OPTIONAL) with the url of your redis server.

example: node test --mongodb://localhost:27017/test --redis://localhost:6379
- No Redis: node test --mongodb://localhost:27017/test

What you need to know

When using database ttl, the existing cache will automatically be detected and deleted once the main data has been deleted from the database.
If redis is somehow unavailable, the library will rely on raw mongoose.
All functions such as .delete() and .drop() are designed to delete existing cache related to the key when executed, so don't worry about useless cache taking up space.

Documentation

All functions in this package return a promise. So await is needed to get the result.

database.connect() (async Function) - Click here to go
database.table() (async Function) - Click here to go
database.migrate() (async Function) - Click here to go
database.ping() (async Function) - Click here to go
database.isOnline() (Function) - Click here to go
database.DatabaseManager (Class)Click here to go

Connect

connect(url, options, mongooseOptions)

	url	options	mongooseOptions
Type	string	Click Here	Click Here

connect-options

defined in src\types\index.ts [5-17]

{cache?: {toggle?: boolean;// enable cache by defaultcacheOnly?: boolean;// only use cache by default};
logs?: {hidden?: boolean;// hide logs
file?: string;// log file path (ex: src/database.log)};
redis?: {
url: string;// redis connection url (ex: redis://localhost:6379)};};

connect-mongooseOptions

Mongoose options are defined in the Mongoose Documentation. They are the options mongoose uses to connect to the database.

default:

{keepAlive: true,minPoolSize: 3,maxPoolSize: 10,serverSelectionTimeoutMS: 10000,socketTimeoutMS: 60000}

TABLE

table(name, options)

This function is used to create a table in the database or fetch an existing table; its as if you are creating a model in mongoose.

returns:CustomizedTable (src/types/index.ts [56-154])

options:

tableName: string,// table nametableOptions?: {cacheLargeData?: boolean;// cache large data (first key) like table.set(key) instead of table.set(key.subKey)
catchErrors?: boolean;// log the errors of the table if there are ever any
watchDeletions?: boolean;// watch for deletions in the table (used to delete cache)}

watchDeletions This is used whenever you want to create a schema with cache on, and delete its cache automatically once the schema is deleted.

IF CACHE IS DISABLED, THIS WON'T BE NEEDED!

.watch in mongoose lowers the performance of the database, so it is recommended to only use this when you need it. Also keep in mind that when you restart your instance, you will have to re-initialize the watch and setting the data. Therefore, its recommended you only use this while deleting data in a short period of time. If you still want to use ttl, use ttl but with cache disabled. You can simply add { cache: {toggle: false} } to the options of the function and that is it.

Create the table and enable watchDeletions to delete the cache when a schema in the table is deleted.

constschema=awaitnewdatabase.table("users",{watchDeletions: true});

Start by setting the data.

awaitschema.set("discord",{id: "710465231779790849",username: "Peter_#4444"},{cache: {toggle: true,cacheOnly: false},database: {ttl: 1//schema will be deleted after 1 second}});

Since cache is on and watchDeletions is enabled, the cache will be deleted once a schema with ttl is deleted in the table.

example:

constusers=awaitnewdatabase.table("users",{catchErrors: true});if(users){awaitusers.set(message.author.id,{username: message.author.username,discriminator: message.author.discriminator,id: message.author.id,avatar: message.author.avatarURL,badges: [],message_count: 0});constusername=awaitusers.get(`${message.author.id}.username`);console.log(username);// -> "Peter_#4444"}

get(key, options)

This function is used to get the value of a key in the database.

returns:Promise<null | string | number | unknown>)

params:

key: string,// the key to get the value ofoptions?: {cache?: {toggle?: boolean;// if true, it will cache the data, if false it will only take the data from the databasecacheOnly?: boolean;// if true, it will only take the data from the cache};}

example:

constuser=awaitusers.get(message.author.id,{cache: {toggle: true,cacheOnly: false}});console.log(user);// -> { username: "Peter_#4444", discriminator: "4444", id: "710465231779790849", avatar: "avatar", badges: [], message_count: 0 }constusername=awaitusers.get(`${message.author.id}.username`);console.log(username);// -> "Peter_#4444"

set(key,value, options)

This function is used to set the value of a key in the database.

returns:Promise<null | boolean | unkown>)

params:

 key: string,// the key to set the value ofvalue: string|number|boolean|unknown,// the value to set the key tooptions?: {cache?: {toggle?: boolean;// if true, it will cache the data, if false it will only take the data from the databasecacheOnly?: boolean;// if true, it will only take the data from the cache};
returnData?: boolean;// if true, it will return the data instead of a boolean
database?: {ttl?: number;// the time to expire the data in the database (in seconds)};
redis?: {ttl?: number;// the time to expire the data in the redis cache (in seconds)};}

example:

awaitusers.set(`${message.author.id}.username`,"Peter_#4444",{cache: {toggle: true,cacheOnly: false},returnData: true,database: {ttl: 60},redis: {ttl: 60}});// -> { username: "Peter_#4444", discriminator: "4444", id: "710465231779790849", avatar: "avatar", badges: [], message_count: 0 }

add(key, number, options)

This function is used to add a number to a key in the database.

returns:Promise<null | boolean | unkown>)

params:

 key: string,// the key to add the number tovalue: number|string,// the number to add to the keyoptions?: {cache?: {toggle?: boolean;// if true, it use the cachecacheOnly?: boolean;// if true, it will only set the data in the cache};
returnData?: boolean;// if true, it will return the data instead of a boolean}

example

awaitusers.add(`${message.author.id}.message_count`,1);

subtract(key, number,options)

This function is used to subtract a number from a key in the database.

returns:Promise<null | boolean | unkown>)

params:

 key: string,// the key to subtract the number fromvalue: number|string,// the number to subtract from the keyoptions?: {cache?: {toggle?: boolean;// if true, it use the cachecacheOnly?: boolean;// if true, it will only set the data in the cache};
returnData?: boolean;}

example

awaitusers.subtract(`${message.author.id}.message_count`,1);

has(key, options)

This function is used to check if a key exists in the database or cache. (boolean)

returns:Promise<null | boolean>)

params:

 key: string,// the key to check if it existsoptions?: {cache?: {cacheOnly?: boolean;// if true, it will only check the cache};}

example

awaitusers.has(message.author.id,{cache: {cacheOnly: true}});// -> trueawaitusers.has(`${message.author.id}.invalid_property`);// -> false

delete(key, options)

This function is used to delete a key in the database.

returns:Promise<null | boolean>)

params:

 key: string,// the key to check if it existsoptions?: {cache?: {cacheOnly?: boolean;// if true, it will only delete the cache};}

example

awaitusers.delete(message.author.id);// -> true

all(options)

This function is used to get all the collections from the table in the database.

returns:Promise<unknown>)

params:

 documentForm?: boolean;// if true, it will return the data in a document form
cache?: {cacheOnly?: boolean;// if true, it will only get the data from the cache};
limit?: number;// the limit of the data to get
sort?: string;// the key to sort the data by
filter?: (data)=>boolean;// a filter function to filter the data

example

constusersInDatabase=awaitusers.all({documentForm: true,cache: {cacheOnly: false},limit: 10,sort: "message_count",filter: (data)=>data.message_count>100});

push(key, element, options)

This function is used to push an element to an array in the database or create an array if it doesn't exist.

returns:Promise<null | boolean | unknown>)

params:

 key: string,// the key to push the element tovalue: string|number|boolean|unknown,// the element to push to the arrayoptions?: {cache?: {toggle?: boolean;// if true, it will use the cache};
returnData?: boolean;// if true, it will return the data instead of a boolean}

example

awaitusers.push(`${message.author.id}.badges`,"bot_owner",{returnData: false,cache: {toggle: false}});// -> true

pull(key, element, options)

This function is used to pull an element from an array in the database.

returns:Promise<null | boolean | unknown>)

params:

 key: string,// the key to push the element tovalue: string|number|boolean|unknown,// the element to push to the arrayoptions?: {cache?: {toggle?: boolean;// if true, it will use the cache};
returnData?: boolean;// if true, it will return the data instead of a boolean}

example

awaitusers.pull(`${message.author.id}.badges`,"bot_owner",{returnData: false,cache: {toggle: false}});// -> true

shift(key, element, options)

This function is used to shift an element from an array in the database.

returns:Promise<null | boolean | unknown>)

params:

 key: string,// the key to push the element tovalue: string|number|boolean|unknown,// the element to push to the arrayoptions?: {cache?: {toggle?: boolean;// if true, it will use the cache};
returnData?: boolean;// if true, it will return the data instead of a boolean}

example

awaitusers.shift(`${message.author.id}.badges`,"bot_owner",{returnData: false,cache: {toggle: false}});// -> true

unshift(key, element, options)

This function is used to unshift an element to an array in the database.

returns:Promise<null | boolean | unknown>)

params:

 key: string,// the key to push the element tovalue: string|number|boolean|unknown,// the element to push to the arrayoptions?: {cache?: {toggle?: boolean;// if true, it will use the cache};
returnData?: boolean;// if true, it will return the data instead of a boolean}

example

awaitusers.unshift(`${message.author.id}.badges`,"bot_owner",{returnData: false,cache: {toggle: false}});// -> true

drop()

This function is used to drop the entire table!

returns:Promise<null | boolean>)

example

awaitusers.drop();// -> true

stats()

This function is used to get the stats of the table.

returns:Promise<unknown>

example

conststats=awaitusers.stats();console.log(stats);

table

extended from Model<CollectionInterface<unknown>>(refer to src\types\index.ts)

What if those functions are not enough and you want to use mongoose functions? You can use <your-table>.table.<function> to call the function.

Examples:

users.table.find({})
guilds.table.findOne({})

or

constusers=awaitnewdatabase.table("users",{catchErrors: true// logs errors if there's an error in the function});// if the database is onlineif(users){constcustomTable=awaitusers.table.findOne({});console.log(customTable);}

or

// node js v14+constcustomMongoDB=(awaitnewdatabase.table("users",[(catchErrors: true)// logs errors if there's an error in the function]))?.table;constuser=awaitusers.findOne({id: "710465231779790849"});console.log(user);

MIGRATE

migrate(schema, newConnection, options)

This function is used to migrate the table to a new database connection.

params:

schema: string,// the schema to migrate tonewConnection: string,// the new connection URL to migrate tooptions: {logs?: {hidden?: boolean;// log realtime migration debug information};}

example

constusers=awaitnewdatabase.table("users",{catchErrors: true});// if the database is onlineif(users){awaitdatabase.migrate("users","mongodb://localhost:27017/test2",{logs: {hideLogs: false}});}

ping

Get the execution time of your queries.

results:

cached: boolean;// if the cache is enabled
tableName: string;// the table name
dataToGet: string;// the data to get
timeToGetTable: number;// the time to get the table
timeToGetData: number;// the time to get the data
totalPing: number;// the total ping
redisPing: number|"Redis not enabled.";// the redis ping

If the table or data is not found it will return false data.

example:

constdatabase=require('pogy.db');constping=databasedb.ping({tableName: "users",dataToGet: "710465231779790849"});console.log(ping);

{
"cached": true, //if the data is cached"tableName": "users", // the table name provided"dataToGet": "710465231779790849", // the data requested"timeToGetTable": 0.03450000286102295, // the time taken to get the table in ms"timeToGetData": 0.011600017547607422, // the time taken to get the data in ms"redisPing": 1.380900003015995, // the time taken to ping redis, if redis is enabled"totalPing": 0.04610002040863037, // the total ping (table and data) in ms
}

isOnline

Check if the database is online.

constdatabase=require("pogy.db");constisOnline=database.isOnline();console.log(isOnline);// true or false

DatabaseManager

The database manager that holds the mongoose client, cache, and tables.

constdatabase=require("pogy.db");//orimportdatabasefrom"pogy.db";constDatabaseManager=database.DatabaseManager;DatabaseManager.client;// the mongoose client, returns null if mongoose is not connected.DatabaseManager.cache;// the cache, returns null if the cache is not enabled.DatabaseManager.tables;// the tables, returns an empty array if no tables are created.DatabaseManager.events;// the event emitter, databaseUp or databaseDownDatabaseManager.redis;// the redis client, returns null if redis is not connected.DatabaseManager.redisURL;// the redis url specified

How is this useful?

You can use this to check for instance if the database is connected.

TIP: If you're using redis, and redis somehow disconnects, the database will carry on until redis is reconnected.

constDiscordClient=require("discord.js");constclient=newDiscordClient();constdatabase=require("pogy.db");//orimportdatabasefrom"pogy.db";client.database=database.DatabaseManager.client;console.log(client.database);// database connection or null/* ----------------- or -------------- */constpogyEvents=require("pogy.db").DatabaseManager.events;pogyEvents.on("databaseUp",(data)=>{console.log(data)// { reason: 'CONNECTED - The database connected.', date: 1661543764711 }console.log("database is up");}pogyEvents.on("databaseDown",(data)=>{console.log(data)// { reason: 'DISCONNECTED - The database disconnected.', date: 1661543764711 }console.log("database is down :(");}/* ----------------- or -------------- */constdatabase=require("pogy.db");console.log(database.isOnline());// true or false

Detailed Events

If you want to keep track of your events you may use.

(require("pogy.db").DatabaseManager.events).on("eventName",(data)=>{console.log(data)// the event data}

For Mongoose

Event	Description
databaseUp	Emitted when the database is up.
databaseDown	Emitted when the database is down.

For Redis

Event	Description
redisConnecting	Emitted when redis is connecting on startup.
redisConnected	Emitted when redis is ready and connected.
redisEnd	Emitted when redis is unexpectedly closed using `DatabaseManager.redis.disconnect()` or `DatabaseManager.redis.quit()`
redisError	Emitted when redis encounters an error.
redisReconnecting	Emitted when redis is reconnecting.

Enabling Migrations

Do you want to create a routine to backup your database every once in a while?

constcron=require("node-cron");constdatabase=require("pogy.db");constmongoose=require("mongoose");//orimportdatabasefrom"pogy.db";asyncfunctionconnect(){awaitdatabase.connect("mongodb://localhost:27017/test",{cache: {toggle: true,cacheOnly: false},logs: {file: "database.log",hidden: false},redis: {url: "redis://localhost:6379"}},{keepAlive: true,minPoolSize: 3,maxPoolSize: 10,serverSelectionTimeoutMS: 10000,socketTimeoutMS: 60000});// every 24 hourscron.schedule("0 0 * * *",async()=>{// migrate a certain tableconstmigration1=awaitdatabase.migrate("users","mongodb://localhost:27017/test2",{model: newmongoose.Schema({id: String,data: Object})// the model of the table, this is the default one but you can change it if you wanna customize your table.});// or automatically migrate all tablesdatabase.DatabaseManager.tables.forEach(async(table)=>{constmigrationData=awaitdatabase.migrate(table,"mongodb://localhost:27017/test2",{model: newmongoose.Schema({id: String,data: Object})// the model of the table, this is the default one but you can change it if you wanna customize your table.});});}).start();}connect();

Migration Results

errors: Array<{error: Error|boolean;date: number;step: number;}>;// the errors if there's any
date: number;// the date of the migration
timeTaken: number;// the time taken to migrate
table: string;// the table migrated
dataCreated: number;// the amount of data created

Using Redis

To use Redis, you must have RedisJSON installed on your Redis server. RedisJSON allows us to store JSON data in Redis.

RedisJSON Docs

run this command after instalation

redis-server --loadmodule ./target/release/librejson.so

Hosting Mongo Locally

If you are hosting mongo locally you must turn your database into a Replica Set.

First, host mongo on another port ex.27018

mongod --port 27018 --replSet any-name --bind_ip localhost

Then, open another terminal and execute:

mongosh --port 27018

then

rs.initiate()

This will initialize your database as a replica set.

Name		Name	Last commit message	Last commit date
.vscode		.vscode
lib		lib
src		src
test		test
.eslintrc.json		.eslintrc.json
.gitattributes		.gitattributes
.gitignore		.gitignore
.npmignore		.npmignore
.prettierrc		.prettierrc
CNAME		CNAME
LICENSE.md		LICENSE.md
README.md		README.md
TODO		TODO
package.json		package.json
tsconfig.json		tsconfig.json

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Pogy.db

Help

Download

Example

Test this Library

What you need to know

Documentation

Connect

connect(url, options, mongooseOptions)

connect-options

connect-mongooseOptions

TABLE

table(name, options)

get(key, options)

set(key,value, options)

add(key, number, options)

subtract(key, number,options)

has(key, options)

delete(key, options)

all(options)

push(key, element, options)

pull(key, element, options)

shift(key, element, options)

unshift(key, element, options)

drop()

stats()

table

MIGRATE

migrate(schema, newConnection, options)

ping

isOnline

DatabaseManager

How is this useful?

Detailed Events

For Mongoose

For Redis

Enabling Migrations

Migration Results

Using Redis

Hosting Mongo Locally

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages