Opensource Data

Data source for opensource frontend. This repo scrape data hourly and publish it as JSON files.

Usage

Data sources are available at: developersdo.github.io/opensource-data.

Development

You will need to:

1. Clone this repo.
2. Install dependencies: npm i
3. Run migrations: npm run migrations:run
4. Copy .env.default to .env.
5. Create a personal GitHub access token (with no scopes) and add the value to OSD_GH_TOKEN in your .env.
   • Note: You can skip that if you don't need to run the scraper locally.

Scripts

Script	Description
npm run migrations:run	Run all migrations.
npm run migrations:add [name]	Add a new migration.
npm run scrape	Scrape all data.
npm run scrape -- --only=users	Scrape users data only.
npm run scrape -- --only=repos	Scrape repos data only.
npm run generate	Generate public site and JSON files.

Debugging

Prepend any commmand with DEBUG=osd:*. For example:

DEBUG=osd:* npm run scrape