Lyric

A model-agnostic, tabular data submission system designed to manage and validate submissions. This monorepo contains both the server application and shared libraries, managed using PNPM package manager.

Lyric is part of Overture, a collection of open-source software microservices used to create platforms for researchers to organize and share genomics data.

Local Development

Development Tools

• PNPM Project manager
• Node.js Runtime environment (v20 or higher)
• VS Code As recommended code editor. Plugins recommended: ESLint, Prettier - Code formatter, Mocha Test Explorer, Monorepo Workspace

System Dependencies

• A Postgres Database for data storage

Complementary Services

• Lectern Dictionary Management and validation

Quickstart Development

1. Install Dependencies:

   pnpm i
   

2. Build the Workspace:

   pnpm build:all
   

3. Set Environment Variables: Create a .env file based on .env.schema. See Environment Variables section below.

4. Start the Server in Development Mode:

   pnpm start:dev
   

   Server runs on port 3030 by default.

5. Interact with API Endpoints: Access Swagger UI at http://localhost:3030/api-docs/

Environment Variables

Name	Description	Default
ALLOWED_ORIGINS	Specifies a list of permitted origins for Cross-Origin Resource Sharing (CORS). These origins, separated by commas, are allowed to make requests to the server, ensuring only trusted domains can access resources. (Example: https://www.example.com,https://subdomain.example.com)
AUDIT_ENABLED	Ensures that any modifications to the submitted data are logged, providing a way to identify who made changes and when they were made.	true
CORS_ENABLED	Controls whether the CORS functionality is enabled or disabled.	false
DB_HOST	Database Hostname
DB_NAME	Database Name
DB_PASSWORD	Database Password
DB_PORT	Database Port
DB_USER	Database User
ID_CUSTOM_ALPHABET	Custom Alphabet for local ID generation	'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
ID_CUSTOM_SIZE	Custom size of ID for local ID generation	21
ID_USELOCAL	Generate ID locally	true
KAFKA_BROKERS	Comma-separated list of Kafka broker addresses (e.g. localhost:9092). When set, Lyric publishes each committed record to a Kafka topic. Omit to run without Kafka publishing.
KAFKA_CLIENT_ID	Kafka client identifier. Use a unique value per environment to distinguish producers in broker logs.	lyric
KAFKA_TOPIC	Kafka topic that committed records are published to. The topic is created automatically on startup if it does not exist. Required when KAFKA_BROKERS is set.
LECTERN_URL	Schema Service (Lectern) URL
LOG_LEVEL	Log Level	'info'
PLURALIZE_SCHEMAS_ENABLED	This feature automatically convert schema names to their plural forms when handling compound documents. Pluralization assumes the words are in English	true
PORT	Server Port	3030
VALIDATOR_CONFIG	Defines a configurable endpoint for validating the existence of a record with a specific value. The configuration is provided as an array of objects, where each object must include 'categoryId', 'entityName', and 'fieldName'. 'categoryId' accepts either a category's numeric id or its alias. These entries determine which entity-field combinations are enabled for validation. (Example: [{"categoryId": "1", "entityName": "sample", "fieldName": "sample_id" }])

Script Commands

Command	Description
pnpm build:all	Compile typescript code and generate database schemas
pnpm start:dev	Run database migration and start Server for development
pnpm start:prod	Run database migration and start Server for production

Documentation

• Developer Documentation: Technical resources for developers working with or contributing to the project. This content can also be read and updated within the /docs folder of this repository.

Support & Contributions

• For support, feature requests, and bug reports, please see our Support Guide.

• For detailed information on how to contribute to this project, please see our Contributing Guide.

Related Software

The Overture Platform includes the following Overture Components:

Software	Description
Score	Transfer data to and from any cloud-based storage system
Song	Catalog and manage metadata associated to file data spread across cloud storage systems
Maestro	Organizing your distributed data into a centralized Elasticsearch index
Arranger	A search API with reusable search UI components
Stage	A React-based web portal scaffolding
Lyric	A model-agnostic, tabular data submission system
Lectern	Schema Manager, designed to validate, store, and manage collections of data dictionaries.

Funding Acknowledgement

Overture is supported by grant #U24CA253529 from the National Cancer Institute at the US National Institutes of Health, and additional funding from Genome Canada, the Canada Foundation for Innovation, the Canadian Institutes of Health Research, Canarie, and the Ontario Institute for Cancer Research.