Lectern - Data Dictionary Management and Validation

Lectern is Overture's Data Dictionary Schema Manager, providing a system for defining Schemas that will validate the structured data collected by an application. The core of Lectern is a web-server application that handles storage and version management of data dictionaries. Lectern data dictionaries are collections of schemas that define the structure of tabular data files (like TSV). This application provides functionality to validate the structure of data dictionaries, maintain a list of dictionary versions, and to compute the difference between dictionary versions.

Lectern is part of Overture, a collection of open-source software microservices used to create platforms for researchers to organize and share genomics data.

Repository Structure

This repository is organized as a monorepo using pnpm-workspace and nx.

Note: You will need to use pnpm instead of npm to manage dependencies in this code base. PNPM will take care of linking all modules together correctly.

Workspace Modules

The repository is organized with the following directory structure:

.
├── apps/
│ └── server
└── packages/
├── client
├── common
├── dictionary
└── validation

The modules in the monorepo are organized into two categories:

• apps/ - Standalone processes meant to be run. These are published to ghcr.io as container images.
• packages/ - Reusable packages shared between applications and other packages. Packages are published to NPM.
• scripts - Utility scripts for use within this repo.

Component Overview

Component	Package Name	Path	Published Location	Description
Lectern Server	@overture-stack/lectern-server	apps/server/		Lectern Server web application.
Lectern Client	@overture-stack/lectern-client	packages/client		TypeScript Client to interact with Lectern Server and Lectern data dictionaries.
Lectern Dictionary	@overture-stack/lectern-dictionary	packages/dictionary		Dictionary meta-schema definition, includes TS types, and Zod schemas.
Lectern Validation	@overture-stack/lectern-validation	packages/validation/		Validate data using Lectern Dictionaries.

Development Environment

Prerequisites

• PNPM (instead of npm)
• Node.js
• Docker (for running containers)

Local Development

You can install all dependencies for the entire repo from the root (as defined in the pnpm-lock.yaml) with the command:

pnpm install

Common Commands

Run these from the root directory, or if you are in a sub directory then use pnpm -w:

Build Everything

pnpm build:all

Test Everything

pnpm test:all

Using nx will ensure all local dependencies are built, in the correct sequence. For example:

pnpm nx build @overture-stack/lectern-server
pnpm nx build @overture-stack/lectern-client

For convenience, use short aliases:

pnpm build:client

Documentation

Technical resources for those working with or contributing to the project are being updated to our official documentation site, this content can also be updated and read within the /docs folder of this repository.

Meta-Schema

Lectern provides a meta-schema definition that describes the structure of Lectern Dictionaries. The generated JSON Schema formatted copy of this schema can be found at ./generated/DictionaryMetaSchema.json.

Note

Don't manually update any files in the ./generated path. This content is programatically generated from the source code.

Support & Contributions

• For support, feature requests, and bug reports, please see our Support Guide.

• For detailed information on how to contribute to this project, please see our Contributing Guide.

Related Software

The Overture Platform includes the following Overture Components:

Software	Description
Score	Transfer data to and from any cloud-based storage system
Song	Catalog and manage metadata associated to file data spread across cloud storage systems
Maestro	Organizing your distributed data into a centralized Elasticsearch index
Arranger	A search API with reusable search UI components
Stage	A React-based web portal scaffolding
Lyric	A model-agnostic, tabular data submission system
Lectern	Schema Manager, designed to validate, store, and manage collections of data dictionaries.

Funding Acknowledgement

Overture is supported by grant #U24CA253529 from the National Cancer Institute at the US National Institutes of Health, and additional funding from Genome Canada, the Canada Foundation for Innovation, the Canadian Institutes of Health Research, Canarie, and the Ontario Institute for Cancer Research.