The data portal is a series of client-side dashboards used to access cached data and metadata from AIND data assets. Current views include an overview of all Assets, individual Subject pages, a tool for tracking Contributions, and a Project view.
The data portal also hosts semi-hidden dashboards that are for specific projects or purposes. These include a Behavior Sessions dashboard used to track which projects and experimenters are running behavioral training at AIND, the SmartSPIM dashboard used to view all SmartSPIM platform data assets and easily view them, and a client-side only version of the QC Portal which reduces load on the Panel app used for interactive editing.
You can also directly view, download, and pull into Python code the backing cache tables.
Data Portal apps are lightweight front-ends that pull all of their data from cached tables. The principle for the site is that while static immutable data assets with standardized metadata are critical for ensuring data meets the FAIR standards, they are often unwieldy for data analysis. The first thing that happens in almost every analysis is that data from a diverse set of incoming assets gets re-formatted into a set of tables. The second thing that happens is that the tables get materialized into figures -- this portal is intended to make this second step intuitive and interactive.
Some of the properties we aim to embed in these portals:
- Intuitive: Portals should be clear about what they are and pull users in without requiring reading, tutorials, or text-based interaction.
- Interactive: Tables should be filterable, timelines should be windowable, figures that share axes should share filters, links should move you between portal views, 3D views should rotate and be clickable, etc...
- Playful: Diverse affordances should create the possibility of discovering things in the data (and metadata) that weren't surfaced intentionally by the developers.
Set AWS_PROFILE before starting the server so it can read the S3 Parquet files.
In production, nginx serves the static Vite build on port 8000 and forwards the required S3-list, log-server, and DocDB requests to the Python proxy on port 3001. DuckDB runs in each browser through DuckDB-WASM. Supervisor manages nginx and the Python proxy.
# Python dependencies for local proxy endpoints
uv sync
# Node dependenciescd web && npm installcd web
npm start # launches the Python proxy and Vite dev serverOr run them separately in two terminals:
# Terminal 1 — Python proxy for S3 listing, log-server, and DocDB requestscd web && npm run docdb
# Terminal 2 — Vite dev servercd web && npm run devOpen http://localhost:5173.
cd web && npm testA Docker container bundles the static Vite build and Python proxy behind nginx. DuckDB-WASM remains browser-side and requires no container service.
docker build -t zombie .
docker run -p 8000:8000 zombieBecause they eat brains!