FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures

About

This is the official implementation of the FISBe (FlyLight Instance Segmentation Benchmark) evaluation pipeline. It is the first publicly available multi-neuron light microscopy dataset with pixel-wise annotations.

Download the dataset:https://kainmueller-lab.github.io/fisbe/

The benchmark supports 2D and 3D segmentations and computes a wide range of commonly used evaluation metrics (e.g., AP, F1, coverage). Crucially, it provides specialized error attribution for topological errors (False Merges, False Splits) relevant to filamentous structures.

Key Features

• Official Protocol: Implements the exact ranking score ($S$) and matching logic defined in the FISBe paper.
• Topology-Aware: Uses skeleton-based localization (clDice) to handle thin structures robustly.
• Error Attribution: Explicitly quantifies False Merges (FM) and False Splits (FS) via many-to-many matching.
• Flexibility: Supports HDF5 (.hdf, .h5) and Zarr (.zarr) files.
• Modes: Single file, folder evaluation, or 3x stability analysis.
• Partly Labeled Support: Robust evaluation that ignores background conflicts for sparse Ground Truth.

---

Installation

The recommended way to install is using uv (fastest) or micromamba.

Option 1: Using uv (Fastest)

pip install uv
git clone https://github.com/Kainmueller-Lab/evaluate-instance-segmentation.git
cd evaluate-instance-segmentation
uv venv
uv pip install -e .

Option 2: Using micromamba or conda

micromamba create -n evalinstseg python=3.10
micromamba activate evalinstseg
git clone https://github.com/Kainmueller-Lab/evaluate-instance-segmentation.git
cd evaluate-instance-segmentation
pip install -e .

Usage: Command Line (CLI)

The evalinstseg command is automatically available after installation.

1. Evaluate a Single File

evalinstseg \
--res_file tests/pred/sample_01.hdf \
--res_key volumes/gmm_label_cleaned \
--gt_file tests/gt/sample_01.zarr \
--gt_key volumes/gt_instances \
--split_file assets/sample_list_per_split.txt \
--out_dir tests/results \
--app flylight

2. Evaluate an Entire Folder

If you provide a directory path to --res_file, the tool will look for matching Ground Truth files in the --gt_file folder. Files are matched by name.

evalinstseg \
--res_file /path/to/predictions_folder \
--res_key volumes/gmm_label_cleaned \
--gt_file /path/to/ground_truth_folder \
--gt_key volumes/gt_instances \
--out_dir /path/to/output_folder \
--app flylight

3. Stability & Robustness Mode

Compute the Mean ± Std of metrics across exactly 3 different training runs (e.g., different random seeds).

evalinstseg \
--stability_mode \
--run_dirs experiments/seed1 experiments/seed2 experiments/seed3 \
--gt_file data/ground_truth_folder \
--out_dir results/stability_report \
--app flylight

Requirements:

• --run_dirs: Provide exactly 3 folders.
• --gt_file: The folder containing Ground Truth files (filenames must match predictions).

4. Partly Labeled Data

If your ground truth is sparse (not fully dense), use the --partly flag. See the Partly Labeled Data Mode section for details on how False Positives are handled.

Usage: Python Package

You can integrate the benchmark directly into your Python scripts or notebooks.

Evaluate a File

fromevalinstsegimportevaluate_file# Run evaluationmetrics=evaluate_file(
res_file="tests/pred/sample_01.hdf",
gt_file="tests/gt/sample_01.zarr",
res_key="volumes/labels",
gt_key="volumes/gt_instances",
out_dir="output_folder",
ndim=3,
app="flylight", # Applies default FISBe configpartly=False# Set True for sparse GT
)
# Access metrics directlyprint("AP:", metrics['confusion_matrix']['avAP'])
print("False Merges:", metrics['general']['FM'])

Evaluate Raw Numpy Arrays

If you already have the arrays loaded in memory:

importnumpyasnpfromevalinstsegimportevaluate_volumepred_array=np.load(...) # Shape: (Z, Y, X)gt_array=np.load(...)
metrics=evaluate_volume(
gt_labels=gt_array,
pred_labels=pred_array,
ndim=3,
outFn="output_path_prefix",
localization_criterion="cldice", # or 'iou'assignment_strategy="greedy",
add_general_metrics=["false_merge", "false_split"]
)

FISBe Benchmark Protocol

For a complete reference of all calculated metrics, see docs/METRICS.md.

Note: Some output keys use internal names; see the documentation for the exact mapping to website/leaderboard columns.

Official FlyLight Configuration (--app flylight)

The flylight preset implements the specific metrics described in the FISBe paper for evaluating long-range thin filamentous neuronal structures.

Primary Ranking Score ($S$) The single scalar used to rank methods on the leaderboard: $$S = 0.5 \cdot \text{avF1} + 0.5 \cdot C$$

Key Metrics

• avF1: Average F1 score across clDice thresholds.
• C (Coverage): Average GT skeleton coverage (assignment via max clPrecision; scoring via clRecall on union of matches).
• clDiceTP: Average clDice score of matched TPs at threshold 0.5.
• tp: Relative number of TPs at threshold 0.5 (TP_0.5 / N_GT).
• FS (False Splits): Sum over GT of max(0, N_assigned_pred - 1).
• FM (False Merges): Sum over predictions of max(0, N_assigned_gt - 1).

Partly Labeled Data Mode (--partly)

FISBe includes 71 partly labeled images where only a subset of neurons is annotated.

• Logic: Unmatched predictions are only counted as False Positives if they match a Foreground GT instance.
• Background Exclusion: Predictions matching background (unlabeled regions) are ignored.

Output Structure

Metrics returned by the API or saved to disk are grouped into category-specific dictionaries:

metrics["confusion_matrix"]
├── TP/FP/FN# Counts across all images
├── precision/recall# Standard detection metrics
└── avAP# Mean precision × recall proxymetrics["general"]
├── aggregate_score# S (Official Ranking Score)
├── avg_gt_skel_coverage# C (Coverage)
├── FM# Global False Merge count
└── FS# Global False Split countmetrics["curves"]
└── F1_0.1 … F1_0.9# Per-threshold performance