A swarm-based metabarcoding pipeline

nf-metabarcoding is a fast, user-friendly, and scalable eDNA workflow built around swarm. It is a first attempt at converting the reference pipeline to Nextflow. Its modular structure allows for meta-studies grouping several independent datasets.

The pipeline aims to produce single-nucleotide resolution clustering results, equivalent to Amplicon Sequence Variants (ASVs). Clustering results, chimera detection results, and taxonomic assignment results are grouped in a ready-to-use occurrence table.

Project layout

main.nf <- entry workflow + fetch / Part B/C standalone routers
nextflow.config <- manifest, parameter defaults, profiles
nextflow_schema.json <- parameter schema (nf-schema validation + --help)
nf-test.config <- nf-test wiring
modules/local/ <- one .nf per process; functions.nf holds the
Groovy helpers (normalize_path, validate_params, ...)
subworkflows/local/ <- part_a / part_b / part_c / fetch subworkflow wiring
SPECIFICATIONS.md <- behaviour spec (single source of truth)
DECISIONS.md <- open spec questions blocking [Sxx] IDs
tests/
README.md <- how to run the tests
COVERAGE.md <- [Sxx] -> test mapping
coverage-gate.sh <- audit script (run in CI)
main.nf.test <- workflow-level test
functions/ <- .nf.test for the Groovy helpers
processes/ <- one .nf.test per process (mirrors modules/local)
data/
README.md <- description of each fixture
generate.sh <- reproducible fixture generation

Prerequisites

Required to run the workflow:

• nextflow >= 25.04.0
• bash >= 4
• vsearch, cutadapt, swarm, mumu available on PATH — or resolved automatically with -profile conda (bioconda), -profile modules, or a container engine (-profile docker/podman/singularity/apptainer; see "Running with containers" below). A container engine only needs the engine itself installed — the tools are built into the image.

Required to run the test suite and the linters:

• nf-test >= 0.9.0 (install with curl -fsSL https://code.askimed.com/install/nf-test | bash)
• bats >= 1.10 (install with apt-get install bats or from bats-core) — for tests/bin/*.bats
• python >= 3.10 with pytest — for tests/python/test_*.py
• flake8 — lints bin/*.py
• shellcheck — lints bin/*.sh

Try it (one command)

[S76]. The bundled demo profile runs the whole pipeline (Part A → B → C) on a tiny committed dataset with no setup and no flags — use it to confirm your install (and, with an engine, your container/cluster setup) works end to end before pointing it at real data:

nextflow run main.nf -profile demo # tools from PATH
nextflow run main.nf -profile demo,singularity # tools in a container
nextflow run main.nf -profile slurm,demo,singularity # on the cluster

Results land under demo_results/. The synthetic reads are not biologically meaningful; the demo only checks that the tools and the wiring work. The dataset lives in assets/demo/ and is regenerated by bash assets/demo/make_demo.sh.

How to run

# generate test fixtures (one-time)
bash tests/data/generate.sh
# run on the bundled fixtures (paired_merge_ok matches canonical row 7# — no --fastq_pattern needed). --outdir is where every result lands.
nextflow run main.nf \
--fastq_folder tests/data \
--outdir results \
--threads 1
# run on your own data — auto-detect via the canonical pattern table
nextflow run main.nf \
--fastq_folder /path/to/fastq_dir \
--outdir /path/to/results
# multiple folders (comma-separated)
nextflow run main.nf \
--fastq_folder /data/run17,/data/run18 \
--outdir /path/to/results

Fetching reads from ENA/SRA (--accession)

Recommendation. This fetch mode is provided for convenience — a quick way to pull a public dataset without leaving the pipeline. For anything more demanding we recommend the more battle-tested nf-core/fetchngs, which covers many more accession types and download edge cases. (Note, however, that nf-core/fetchngs no longer appears to be actively maintained.) A typical workflow is to fetch with fetchngs, then point --fastq_folder at the downloaded reads.

[S97]. Instead of pointing at local fastq files, you can have the pipeline download the raw reads for a public bioproject or study accession. Set --accession to a single accession or a comma-separated list:

# one bioproject
nextflow run main.nf \
--accession PRJEB89924 \
--outdir /path/to/data
# several accessions at once (bioprojects and/or studies)
nextflow run main.nf \
--accession PRJEB89924,PRJNA248678,SRP012345 \
--outdir /path/to/data

This is a standalone fetch mode — Parts A/B/C do not run. It resolves each accession to its constituent runs and downloads one fastq set per run with fastq-dl (pinned, and resolved on demand — only an --accession run pulls it in, so users of the other modes never install it). Files are published under a per-accession subfolder:

<outdir>/
├── PRJEB89924/ ERR..._1.fastq.gz, ERR..._2.fastq.gz, ...
└── SRP012345/ ERR..._1.fastq.gz, ...

Accepted accession forms (validated at startup, [S98]):

Kind	Prefixes	Example
bioproject	PRJEB / PRJNA / PRJDB	PRJEB89924
study	ERP / DRP / SRP	SRP012345

Run accessions (SRR…), samples, and experiments are rejected. --accession is mutually exclusive with the other input selectors (--fastq_folder / --fasta_folder / --occurrence_table / --representatives_fasta / --input).

Notes:

• Outbound network must be reachable from wherever the download tasks run (the login node and, under -profile slurm, the compute nodes).
• A run that fails to download fails only its own task; the runs that succeeded are published and cached, so re-running with -resume retries just the missing runs.
• Fetch does not chain into Part A yet — run Part A as a second step by pointing --fastq_folder at <outdir>/<accession>/.

Output layout (--outdir)

All published artefacts land under --outdir (default results), in a fixed layout:

<outdir>/
├── per_sample/ per-sample data: .fas / .qual / .stats (Part A)
├── occurrence_table/ the occurrence table(s) and the
│ taxonomy-annotated tables (Part B + Part C)
├── logs/ every per-step log, grouped by pipeline stage:
│ ├── part_a/
│ │ ├── per_sample/ Part A step logs (merging / trimming /
│ │ │ dereplicating / clustering)
│ │ └── <basename>_read_counts.tsv per-sample read-tracking
│ │ summary (fastq runs)
│ ├── part_b/
│ │ ├── <basename>_<step>.log Part B step logs
│ │ └── <basename>_read_counts.tsv per-sample read/cluster
│ │ tracking summary (Part B runs)
│ └── part_c/ Part C taxonomy log
└── pipeline_info/ software_versions.yml (tool versions) +
execution_{report,timeline}.html,
execution_trace.txt, pipeline_dag.html

The pipeline_info/ reports ([S84]) are Nextflow's own execution records. execution_trace.txt and execution_report.html list the requested vs observed CPU/memory and the real peak_rss / realtime per task — the easiest way to size --dataset_size_gb / --reference_size_gb (or a per-step override) when tuning the pipeline for a new cluster (see "Running on an HPC cluster").

Inputs are never written to — --fastq_folder / --fasta_folder are read-only.

Migration note (0.1.0, breaking). Earlier versions published Part A artefacts back into --fastq_folder and Part B/C artefacts into --results_folder. As of 0.1.0 everything goes under --outdir in the layout above. --results_folder still works as a deprecated alias for --outdir (with a warning), but the files now live in the occurrence_table/ sub-directory rather than at the top level — update any scripts that read <results_folder>/*.tsv to <outdir>/occurrence_table/*.tsv. Per-step *.log files also moved out of the data directories into a dedicated <outdir>/logs/ tree grouped by pipeline stage (logs/part_a/per_sample/, logs/part_b/, logs/part_c/), and the read-count summary is published to logs/part_a/<basename>_read_counts.tsv — update any scripts that read logs from per_sample/ or occurrence_table/.

Input discovery

The pipeline globs every fastq file in the listed folders (*.fastq, *.fq, with optional .gz / .bz2) and identifies paired-end pairs using the canonical pattern table documented in SPECIFICATIONS.md. Files that match no paired-end pattern (or whose R2 partner is missing) are processed as single-end samples that skip the merging step.

Adding a custom pattern (--fastq_pattern)

If your file names don't match any canonical row, supply your own glob via --fastq_pattern. Format: a shell-style glob containing the literal token {1,2} (or {R1,R2}) that marks the R1/R2 discriminator. A * before the discriminator captures the sample ID.

# study42-mateA.fastq.gz + study42-mateB.fastq.gz → sample "study42"
nextflow run main.nf \
--fastq_folder /data/run17 \
--fastq_pattern '*-mate{A,B}.fastq.gz'# fixed prefix, no '*': the literal prefix (minus trailing _/./-)# becomes the sample ID
nextflow run main.nf \
--fastq_folder tests/data \
--fastq_pattern 'paired_merge_ok_{1,2}.fastq.gz'

User patterns take precedence over the canonical table; supported glob meta-characters are limited to * (any chars) and the {R1,R2} brace token — anything else is matched literally.

A note on input trust

The workflow reads and stages whatever files you point it at. A samplesheet (--input) may list any path on your filesystem — including absolute paths and .. outside the launch directory — and a scanned --fastq_folder / --fasta_folder is read verbatim. This is intentional: you point the pipeline at your own data wherever it lives.

The practical consequence is that the samplesheet and the input folders are a trust boundary. Whoever writes the samplesheet or fills the input folder decides which files the run touches, so treat them the way you would a script you are about to execute: run only samplesheets and folders you or a trusted colleague produced. The pipeline validates sample-ID characters and rejects cells that would corrupt its internal tables, but it does not — and is not meant to — sandbox which paths a samplesheet is allowed to reference.

Running with containers

[S08]. Four engine profiles run every tool inside a container so the only thing you install is the engine itself:

nextflow run main.nf -profile docker ... # Docker
nextflow run main.nf -profile podman ... # Podman
nextflow run main.nf -profile singularity ... # Singularity
nextflow run main.nf -profile apptainer ... # Apptainer

You do not build or pull an image by hand. Each profile turns on Seqera Wave, which builds the image on the fly from environment.yml — the same pinned dependency list the conda profile uses — and caches it for reuse. environment.yml stays the single source of truth: bump a version there and the next run rebuilds the image automatically (see DECISIONS.md D10).

Requirements and notes:

• Outbound network must be reachable from wherever tasks run (the login node and the compute nodes), because Wave resolves and builds the image at task start.
• Combine with the executor for HPC, e.g. -profile slurm,singularity (see below). singularity / apptainer set autoMounts so the work directory and inputs are visible inside the container.
• Pick one dependency mechanism: a container profile, conda, or modules — not several at once.
• Container execution is validated by a manual cluster smoke test; the profile wiring is checked in CI by tests/check-container-profiles.sh.

Air-gapped clusters (no outbound network on compute nodes)

[S83]. Wave builds the image at task start, so it needs outbound network from wherever tasks run. If your compute nodes are air-gapped, use a pre-built image instead — two options, neither needing network from the compute nodes:

1. Build once, run offline. On a connected node (or the login node, if it has network), run the pipeline once under an engine profile so Wave builds the image and caches it on shared scratch:

   # point the cache at shared scratch (see conf/site.config.example)
   nextflow run main.nf -profile slurm,singularity -c my-site.config -profile demo

   Subsequent runs reuse the cached .sif from singularity.cacheDir with no further network.

2. Bring your own image. Pre-pull/transfer an image, then enable the engine and set process.container in your -c site.config, and run with -profile slurm (not-profile singularity, which turns Wave on):

   // my-site.config
   singularity.enabled =true
   singularity.autoMounts =true
   process.container ='/scratch/shared/nf-metabarcoding/nf-metabarcoding.sif'

   nextflow run main.nf -profile slurm -c my-site.config \
   --fastq_folder /scratch/me/run17 --forward_primer ... --reverse_primer ...

   conf/site.config.example carries a commented block for this. A plain -profile slurm with no such -c sets no container, so nothing changes for everyone else.

Running on an HPC cluster (slurm)

[S07]/[S08]. Combine the slurm executor profile with one dependency profile (conda, modules, or a container engine — singularity / apptainer are the usual HPC choices, see "Running with containers" above):

# tools in a container, built on the fly from environment.yml via Wave
nextflow run main.nf -profile slurm,singularity \
--fastq_folder /scratch/me/run17 \
--slurm_queue normal \
--threads 8
# tools from bioconda (Nextflow builds the env once and caches it)
nextflow run main.nf -profile slurm,conda \
--fastq_folder /scratch/me/run17 \
--slurm_queue normal \
--threads 8
# tools from the cluster's module system (set the module names/versions)
nextflow run main.nf -profile slurm,modules \
--fastq_folder /scratch/me/run17 \
--slurm_queue normal \
--module_vsearch vsearch/2.31.0 \
--module_swarm swarm/3.1.7 \
--module_cutadapt cutadapt/4.9 \
--module_mumu mumu/1.1.1 \
--threads 8

What the slurm profile does:

• submits every process as an sbatch job (process.executor = 'slurm'); tune the queue/account/concurrency with --slurm_queue, --slurm_account, --slurm_queue_size. For the per-sample fan-out at scale, --slurm_array_size N submits tasks as slurm job arrays (sbatch --array, up to N tasks per submission) instead of one job each — much gentler on the scheduler, and the pattern HPC admins ask for. Off by default; some cluster profiles set it (e.g. genotoul=50). The driver also ramps submissions at submitRateLimit (50/min).
• sets per-process resources by tier. --threads is the single knob for cores: it feeds task.cpus, which is what the tools actually request (vsearch/swarm/cutadapt --threads/--cores). The single-threaded mumu and the bash/awk/python glue stay at one core; chimera_detection reserves two (it is a filtering | uchime_denovo pipe).
• scales the memory-bound steps off --dataset_size_gb: swarm fastidious global clustering asks for ~3× the dataset size (the heaviest step), the other whole-dataset steps (vsearch global dereplication, chimera detection, the all-vs-all self-search, mumu and occurrence-table assembly) ~1×. The taxonomic-assignment steps (vsearch --usearch_global/--sintax) are reference-bound instead — they load the reference database and a k-mer index, so they scale off --reference_size_gb (~4× to cover the index). Fixed fallbacks apply when those params are unset — and the workflow warns at startup when they are ([S79]), since the fallback may be too small for a large run and OOM mid-pipeline; set the size hints, or override a specific step's memory in a -c site.config (see "Tuning for your cluster" and conf/site.config.example). On an OOM or timeout kill, a process retries up to twice with proportionally more memory and wall-time.
• keeps the [S49] stampa scatter at its slurm default (stampa_chunk_size = 1000); the local profile sets 0 to feed the whole fasta to a single vsearch instead.

Pre-configured clusters (-profile <name>)

[S87]. If you run on a cluster the pipeline already knows about, use its profile instead of hand-writing slurm settings. A cluster profile impliesslurm (you do not list slurm yourself) and sets that site's partitions, account routing, resource ceiling, and container bind mounts — so you just add a dependency engine:

nextflow run main.nf -profile meso,singularity \
--fastq_folder /scratch/me/run17 \
--forward_primer ... --reverse_primer ...

Profile	Site	Engine to pair
abims	ABiMS, Station Biologique de Roscoff	singularity
genotoul	Genotoul, INRAE Toulouse	apptainer²
ifb_core	IFB Core cluster	singularity
meso	meso, CIRAD	singularity
saga	Saga, Sigma2 / NRIS (Norway)	singularity¹

¹ The saga profile's hardware/partition/scratch values come from the Sigma2 docs; confirm with a smoke run before a large production job. Saga also requires a project account: add --slurm_account nnXXXXk.

² Genotoul exposes apptainer/singularity as modules, which the genotoul profile auto-loads (via beforeScript) so -profile genotoul,apptainer (or ,singularity) works directly — the Wave-built image also covers mumu. -profile genotoul,conda is an alternative; -profile genotoul,modules works for VSEARCH/swarm/ Cutadapt but Genotoul has no mumu module.

These configs are vendored in conf/clusters/ — copied into the repo and pinned, not fetched from nf-core/configs at runtime — so they work on air-gapped compute nodes ([S83]) and stay auditable. To add your own site, copy conf/clusters/_template.config, fill in the values, and register it as a profile in nextflow.config (the template documents both steps); docs/clusters/discovering-cluster-values.md is the command-by-command recipe for reading those values off a live cluster (and for checking the shipped ones — every nf-core-derived value here was stale until verified on the hardware). A cluster profile still composes with -c site.config below for anything site-local you want to layer on top (e.g. --dataset_size_gb, a shared image cache).

Tuning for your cluster (-c site.config)

[S75]. You should not edit nextflow.config to adapt the pipeline to your site. Instead copy the template conf/site.config.example, edit the values for your cluster, and pass it with -c (native Nextflow merges it over the defaults and the active profiles):

nextflow run main.nf -profile slurm,singularity \
-c /path/to/my-site.config \
--fastq_folder /scratch/me/run17 \
--forward_primer ... --reverse_primer ...

The template covers the knobs a site typically overrides: the slurm queue/account/concurrency, the resource ceiling (max_cpus/max_memory/max_time) and dataset/reference sizes, the container/conda cache directories (so the image or env is built once on shared scratch and reused by every node), and the environment-module names. One knob is new and has no CLI equivalent:

• --slurm_clusterOptions — free-form sbatch options appended verbatim to every job's submit line (e.g. '--qos=long --constraint=haswell', a reservation, …). Combined automatically with --slurm_account. Use it for cluster-specific submission flags that --slurm_queue / --slurm_account can't express.

Launching the run:

• The nextflow driver process is long-lived (it stays up submitting and reaping sbatch jobs for the whole run), so start it from a login node inside tmux or screen — not as an sbatch job itself, and not on a session that will disconnect:

  tmux new -s metabarcoding
  export NXF_OPTS='-Xms512m -Xmx4g'# cap the driver JVM heap
  nextflow run main.nf -profile slurm,conda \
  --fastq_folder /scratch/me/run17 \
  --slurm_queue normal \
  --threads 8 \
  -resume # reuse cached tasks after a stop

• -resume lets an interrupted run pick up where it left off, but it needs the per-task work/ directories to still exist. cleanup defaults to false ([S82]) precisely so work/ survives a successful run and -resume works across separate invocations — clean work/ by hand when you are done (see "Cleaning up after a run"). For throwaway runs that should auto-reclaim work/ on success, set cleanup = true in a -c override.

Notes:

• Run work/ on shared scratch visible to every compute node, and point NXF_CONDA_CACHEDIR at shared storage so the conda env is built once and reused.
• publishDir defaults to hard links (--publish_mode link). Switch to --publish_mode copy when --results_folder/--fastq_folder live on a different filesystem than work/.
• This path is not part of automated CI; it is exercised by a manual smoke test on the cluster ([S07]/[S08] in tests/COVERAGE.md).

How to test

# run the CI-tagged tests (pending tests excluded)
nf-test test --tag ci
# run a single test file
nf-test test tests/processes/part_a/merge_fastq_pairs.nf.test
# audit spec <-> test coverage
bash tests/coverage-gate.sh

To validate the whole Part A→B→C wiring without installing any tool or staging real data — handy as a first check on a new machine/cluster — run the pipeline in stub mode ([S85]); every process ships a stub: that produces placeholder outputs:

nextflow run main.nf -profile demo -stub-run # seconds, no vsearch/swarm/cutadapt/mumu

See tests/README.md for the TDD workflow and the pending-test convention.

Cleaning up after a run

cleanup defaults to false ([S82]), so per-task work/ directories are kept after a successful run (this is what lets -resume work across invocations). Clean them up by hand when you no longer need to resume — and each runner leaves some other state behind too:

# nextflow runs (in the project root or wherever you launched from)
rm -rf work/ .nextflow.log* .nextflow/
# published outputs land in --fastq_folder; remove them from the# bundled fixture dir after demo runs
rm -f tests/data/*.fas tests/data/*.qual tests/data/*.stats
# nf-test runs
rm -rf .nf-test/ .nf-test.log
# pytest cache
rm -rf .pytest_cache/ tests/python/__pycache__/

Releasing

The version number lives in two files that must be kept in sync: nextflow.config (manifest.version) and CITATION.cff (version:). When bumping, update both and also refresh CITATION.cff's date-released. CI enforces that the two versions match ([S77], tests/python/test_version_sync.py), so a half-finished bump fails the build rather than shipping.

Status

The authoritative, per-specification status lives in tests/COVERAGE.md (each [Sxx] ID → its test(s) → done / TODO / blocked). The CI suite (nf-test test --tag ci, plus bats and pytest) is green and the coverage gate maps every [Sxx] in SPECIFICATIONS to at least one test. Summary at the time of writing:

Implemented

• Part A — per-sample processing (merge pairs, trim primers, dereplicate, extract quality, local clustering with swarm), plus the experimental shadow pipeline for unmergeable pairs — opt-in via --recover_unmerged, off by default ([S04] / [S78])
• Part B — occurrence-table assembly (global dereplication + swarm clustering, cluster cleaving, chimera detection, substring-OTU merging, mumu post-clustering curation)
• Part C — taxonomic assignment (the stampa scatter-gather and the sintax shadow path), with optional majority-rule assignment ([S66])
• --input samplesheet input ([S70]) and the unified --outdir output layout ([S71])
• Fetch mode (--accession, [S97]) — a convenience path to download raw reads for an ENA/SRA bioproject/study accession via fastq-dl (for heavier use we recommend nf-core/fetchngs); see "Fetching reads from ENA/SRA" above
• HPC / slurm profile + conda / modules dependency profiles ([S07], [S08]) — see "Running on an HPC cluster (slurm)" below. The conda profile now resolves every tool — including mumu — from bioconda
• Container profiles — docker / podman / singularity / apptainer ([S08]), built on the fly from environment.yml via Seqera Wave; see "Running with containers" below
• Self-contained demo profile ([S76]) — runs Part A→B→C on a bundled dataset with one command; see "Try it (one command)" above
• Multi-layer test suite: nf-test (process + workflow), bats, pytest, with shellcheck / flake8 linting and a spec ↔ test coverage gate

Planned / not yet done

• Two-table occurrence output (--split-occurrence-table, [S15])
• Per-sample marker for unmerged-pair clusters ([S05], blocked on DECISIONS.md D02)
• Multiplexed-input subworkflow (demultiplexing is out of scope, [S16])
• Reference database auto-deduction from primers
• Force-rerun controls (see "Re-run policy" below)

Re-run policy

These inputs should trigger a complete or partial re-run when they change (currently a developer note — not yet enforced by the workflow):

• new versions of vsearch, cutadapt, swarm
• new versions of external python scripts
• new version of the reference database
• new set of fastq files
• user-requested global or partial re-run