Optical sim/reco and overlay updates

[mvicenzi]

This PR contains the required updates and fhicls to enable the reprocessing of light products for the SBN2025 spring productions (both MC and data). In particular, it includes fixes to the trigger/overlays procedure, as well as model improvements such as the use of time-dependent gain and pmt status databases.

Some highlights:

• added new MC2 tune (SPR + sim gain tune)
• added new MC3/4 tune (SPR + sim gain tune)
• noise in MC waveforms for overlays turned off
• introduced new PMT overlays module
• updated trigger shifting module to emit shifted trigger reference
• added DetectorClocksService configuration overrides for simulation
• introduced new simulation of timing delays from database in detsim overlays
• introduced new PMT channel status/voltage database for both detsim and stage0
• introduced new interface to the PMT gain database in detsim and stage0 for MC-overlays/data
• introduced new OpHit recalibrator module for data
• formally enforced same reconstruction algorithms for data and MC (including pedestal's)
• switched optical reconstruction entry point to local module
• re-tuned flash PE thresholds based on new PE scale
• added optional debug tree in optical detsim module
• updated FHiCLs for database use in MC-overlays/data (fixed/constant values still used for legacy MC)
• added run number overrides for calibration services
• added vertical track selection module
• added DetSim + Overlay stage reprocessing fhicl(s)
• added “MCStage0” reprocessing fhicl(s)
• added data “stage0” reprocessing fhicl(s)
• enfocing use of shared GENIE configuration from sbncode
• fixes on CRT hit time reconstruction for simulation and overlays

This PR requires three icarus_data updates:

• new SPR response files
• pmt_speareas_data.db to v1r2
• pmt_voltage_data.db to v1r3

These requirements are satisfied by icarus_data v10_20_04.

This PR depends on:

• Feature/gp fast gaus opt icarusalg#97
• Feature/gp overlay fix icarusalg#99
• Feature/gp plane crossers sbnalg#5
• Feature/gp overlay fix sbncode#653

(this PR includes developements previously discussed in #876 and #785 )

[Copilot]

Pull request overview

This PR updates ICARUS optical simulation/reconstruction and overlay workflows to support SBN2025 spring reprocessing (MC + data), including new/updated database-backed calibrations (PMT gain, PMT channel status, PMT timing), trigger/beam-gate handling changes, and a set of new/updated FHiCL “partial” configs for targeted reprocessing.

Changes:

• Introduces DB-backed PMT channel status and SPE-area (gain) calibration services, and wires them into stage0/detsim/simulation service stacks.
• Refactors trigger/beam-gate configuration and multiple trigger modules to use art::InputTag consistently, plus adds shifting/overlay workflow updates.
• Adds/updates many reprocessing/partial FHiCLs for detsim, overlay, and stage0/stage1, and adjusts build system dependencies accordingly.

Reviewed changes

Copilot reviewed 138 out of 138 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
scripts/manageDataRunDefinitions.py	Extends SAM-definition tooling (raw/compressedraw, extra dims, suffix).
icaruscode/Utilities/CMakeLists.txt	Adds new DataProductReader plugin build target.
icaruscode/TPC/SignalProcessing/RecoWire/SimTestPulse/CMakeLists.txt	Updates DetectorClocksService linkage.
icaruscode/Timing/timing_icarus.fcl	Adds run-number override option for PMT timing service.
icaruscode/Timing/PMTTimingCorrectionsProvider.h	Adds run override + DB tag accessors.
icaruscode/Timing/PMTTimingCorrectionsProvider.cxx	Implements override run-number DB queries.
icaruscode/Timing/PMTTimingCorrections.h	Extends interface with DB tag getters.
icaruscode/PMT/Trigger/triggerwindowdefs_icarus.fcl	Moves trigger-window definitions include usage.
icaruscode/PMT/Trigger/triggeremu_settings_icarus.fcl	Adds timing delay parameters + includes window defs.
icaruscode/PMT/Trigger/TriggerEfficiencyPlotsBase.h	Switches TriggerGatesTag to art::InputTag + helper.
icaruscode/PMT/Trigger/TriggerEfficiencyPlotsBase.cxx	Builds per-threshold tags via helper with validation.
icaruscode/PMT/Trigger/TriggerEfficiencyPlots_module.cc	Same tag handling updates as base class.
icaruscode/PMT/Trigger/trigger_icarus.fcl	Updates thresholds, gates, and patterns for Run2+ tuning.
icaruscode/PMT/Trigger/trigger_emulation_icarus.fcl	Updates trigger emulation workflow + baseline handling.
icaruscode/PMT/Trigger/SlidingWindowTriggerSimulation_module.cc	Uses art::InputTag and validated tag building.
icaruscode/PMT/Trigger/SlidingWindowTrigger_module.cc	Uses art::InputTag and validated tag building.
icaruscode/PMT/Trigger/MajorityTriggerSimulation_module.cc	Uses art::InputTag and validated tag building.
icaruscode/PMT/Trigger/LVDSgates_module.cc	Switches optional default gate tag to art::InputTag.
icaruscode/PMT/Trigger/FixBeamGateInfo_module.cc	Refactors beam-type selection + adds trace logging.
icaruscode/PMT/Trigger/DiscriminatePMTwaveforms_module.cc	Consumes baseline “parallel data product” when configured.
icaruscode/PMT/Trigger/DiscriminatedAdderSignal_module.cc	Simplifies optional interval parsing + improves error text.
icaruscode/PMT/Trigger/CMakeLists.txt	Updates link deps and target naming to newer conventions.
icaruscode/PMT/Trigger/Algorithms/WindowTopologyAlg.cxx	Fixes trace message formatting.
icaruscode/PMT/Status/PMTChannelStatusService_service.cc	New art service for PMT channel status DB.
icaruscode/PMT/Status/PMTChannelStatusProvider.h	New provider interface + configuration docs.
icaruscode/PMT/Status/PMTChannelStatusProvider.cxx	Implements DB reads, caching, override run number.
icaruscode/PMT/Status/PMTChannelStatus.h	New status/voltage provider interface.
icaruscode/PMT/Status/pmt_channel_status_icarus.fcl	Standard FHiCL config for PMT status service.
icaruscode/PMT/Status/IPMTChannelStatusService.h	Wrapper service interface for provider access.
icaruscode/PMT/Status/CMakeLists.txt	Builds PMT status library + service plugin.
icaruscode/PMT/SampledWaveformFunctionTool_tool.cc	Adds BiasRatio option plumbed into pulse function.
icaruscode/PMT/OpReco/ICARUSOpHitFinder_module.cc	Adds channel-status filtering + optional rise-time tool.
icaruscode/PMT/OpReco/ICARUSFlashAssAna_module.cc	Adds per-PMT integral branch in analysis tree.
icaruscode/PMT/OpReco/FlashFinder/ICARUSFlashFinder_module.cc	Modernizes module, uses PE() directly, SharedProducer.
icaruscode/PMT/OpReco/FlashFinder/FlashFinderFMWKInterface.h	Cleans includes, adds standard types.
icaruscode/PMT/OpReco/FlashFinder/FlashFinderFMWKInterface.cxx	Cleans includes, adds ServiceHandle include.
icaruscode/PMT/OpReco/FlashFinder/CMakeLists.txt	Simplifies library deps + plugin link list.
icaruscode/PMT/OpReco/fcl/tpcpmtbarycentermatch_config.fcl	Unifies TriggerDelay setting.
icaruscode/PMT/OpReco/fcl/icarus_spe.fcl	Adds Run2/3/4 SPE defaults + documentation.
icaruscode/PMT/OpReco/fcl/icarus_ophitfinder.fcl	Switches to ICARUSOpHitFinder + enables DB services.
icaruscode/PMT/OpReco/fcl/icarus_opana_modules.fcl	Retunes MC OpHit config to new SPE defaults.
icaruscode/PMT/OpReco/fcl/icarus_flashfinder.fcl	Removes PECalib configuration (uses OpHit PE).
icaruscode/PMT/OpReco/fcl/icarus_flashcalib.fcl	Removes obsolete flash calibration tables.
icaruscode/PMT/OpReco/fcl/icarus_flashalgo.fcl	Retunes PE thresholds to new PE scale.
icaruscode/PMT/OpReco/Algorithms/OpRecoFactoryStuff.h	Allows injecting optional rise-time calculator into algos.
icaruscode/PMT/OpReco/Algorithms/CMakeLists.txt	Updates dependencies for OpReco algorithm library.
icaruscode/PMT/opdetsim_pmt_icarus.fcl	Adds Run-dependent SimPMT configurations + no-noise variants.
icaruscode/PMT/CopyBeamTimePMTwaveforms_module.cc	Simplifies optional interval parsing.
icaruscode/PMT/CMakeLists.txt	Adds Status subdir + updates module/tool link deps.
icaruscode/PMT/Calibration/PhotonCalibratorFromDB.h	New DB-backed IPhotonCalibrator implementation.
icaruscode/PMT/Calibration/PhotonCalibratorFromDB.cxx	Implements DB reads + PE conversion for SPE-area table.
icaruscode/PMT/Calibration/ICARUSPhotonCalibratorServiceFromDB.h	New art service wrapper for DB calibrator provider.
icaruscode/PMT/Calibration/ICARUSPhotonCalibratorServiceFromDB_service.cc	Implements preBeginRun cache refresh.
icaruscode/PMT/Calibration/fcl/pmt-calibration.fcl	Removes legacy calibration job config.
icaruscode/PMT/Calibration/fcl/pmt_calibration_icarus.fcl	New consolidated calibration config incl. IPhotonCalibrator.
icaruscode/PMT/Calibration/fcl/ophit_recalibrator_icarus.fcl	New OpHitRecalibrator defaults + timing tag config.
icaruscode/PMT/Calibration/fcl/decodePMT_icarus_laser.fcl	Updates include to new calibration config file.
icaruscode/PMT/Calibration/fcl/debug_speAreas_db.fcl	Adds a debugging stage0 job for SPE DB configs.
icaruscode/PMT/Calibration/fcl/bkgphotons-calibration.fcl	Updates include to new calibration config file.
icaruscode/PMT/Calibration/CMakeLists.txt	Builds new DB calibrator library + service plugin + new modules.
icaruscode/PMT/Algorithms/SampledWaveformFunction.h	Adds integral + bias constant to sampled pulse function.
icaruscode/PMT/Algorithms/PhotoelectronPulseFunction.h	Extends base interface with integral + bias constant.
icaruscode/PMT/Algorithms/FastGaussianNoiseGeneratorAlg.h	Updates documentation + increases gauss adaptor resolution.
icaruscode/PMT/Algorithms/DiscretePhotoelectronPulse.h	Improves pulse-termination logic API.
icaruscode/PMT/Algorithms/DiscretePhotoelectronPulse.cxx	Implements “close-to-baseline” termination with persistence.
icaruscode/PMT/Algorithms/CMakeLists.txt	Links PMT algorithms with new calibration library.
icaruscode/Overlays/CMakeLists.txt	Adds RebuildShiftedTrigger plugin target.
icaruscode/Generators/CMakeLists.txt	Adds new generator-side selection module build target.
icaruscode/Generators/beamspilltimings.fcl	Delegates default spill timing config to SBN shared config.
icaruscode/Decode/OverlayProducts_module.cc	Temporarily truncates tail filling after data waveform end.
icaruscode/Decode/CMakeLists.txt	Adds OverlayPMTwaveforms plugin target; updates deps.
icaruscode/CRT/CRTUtils/CRTHitRecoAlg.h	Exposes isData() accessor.
icaruscode/CRT/CRTUtils/CRTHitRecoAlg.cc	Avoids trigger-offset subtraction for simulation.
icaruscode/CRT/CRTUtils/CMakeLists.txt	Updates DetectorClocksService linkage.
icaruscode/CRT/CRTSimHitProducer_module.cc	Cleans trigger handling; validates/ignores trigger tag by mode.
icaruscode/CRT/CRTDetSim_module.cc	Comment typo fixes in module header docs.
icaruscode/CRT/CMakeLists.txt	Updates DetectorClocksService linkage.
icaruscode/Analysis/trigger/CMakeLists.txt	Updates DetectorClocksService linkage.
fcl/utilities/dump_opflashes_icarus.fcl	New utility job to dump standard OpFlash products.
fcl/utilities/dump_opdetwaveforms_icarus.fcl	Uses simulation DetectorClocksService config in utility.
fcl/SkipTesting.txt	Adds override_services_for_simulation.fcl to skip list.
fcl/services/services_icarus_simulation.fcl	Wires timing/status/gain services + simulation clocks override.
fcl/services/override_services_for_simulation.fcl	New reusable services override for simulation workflows.
fcl/reco/Stage1/mc/stage1_run2_icarus_MC.fcl	Includes simulation services override.
fcl/reco/Stage0/partial/stage0_run4_opdetonly_icarus_data_reprocessing.fcl	New Run4 optical-only data reprocessing config.
fcl/reco/Stage0/partial/stage0_run2_opdetonly_icarus_overlay_reprocessing.fcl	New optical-only overlay stage0 reprocessing config.
fcl/reco/Stage0/partial/stage0_run2_opdetonly_icarus_data_reprocessing.fcl	New Run2 optical-only data reprocessing config.
fcl/reco/Stage0/partial/CMakeLists.txt	Installs new partial stage0 configs.
fcl/reco/Stage0/mc/stage0_run2_wc_icarus_mc.fcl	Includes override + tunes optical reconstruction to Run2.
fcl/reco/Stage0/mc/stage0_run2_wc_icarus_mc_notriggersim.fcl	Restores non-shifted inputs for no-trigger-sim workflows.
fcl/reco/Stage0/mc/stage0_run2_icarus_mc.fcl	Includes override + enables fast cloning (no SelectEvents).
fcl/reco/Stage0/mc/stage0_run2_icarus_mc_notriggersim.fcl	Restores non-shifted inputs for no-trigger-sim workflows.
fcl/reco/Stage0/data/partial/decodePMT_icarus.fcl	Adds channel status service to PMT decode services.
fcl/reco/Stage0/CMakeLists.txt	Adds new partial stage0 configs subdirectory.
fcl/reco/Definitions/stage0_icarus_mc_defs.fcl	Splits corrected/uncorrected ophit path; overlay PMT sequence.
fcl/reco/Definitions/stage0_icarus_driver_common.fcl	Adds photon calibrator + channel status services.
fcl/reco/Definitions/stage0_icarus_defs.fcl	Includes new calibration + recalibrator configs.
fcl/reco/Definitions/enable_overlay_sp.fcl	Switches to overlay PMT path + timing correction override.
fcl/overlays/partial/overlay_opdetonly_waveforms_reprocessing.fcl	New overlay-only waveform reprocessing job.
fcl/overlays/partial/CMakeLists.txt	Installs new partial overlay configs.
fcl/overlays/overlay_waveforms.fcl	Replaces OverlayProducts PMT overlay with OverlayPMTwaveforms module.
fcl/overlays/CMakeLists.txt	Adds partial overlay configs subdirectory.
fcl/gen/numi/genie_icarus_numioffaxis.fcl	Switches to SBN shared NuMI GENIE base + volume/log updates.
fcl/gen/genie/genie_icarus_bnb.fcl	Switches to SBN shared BNB GENIE base + volume/log updates.
fcl/g4/PDFastSim_icarus.fcl	Updates SimulationLabel and minor formatting.
fcl/detsim/partial/detsim_opdetonly_icarus.fcl	New optical-only detsim job config.
fcl/detsim/partial/detsim_opdetonly_icarus_overlay_reprocessing_run4.fcl	New Run3/4 overlay optical detsim reprocessing config.
fcl/detsim/partial/CMakeLists.txt	Installs new partial detsim configs.
fcl/detsim/detsim_2d_icarus.fcl	Explicitly disables DB effects for non-overlay MC opdaq.
fcl/detsim/detsim_2d_icarus_refactored.fcl	Explicitly disables DB effects for non-overlay MC opdaq.
fcl/detsim/detsim_2d_icarus_refactored_yzsim.fcl	Explicitly disables DB effects for non-overlay MC opdaq.
fcl/detsim/detsim_2d_icarus_refactored_yzsim_Run4.fcl	Selects Run3/4 optical tune; keeps DB effects off for non-overlay.
fcl/detsim/detsim_2d_icarus_refactored_yzsim_overlay.fcl	Enables DB effects and no-noise for overlay opdaq.
fcl/detsim/detsim_2d_icarus_refactored_yzsim_overlay_Run4.fcl	Run3/4 overlay tune with DB effects + no-noise.
fcl/detsim/detsim_2d_icarus_refactored_yzsim_notrigger.fcl	Explicitly disables DB effects for non-overlay MC opdaq.
fcl/detsim/detsim_2d_icarus_refactored_yzsim_notrigger_overlay.fcl	Enables DB effects + no-noise for overlay notrigger.
fcl/detsim/detsim_2d_icarus_refactored_Run4.fcl	New Run3/4 optical tune config (non-overlay).
fcl/detsim/detsim_2d_icarus_refactored_overlay.fcl	Enables DB effects + no-noise for overlay opdaq.
fcl/detsim/detsim_2d_icarus_refactored_overlay_Run4.fcl	New Run3/4 overlay tune config (DB + no-noise).
fcl/detsim/detsim_1d_icarus.fcl	Explicitly disables DB effects for non-overlay MC opdaq.
fcl/detsim/commissioning/run4759like_detsim_icarus.fcl	Switches includes to pmtsimulation_icarus tuning table.
fcl/detsim/commissioning/run4642like_detsim_icarus.fcl	Switches includes to pmtsimulation_icarus tuning table.
fcl/detsim/CMakeLists.txt	Adds partial detsim configs subdirectory.
fcl/decoder/decoder_run2_icarus.fcl	Updates inline comments; clarifies PMT discrimination purpose.
fcl/caf/cafmakerjob_icarus.fcl	Includes simulation services override.
CMakeLists.txt	Adds sbnalg dependency to build.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[PetrilloAtWork]

It appears all the comments are marked as "resolved" so we are good to go.

[PetrilloAtWork]

@jas1005: it appears there are merge conflicts. Please contact us when you are ready to merge so that we can solve them on the spot.

[francescopoppi]

tracking everything is a titanic work. Kudos to Gianluca and Matteo for this incredibly huge work. I reviewed output stage1 of several CRT related variables and to my understanding it works. I have to admit, I asked Claude to provide me a summary of this PR. I do not see showstoppers. If it compiles, and this itself iwould be a great success, the result should be the intended one.