Merge upstream master into GhentAnalysis/master (0.3 release cycle) by JulesVandenbroeck · Pull Request #120 · GhentAnalysis/columnflow

JulesVandenbroeck · 2026-06-24T13:03:31Z

Overview

This PR tracks all changes in columnflow/columnflow:master
that are not yet present in GhentAnalysis/master. It is a documentation-only / dummy PR —
its purpose is to give an overview of what an eventual merge from upstream would bring in.

The base branch (GhentAnalysis/master_merge_from_upstream) is identical to GhentAnalysis/master
at the time of this PR's creation.

New Features

Framework & Core

Refactoring for 0.3 release (Refactoring for 0.3 release columnflow/columnflow#628) — major restructuring of the core framework
Accelerate loading time and imports (Accelerate loading time and imports columnflow/columnflow#737) — faster startup by deferring heavy imports
Make default_remote_claw_sandbox configurable via law.cfg — remote sandbox now settable per-project
Add dfs lookup pattern negation — negate patterns in DFS column lookups
Add tmp dir checks, add cf_setup_post_install hook — post-install hook for custom setup steps
Allow skipping parts of post setup — fine-grained control over setup steps
Improve python version handling in setup — more robust Python version detection
Report rucio access (Report access of central LFNs to rucio bookkeeping server columnflow/columnflow#803) — log Rucio site access during dataset queries
Add req helpers to mixins (Add same-task requirement helpers to mixins. columnflow/columnflow#738) — convenience methods for adding task requirements in mixins
Port parameters to WrapperTask's (Port parameters to WrapperTask's columnflow/columnflow#784) — wrapper tasks now expose the same parameters as their wrapped tasks
Control row group merging in MergeReducedEvents — configurable row-group size in Parquet output
Make merging chunk size configurable (Make merging chunk size configurable. columnflow/columnflow#707) — tune memory/IO trade-off in merge steps
Improve reduced events merging — smarter chunk layout in merged Parquet files
Add transfer_produced_columns util — helper to copy produced columns between events arrays
Hook column union — column union now participates in hook callbacks
Forward remote claw sandbox — remote sandbox propagated through task dependency chain

Producers & Calibrators

Generalize normalization weight producer (Generalize normalization weight producer. columnflow/columnflow#718) — handles more generic dataset configurations
Extend DY weight application to use btag multiplicity (Extend dy weight application to use btag multiplicity. columnflow/columnflow#739) — DY weights now depend on b-tag count
Make dy_weight producer compatible with both central and custom DY sf (Update dy_weight producer columnflow/columnflow#770) — unified interface for official/custom scale factors
Adapt DY weight producer for custom weights (adapting dy weight producer for custom weights columnflow/columnflow#724) — pass custom weights directly to producer
Make jet collection used in DY weights more flexible (dy_weights columnflow/columnflow#696) — configurable jet collection for DY reweighting
Adding b-tag producer for working point based SFs (Adding b-tag producer for working point based SFs columnflow/columnflow#777) — new dedicated b-tag WP scale-factor producer
Merge WP bins (Allow merging wp bins in btag_wp_weights producer columnflow/columnflow#792) — merge b-tag working-point bins in scale-factor evaluation
Add btag count selector (Add btag count selector. columnflow/columnflow#775) — ready-made selector for b-tag multiplicity cuts
Enable support for CMS electron weight producer v5 — updated to CMS EGM v5 correctionlib schema
Refactor and fix MET phi calibration (Fix met phi calibrator columnflow/columnflow#725) — cleaner MET-phi corrector with updated format
Update met_phi Calibrator to new format (update met_phi Calibrator to new format columnflow/columnflow#719)
Add muon_sr calibrator (Add muon scale and resolution calibrator. columnflow/columnflow#754) — new calibrator for muon scale and resolution corrections
[cms] Refactor egamma calibrators (Fix and refactor egamma calibrators. columnflow/columnflow#745) — cleaner EGamma calibration interface
Efficiency information from muon correctionlib files (efficiency information from muon correctionlib files columnflow/columnflow#791) — expose muon efficiency from correctionlib
Remove year from intrinsic btag weight names (Remove year from intrinsic btag weight names. columnflow/columnflow#726) — year-agnostic naming convention
Externalize coffea from_root handling — coffea ROOT-file reading decoupled from core IO
Update sandboxes, avoid dask_awkward in IO (Update sandboxes, avoid dask_awkward in IO columnflow/columnflow#735) — columnar sandbox no longer requires dask-awkward
Add option for hook to update dataset_selection_stats in norm weight producer — hook-based stats update
Allow evaluating multiple working points with single electron_weights Producer (allow evaluating multiple working points with single electron_weights Producer columnflow/columnflow#694)
Streamline chunked IO (Streamline chunked io. columnflow/columnflow#789) — unified chunk reader/writer API
Add simple column selection to UniteColumns — filter columns when uniting datasets
Extend tes versions — support for newer tau energy scale correction versions
Add t->w->tau children in gen_top_lookup — tau daughters now tracked in top-quark gen lookup
Implement data_per_era tag to jec config (Fix/jec 2023 columnflow/columnflow#700) — JEC keys now selectable per data-taking era
Allow strings in load_correction_set — pass string file paths directly to correctionlib loader

Selectors & Categorization

Improve categorizer calls (Improve categorizer calls in category_ids producer. columnflow/columnflow#702) — cleaner internal dispatch for categorizer TAFs
Add parent_mode flag to create_category_combinations, add change tracking (Add parent_mode flag to create_category_combinations, add change tracking. columnflow/columnflow#736)
Inherit selector shifts in cf_default reducer (Inherit selector shifts in cf_default reducer. columnflow/columnflow#761) — shifts declared in selector automatically available in reducer
Sort configs by ids for multi-config representation (Sort configs by ids for the multi config representation columnflow/columnflow#713) — deterministic ordering in multi-config setups
Check and remove overlapping processes (check and remove overlaping processes columnflow/columnflow#712) — automatic deduplication of overlapping process definitions
Pad with nominal if shift source missing in config (pad with nominal if shift source missing in config columnflow/columnflow#715) — graceful fallback for missing shift sources

Histogramming & Plotting

Multi-variable plots (Multi-variable plots columnflow/columnflow#769) — plot multiple variables in a single task invocation
Histogram user multiconfig (Feature/histogram user multiconfig columnflow/columnflow#709) — histogram tasks now support multiple config objects simultaneously
Update hist axis labels during histogram merging (update hist axis labels during histogram merging columnflow/columnflow#705) — axis metadata preserved across histogram merges
Add category id cache in config_util (Add category id cache. columnflow/columnflow#787) — faster category lookups in large configs
Add subplots_cfg in plot_all (add subplots_cfg in plot_all columnflow/columnflow#742) — subplot configuration passed through plot_all loops
Add plot function for efficiency plots (plotting enhancements columnflow/columnflow#723) — dedicated helper for efficiency curve plotting
Allow defining initial values for categorical hist axes — pre-populate string/int categorical axes
Improve leaf category handling in histograms (Improve leaf category handling in histograms columnflow/columnflow#783) — correct aggregation over leaf categories
Add some customization hooks for merge histograms — pre/post-merge hooks for histogram processing
Remove hard-coded nominal shift source usage (Remove hard-coded nominal shift source usage. columnflow/columnflow#762) — nominal shift sourced from config, not hardcoded
More flexibility in plot_all loop — pass extra kwargs through plot_all
Make plotting faster using non-interactive backend — use Agg backend to avoid GUI overhead
Combinatorics protection during histogram filling (Combinatorics protection during histogram filling. columnflow/columnflow#796) — guard against event x object combinatoric explosions
Reduce processes in plot tasks to those present in datasets — no phantom processes in legend
Use dict-of-keys sparse matrices for vectorized lookups (memory intensive sparse matrix in normalization_weights producer columnflow/columnflow#763) — major performance improvement in SF lookups
Add pilot option to MergeShiftedHistograms — run merge in pilot/test mode
Warn about flow content for FlowStrategy.move — alert when under/overflow is non-zero after moving
Fix data uncertainties for density (fix data uncertainties for density columnflow/columnflow#773) — correct uncertainty propagation for density-normalized histograms
Forward known values to hist hooks (Forward known values to hist hooks. columnflow/columnflow#727) — hook callbacks receive pre-computed axis values
Improve proxy handling in remote jobs (docs: add Lara813 as a contributor for code columnflow/columnflow#765)

Inference / Datacards

Sequentialize and optimize datacard writing (Sequentialize and optimize datacard writing. columnflow/columnflow#728) — faster, memory-efficient datacard production
Refactor datacard parameter transformations (Refactor datacard parameter transformations. columnflow/columnflow#732) — cleaner transformation API
Add flip_(smaller|larger)_if_one_sided transformations (Add transformations for flipping rate effects. columnflow/columnflow#734) — new transformations for one-sided systematics
Add ParameterTransformation for ratifying + envelope if one-sided (add ParameterTransformation for ratifying + envelope if one-sided columnflow/columnflow#731)
Allow removing negative contributions per process (allow removing negative contributions per process columnflow/columnflow#730) — option to zero-out negative process yields
Allow block modification before writing datacards — user hook before final datacard output
Add rounding warning to datacard for small effects — warns when systematics round to zero
Make datacard writer class configurable in task — swap in custom writer subclass via config
Merge workflow reqs of different variables in CreateDatacards (merge workflow reqs of different variables in CreateDatacards columnflow/columnflow#689) — single merged requirement for all variable branches
Inference model caching (inference model caching columnflow/columnflow#714) — avoid re-building inference models on each task invocation
Asymmetric systematic uncertainties (Fix/asymmetric syst unc columnflow/columnflow#710) — support for asymmetric systematics in datacards
Allow multiple processes per dataset in datacard writer (allow multiple processes per dataset in datacard writer columnflow/columnflow#733)
Move empty-process skipping from histogram loading to datacard writing

Tasks & Infrastructure

Add BundleExternalFilesWrapper — wrapper task for bundling external files in parallel
Persistent local files of BundleExternalFiles (Persistent local files of BundleExternalFiles. columnflow/columnflow#752) — cached external files survive task re-runs
Add --merge-columns option to cf_inspect tool — inspect merged-column content in cf_inspect
Simplify requiring producers (Simplify requiring producers. columnflow/columnflow#756) — shorter API for declaring producer dependencies
Rename max-runtime to {htcondor,slurm}-runtime (Rename max-runtime -> {htcondor,slurm}-runtime. columnflow/columnflow#755) — runtime flag split per scheduler
Add variable_repr to control paths (Add variabble_repr to control paths. columnflow/columnflow#751) — custom string representation for variable objects
Add gen_dy_lookup and gen_higgs_lookup — generator-level Drell-Yan and Higgs particle lookups
Refactor generator-level top and top decay product lookup (Refactor generator-level top and top decay product lookup columnflow/columnflow#741)
Add objects for interacting with CMS CAT meta data (Add objects for interacting with CMS CAT meta data. columnflow/columnflow#740) — interface to CMS CAT POG JSON metadata
Allow multiple nano inputs in first tasks (Allow multiple nano inputs in first tasks. columnflow/columnflow#788) — GetDatasetLFNs and friends accept multiple NanoAOD inputs
Add embed_with_mask helper — embed sub-array values back into parent array using a mask
Apply jet_mask to events (Apply jet_mask to events in btag_wp_weights. columnflow/columnflow#780) — jet mask correctly applied before further processing
Add confirm message to cf_remove_tmp — interactive confirmation before removing tmp files
Allow patterns in get_shifts_from_sources — brace/glob patterns in shift source lookup
Accept lists in ExternalFile subpaths / Allow single subpath in ExternalFile — flexible path specification
Add ML model task pinning (Add ml model task pinning. columnflow/columnflow#703) — pin specific ML model versions in task parameters
Add sourcing of CMS setup to ensure scram is available (added sourcing of cms setup to ensure scram is available columnflow/columnflow#699)
Optionally bypass branch-level plot requirements (Accelerate branch-level plot requirements. columnflow/columnflow#716)
Add option to skip auto categories in track_category_changes
Pass-through workflow requirements in CreateHistograms (Pass-through workflow requirements in CreateHistograms. columnflow/columnflow#757)

Bug Fixes

Histograms & Plots

Hotfix rewriting of outputs in MergeHistograms
Hotfix shift id-to-name conversion in histogram post-processing
Hotfix empty histogram issue in inference base task (two separate fixes)
Hotfix nominal shift in PlotShiftedVariablesPerShift1D
Hotfix hist validation in datacard writer
Hotfix sci-style axis label notation in plots / fix horizontal y-offset label position
Hotfix single shift selection in plotting
Hotfix save_div in plot scale factor
Hotfix multi-axis filling with discrete variables
Bug fix in fill_hist function (bug fix in fill_hist function columnflow/columnflow#768)
Fix variance of fake data in datacard writer
Fix plotting process shift map (Fix/plotting process shift map columnflow/columnflow#760)
Hotfix bad import in plot utils
Avoid using flat_np_view for value assignment (Avoid using flat_np_view for value assignment. columnflow/columnflow#759)
Use safe concatenation (Use safe concatenation. columnflow/columnflow#758)
Hotfix multi-config lookup via patterns
Hotfix process object selection for multi-config datacards
Hotfix variable shape type in combine datacard writer
Fix skipping data in CreateDatacards
Hotfix datacard writing when variables are missing in config
Hotfix parameter group cleaning in inference model

Producers & Calibrators

Hotfix kwargs for array function super calls
Hotfix super() calls in all task array functions
Hotfix TAF class attribute inheritance and TAF instance method defaults
Hotfix calibrator/producer requirement handling
Hotfix btag_wp_weights producer: clamp pt and eta efficiency map axes
Hotfix ExternalFile dataclass: drop None values
Hotfix sorted_ak_to_root helper for nested arrays
Hotfix CMS jet veto map usage; correct bit masks (correct bit masks jet veto map columnflow/columnflow#778)
Hotfix CMS muon calibrator: remove outdated rnd_gen argument
Hotfix abs eta in CMS muon weight producer
Hotfix CMS tau energy scale variations
[cms] Hotfix varied energy errors in EGM calibration
[cms] Hotfix egamma calibrator: use same random numbers for all smearing variations
[cms] Hotfix tau energy calibration: skip e-fake mask
Hotfix TEC calibrator: add back charge column
Hotfix inclusive dataset attribute and lookup in norm weight producer
Hotfix norm weight logging
Hotfix combined jets calibrator
Hotfix validation check in stitched normalization weight production
Hotfix missing xsecs for stitched weight producer
Hotfix nbtags variable in DY weight producer
Fix handling of non_zero_mask in murf_envelope (fix handling of non_zero_mask in murf_envelope columnflow/columnflow#704)
Fix JER application on JEC variations (Fix to jer application on jec variations columnflow/columnflow#665)
Add exception when era aux is missing (Fix/jec 2023 columnflow/columnflow#700)
Fix scope issue in seed producer
Hotfix depth limit of gen particles
Hotfix saving of columns in gen_particle lookups
Hotfix higgs gen lookup: consider effective gluon/photon decays
Hotfix typo in gen_top lookup
Skip string columns in finiteness checks (Producing string columns columnflow/columnflow#743)

Tasks & Workflow

Hotfix task_namespace of Wrapper tasks
Hotfix producer group resolution in ProduceColumnsWrapper; fix brace expansion
Hotfix config object resolution and order (deterministic and group lookup)
Hotfix process selection when only datasets are given
Hotfix default version injection into tasks with same family
Hotfix version resolution, pinning, and lookup
Hotfix forwarding of known_shifts for instance caching
Hotfix required producers/calibrators for workflows
Hotfix attributes added by TAF decorators
Hotfix: allow brace patterns in TAF shifts
Fix duplicate producers and requirements
Fix workflow parameter passthrough in pilot mode (Fix workflow parameter passthrough in pilot mode. columnflow/columnflow#785)
Fix bundle unpacking (Fix ExternalFiles bundle unpacking. columnflow/columnflow#786)
Fix total size logged by MergeReductionStats
Fix reduction chunk size control
Fix task key lookup (Fix task pinning columnflow/columnflow#697)
Hotfix requirement order for consistent output removal
Hotfix consistent branch reqs between MergeReducedEvents and MergeSelectionStats
Raise explicit error in reduction on option type masks

IO & Environment

Hotfix ChunkedParquetReader
Multiple fixes regarding empty files or almost-empty chunks (mutliple fixes regarding empty files or (almost) empty chunks columnflow/columnflow#750)
Hotfix reduction to skip empty chunks
Fix mamba setup
Hotfix htcondor/conda software setup in remote jobs
Fix missing datasets in MultiConfig
Hotfix CAT metadata update check for missing POG dirs
Hotfix category flattening
Correct JSON file extension for stats file
Do not consider empty axis as "missing" (histogram axis fix)
Fix opening root files in cf_inspect / after coffea update (Fix cf_inspect script after coffea update. columnflow/columnflow#753)
Log broken parquet file paths
Fix warning message in BundleExternalFiles task
Hotfix repo bundling: add missing user config
Failsafe setting of htcondor requested memory
Apply blinding threshold before process scaling
Hotfix electron weight producer with nested working points

Small Changes

Dependency & Submodule Updates

Multiple law, order, scinum, and boost-histogram submodule/version updates
Update default CMSSW versions in CMSSW sandboxes

Code Quality & Cleanup

Multiple typo fixes across the codebase
Minor code cleanups and consistency changes
Cleanup of e/mu id producers; remove unneeded columns in CMS TEC calibrator
Consistent handling of kwargs in teardown functions
Improve treepath detection in cf_inspect; optimize UniteColumns ROOT compression
Improve tmp file check and flag file handling in venv setup
Use return code in cf_remove_tmp; add local directory check
[cms] Add note on TEC-to-MET propagation
Store jec level as attribute on jec correctors
Bump version in __version__ file
Improve readability of make_jme_keys

Logging & Verbosity

More verbose errors for parquet metadata failures, norm weight missing cross sections, memory issues during filling, and broken histogram files
Warn about flow content for FlowStrategy.move
[cms] Update log in CheckCATUpdates task; add URL to log

Documentation

docs: add LennertGriesing, Lara813, aalvesan, Bogdan-Wiederspan, LuSchaller as contributors

Tests

Adjust unit tests for new behaviour and deterministic config object ordering

This PR was auto-generated to track upstream changes. It is not intended to be merged directly.

* update met_phi Calibrator to new format * use npvsGood * add npvsGood to uses as well... * Minor adjustments, apply mask to all inputs. --------- Co-authored-by: Mathis Frahm <mathis.frahm@uni-hamburg.de> Co-authored-by: Marcel R. <github.riga@icloud.com>

* Generalize normalization weight producer. * Add pull warning. * Add per-dataset weight norm. * Update. * Optionally log brs. * Improve combinatoric treatment, fix single br calculation. * Helper to fill weight table. * Minor adjustments before review.

Co-authored-by: Mathis Frahm <mathis.frahm@uni-hamburg.de> Co-authored-by: Marcel Rieger <riga@users.noreply.github.com>

* docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] --------- Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

mafrahm and others added 30 commits July 29, 2025 14:05

fix missing datasets in MultiConfig

0bdff9f

Update law.

f3af6f2

Add local directory check to cf_remove_tmp.

b9cfaac

Hotfix typo.

6600f0e

Update law.

d3689a3

Hotfix inclusive dataset lookup in norm producer.

0619ee3

Update boost-histogram version.

13d6523

Hotfix norm weight logging.

c586eb9

Revert boost-histogram update.

2c77fb1

Hotfix combined jets calibrator.

650b4f6

Hotfix inclusive dataset attribute in norm weight producer.

433803d

make plotting faster using non-interactive backend

9be2072

[cms] Make datacard writer class configurable in task.

433ba76

Hotfix typo in norm weight producer.

36eb11a

Adapt norm weight producer to more generic cases.

e170f92

Hotfix cf_inspect for root files.

23e992d

Optimize UniteColumns compression for root files.

4407e52

Improve treepath detection in cf_insepct.

17b3968

Hotfix fowarding of known_shifts for instance caching.

1c04755

Minor consistency change.

87db487

Fix brace expansion in ProduceColumnsWrapper.

9bbdbbe

adapting dy weight producer for custom weights (#724)

d83e07b

Consistent handling of kwargs in teardown functions.

fc9553d

Hotfix TAF instance method defaults.

d354824

Refactor and fix met phi calibration.

17d2faf

add plot function for efficeincy plots (#723)

49c8a38

Co-authored-by: Mathis Frahm <mathis.frahm@uni-hamburg.de> Co-authored-by: Marcel Rieger <riga@users.noreply.github.com>

Merge branch 'master' into fix/met_phi_calib

5fbf5d0

Hotfix parameter group cleaning in inference model.

f0cc020

riga and others added 29 commits April 13, 2026 11:28

Hotfix calibrator/producer requirement handling.

a7727ff

Failsafe setting of htcondor requested memory.

3043e58

Fix opening root files in cf_inspect tool.

f33fa03

Update law.

8d7e94d

Hotfix sci-style axis setting for plots.

6d3af1d

More verbose error in case of memory issues during histogram filling.

9f03883

Hotfix cms muon calibrator, remove outdated rnd_gen argument.

8616d27

Update law.

10a065a

docs: add LennertGriesing as a contributor for code (#800)

e07ed50

* docs: update README.md [skip ci] * docs: update .all-contributorsrc [skip ci] --------- Co-authored-by: allcontributors[bot] <46447321+allcontributors[bot]@users.noreply.github.com>

Hotfix scientific axis notation in plots.

0b6fa62

Hotfix sorted_ak_to_root helper for nested arrays.

82b4e07

Update default cmssw versions in cmssw sandboxes.

e772a62

Update law.

df117dc

Update law.

9ab4c98

Hotfix super() calls in all task array functions.

0cdb98f

Hotfix kwargs for array function super calls.

7d257f5

Allow defining initial values for categorical hist axes.

66ebea7

Hotfix empty histogram issue in inference base task.

09c5d81

Hotfix empty histogram issue in inference base task.

c5996f2

Typo in jer calibrator setup.

56e75a4

Move empty process skipping from histogram loading to datacard writing.

0f22d5a

Adapt tests.

37efc16

Hotfix datacard writing in case variables are missing in config.

f7df3a2

Report rucio access. (#803)

fe80be4

Hotfix shift id to name conversion in hist post proc.

4068b02

Adjust tests.

f4544ff

Update law.

fe8cb13

Law update preparations.

b796e4a

Hotfix rewriting of outputs in MergeHistograms.

7ced3f1

JulesVandenbroeck closed this Jun 24, 2026

Navigation Menu

Merge upstream master into GhentAnalysis/master (0.3 release cycle) - #120

Merge upstream master into GhentAnalysis/master (0.3 release cycle)#120
JulesVandenbroeck wants to merge 306 commits into
GhentAnalysis:GhentAnalysis/master_merge_from_upstreamfrom
columnflow:master

JulesVandenbroeck commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

JulesVandenbroeck commented Jun 24, 2026

Overview

New Features

Framework & Core

Producers & Calibrators

Selectors & Categorization

Histogramming & Plotting

Inference / Datacards

Tasks & Infrastructure

Bug Fixes

Histograms & Plots

Producers & Calibrators

Tasks & Workflow

IO & Environment

Small Changes

Dependency & Submodule Updates

Code Quality & Cleanup

Logging & Verbosity

Documentation

Tests

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants