v2025.7.0

[ikrommyd]

Important announcement

This release changes the default behavior of coffea . We are now focusing on doing analysis with the newly developed "virtual arrays" of awkward as the main backend.
For more information on virtual arrays, see this talk at PyHEP.dev 2025.
For examples of virtual array usage, see the following example repositories:
https://github.com/ikrommyd/coffea-virtual-array-tests
https://github.com/ikrommyd/coffea-virtual-array-demo
https://github.com/iris-hep/calver-coffea-agc-demo/blob/2025_IRISHEP_Training/agc-coffea-2025-virtual-arrays-and-executors.ipynb
https://github.com/ikrommyd/virtual-array-agc

The default behavior of NanoEventsFactory.from_root() has changed. It now reads the input root file using virtual arrays by default.
The backend choice is controlled by the mode argument of the method which can be set to "eager", "virtual", or "dask".
The new default is "virtual" while the delayed argument has been removed.
The old delayed=True is now equivalent to mode="dask". The old delayed=False is now equivalent to mode="eager".

At the same time, the coffea 0.7 processors and executors have been revived and analysis can be done using coffea 0.7-like syntax

from coffea.processor import ProcessorABC, Runner, DaskExecutor

class MyProcessor(ProcessorABC):
    def process(self, events):
        ...
        
    def postprocess(self, accumulator):
        ...

run = Runner(
    DaskExecutor(client=client, compression=None),
    chunksize=250_000,
    skipbadfiles=True,
    schema=NanoAODSchema,
    savemetrics=True
)

out, report = run(fileset, processor_instance=MyProcessor())

Analyses still using coffea 0.7 can and should seamlessly transition to this new release.

If you still want to use the dask interface (create task graphs), you should specify mode="dask" to NanoEventsFactory.from_root() when working on single file.
For scaling, you can still use the dataset_tools like the following

from coffea.dataset_tools import apply_to_fileset

apply_to_fileset(MyProcessor(), fileset, uproot_options={"allow_read_errors_with_report": True})

It is recommended to convert all analyses to use the new virtual arrays feature of awkward2 and not stick with packages that are unmaintained for 3 years (coffea 0.7 which still uses awkward1 ).
Please reach out for any help and to report problems.

New features

• feat: EDM4HEPSchema and Newstyle FCCSchema by @prayagyadav in feat: EDM4HEPSchema and Newstyle FCCSchema #1245
• feat: add virtual arrays by @pfackeldey in feat: add virtual arrays #1277
• feat: bring back iterative, futures and dask executors by @ikrommyd in feat: bring back iterative, futures and dask executors #1323
• feat: 0.7 style processor/executor model using ak2 virtual arrays. by @lgray in feat: 0.7 style processor/executor model using ak2 virtual arrays. #1309
• feat: bring back parsl executor by @ikrommyd in feat: bring back parsl executor #1325
• feat: add @original_array attr to events in virtual mode by @ikrommyd in feat: add @original_array attr to events in virtual mode #1327
• feat: make column_accumulator support awkward arrays and add accumulator tests by @ikrommyd in feat: make column_accumulator support awkward arrays and add accumulator tests #1352
• feat: taskvine executor for new coffea by @btovar in feat: taskvine executor for new coffea #1360
• feat: systematics handling for dask mode by @lgray in feat: systematics handling for dask mode #786
• feat: make max_chunks return the first N chunks per dataset (not per file per dataset like it is now) by @ikrommyd in feat: make max_chunks return the first N chunks per dataset (not per file per dataset like it is now) #1359

Bug-fixes and performance

• fix: properly support older numba/numpy mixtures by @lgray in fix: properly support older numba/numpy mixtures #1298
• fix: do not error or return None when calling min/max over length zero chunks in weight statistics, return infinities instead by @ikrommyd in fix: do not error or return None when calling min/max over length zero chunks in weight statistics, return infinities instead #1328
• fix: skip OSErrors when skipping bad files using executors by @ikrommyd in fix: skip OSErrors when skipping bad files using executors #1333
• fix: executor's preprocess requires treename as input argument when there should be a default by @ikrommyd in fix: executor's preprocess requires treename as input argument when there should be a default #1334
• fix: make NanoAODSchema the default in exectors for consistency with apply_to_fileset by @ikrommyd in fix: make NanoAODSchema the default in exectors for consistency with apply_to_fileset #1335
• fix: use awkward for min and max in Weights to avoid inconsistencies between eager/virtual and dask mode by @ikrommyd in fix: use awkward for min and max in Weights to avoid inconsistencies between eager/virtual and dask mode #1337
• fix: make nanoevents properly copiable and do not store the @original_array attribute as that will get copied by @ikrommyd in fix: make nanoevents properly copiable and do not store the @original_array attribute as that will get copied #1346
• perf: use numpy only in eager weight statistics by @ikrommyd in perf: use numpy only in eager weight statistics #1351
• fix: print the original processor error with the "failed processing file" exception by @ikrommyd in fix: print the original processor error with the "failed processing file" exception #1353
• fix: _lazywhere was removed from scipy, use apply_where from scipy._lib.array_api_extra by @lgray in fix: _lazywhere was removed from scipy, use apply_where from scipy._lib.array_api_extra #1356
• fix: make offsets start at zero forListOffsetArray coming from uproot (fix physlite entry start problem) by @ikrommyd in fix: make offsets start at zero forListOffsetArray coming from uproot (fix physlite entry start problem) #1363
• fix: clarify function names in systematics by @lgray in fix: clarify function names in systematics #1366
• fix: Fix bugs in CorrectedMETFactory.build in coffea 202x by @ikrommyd in fix: Fix bugs in CorrectedMETFactory.build in coffea 202x #1342
• fix: error when delayed kwarg is used in nanoevents by @ikrommyd in fix: error when delayed kwarg is used in nanoevents #1367

Other

• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1294
• ci(also docs): enforce constraints so spark / rucio build by @lgray in ci(also docs): enforce constraints so spark / rucio build #1299
• docs: we do not support python 3.8 any more! by @lgray in docs: we do not support python 3.8 any more! #1300
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1301
• ci: update tritonserver, use GHA arm runners by @lgray in ci: update tritonserver, use GHA arm runners #1302
• ci: relax setuptools constraint to !=78.0.1 by @lgray in ci: relax setuptools constraint to !=78.0.1 #1303
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1305
• ci: Use astral-sh/setup-uv to setup Python by @matthewfeickert in ci: Use astral-sh/setup-uv to setup Python #1304
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1310
• ci: bump astral-sh/setup-uv from 5 to 6 by @dependabot[bot] in ci: bump astral-sh/setup-uv from 5 to 6 #1313
• build(deps): skip dask 2025.4.0 by @lgray in build(deps): skip dask 2025.4.0 #1315
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1312
• ci: bump actions/attest-build-provenance from 2.2.3 to 2.3.0 by @dependabot[bot] in ci: bump actions/attest-build-provenance from 2.2.3 to 2.3.0 #1316
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1317
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1320
• chore: remove vector deprecation warning and add a warning for the switch to the virtual mode default by @ikrommyd in chore: remove vector deprecation warning and add a warning for the switch to the virtual mode default #1330
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1324
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1336
• ci: change pre-commit schedule to monthly by @ikrommyd in ci: change pre-commit schedule to monthly #1341
• ci(pre-commit): pre-commit autoupdate by @pre-commit-ci[bot] in ci(pre-commit): pre-commit autoupdate #1344
• ci: bump actions/attest-build-provenance from 2.3.0 to 2.4.0 by @dependabot[bot] in ci: bump actions/attest-build-provenance from 2.3.0 to 2.4.0 #1348
• docs: return None in the cases where getattr fails in linkcode_resolve by @ikrommyd in docs: return None in the cases where getattr fails in linkcode_resolve #1349

Full Changelog: v2025.3.0...v2025.7.0

---

This discussion was created from the release v2025.7.0.