[theoheimel]

This implements the idea discussed at the Heidelberg MadGraph meeting to have an in-memory API for event generation, for instance to pass events directly to Pythia without having to go through an intermediate LHE file.

Summary

Gridpacks made with [gridpack] run_mode = "fix_max_weight" now ship bin/event_stream.py. It is a Python API that generates events in memory, with no survey, no combine pass and nothing written to disk.

import sys; sys.path.insert(0, "<gridpack>/bin")
from event_stream import EventStream

with EventStream(device="cpu", seed=42) as stream:
    for event in stream:                     # ms.LHEEvent, endless
        ...
    for batch in stream.batches(10000, output_type="torch"):
        batch.px                             # (10000, max_particle_count)
    stream.integral, stream.integral_error, stream.n_weighted, stream.channel_status()

• Constructor: the run-card settings that apply to one device: device, seed, cpu_batch_size, gpu_batch_size, thread_pool_size and the cut-repetition settings. Anything left as None comes from grid_run_card.toml.
• batches() yields an EventBatch dataclass of LHE fields:
  • event fields have shape (n,);
  • particle fields have shape (n, max_particle_count);
  • particle_count and particle_mask are included;
  • arrays are numpy, or CPU torch tensors.
• Weights are in units of the channel's maximum weight: ±1, except for overweight events.

How it works

• New C++ madspace::EventStream (driver/event_stream.{hpp,cpp}):
  • A coordinator thread keeps generation jobs running on the context's thread pool.
  • Phase space, matrix element and unweighting go through the existing ChannelEventGenerator::start_job / start_unweight_job.
  • Each channel keeps a buffer of unweighted events. A channel that runs low gets new jobs.
• Channel choice: for every event, the channel is drawn at random.
  • The probability is ∝ the channel's abs-integral estimate (the prior combined with the run so far) ÷ its mean unweighted-event weight. Dividing by the mean weight keeps the result correct when there are overweight events.
  • The selected events are then LHE-completed together as one batch.
• Reproducibility:
  • Each channel's jobs are committed in launch order.
  • A job only enters the integral estimates when its events are consumed.
  • The event sequence therefore depends only on the seed, not on thread timing, prefetching or batch sizes.
• Refactors to avoid duplication:
  • ChannelEventGenerator can be created without event/weight files (empty path). The tensor→EventBuffer packing is split out into fill_event_buffers.
  • fill_lhe_event, the per-event record copy, writing LHE particles, the integral sum over channels and the unweighted-weight formula moved into driver/event_utils.{hpp,cpp}. EventGenerator now uses these.
  • The gridpack setup moved out of gridpack.py into gridpack_setup.py, which is always copied to bin/: locating madspace, contexts, matrix elements, channels, systematics and the LHE meta. Its paths are relative to the gridpack, so it can be imported from any directory.
• Small change: the summed integral error now skips channels with no samples, which previously made it NaN.

Testing

• New madspace/tests/test_event_stream.py (8 tests) checks:
  • identical events across thread counts and batch splits;
  • the effect of the seed;
  • next_events matches next_batch;
  • weights and particles;
  • status;
  • channel fractions match the probabilities;
  • no files written;
  • an error when there is no fixed max weight.
• The existing combine, scheduling, survey-failure, LHE, histogram and systematics madspace tests pass.
• End to end with an e+ e- > mu+ mu- fix_max_weight gridpack:
  • the refactored bin/generate_events still runs;
  • streaming from another directory writes no files;
  • the stream's integral matches the gridpack's;
  • torch output works;
  • events are identical for 1 and 4 threads with different batch splits.

Known limitations / open points

• Not tested on a GPU yet. LHE completion runs single-threaded on the calling thread.
• Iterating event by event takes events in chunks of 1000. If iteration stops partway, the rest of the chunk is dropped.
• Building a fix_max_weight gridpack for e+ e- > u u~ g failed in fix_max_weights() with a NaN integral in channel 0.0, after the survey had passed. These changes don't touch that code path, but I have not checked whether main shows the same failure.

[oliviermattelaer]

The two Build wheels jobs (ubuntu-latest, macos-14) fail at the stub-generation step:

Generated stub is not valid Python: .../_madspace_py.pyi: parameter without a default follows parameter with a default (line 5511)

This comes from this PR, not from an outdated base. Wheels build fine on main (48e611a) and on feat-improve-gridpacks (aac7114).

Cause: in madspace/src/python/madspace.cpp, ChannelEventGenerator.load_json now gives defaults to event_file and weight_file, but the next argument, config, has none:

py::arg("event_file") = "",
py::arg("weight_file") = "",
py::arg("config"),            // no default after defaulted args

pybind11 accepts this at runtime, but the generated .pyi is invalid Python, so pybind11-stubgen --exit-code fails. Local install.py builds don't hit this because they don't run stubgen (it only runs with ENABLE_DOCS, which the wheel build sets).

Suggested fix: add py::kw_only() before py::arg("event_file"):

py::arg("channel_json"),
py::arg("contexts"),
py::kw_only(),
py::arg("event_file") = "",
py::arg("weight_file") = "",
py::arg("config"),

The stub then becomes load_json(channel_json, contexts, *, event_file='', weight_file='', config), which is valid. Every existing caller already passes these by keyword (gridpack_setup.py, tests/test_event_stream.py), so no caller needs to change. Giving config a default GeneratorConfig() would also work.

You can check this locally with pybind11_stubgen madspace._madspace_py --exit-code (plus the --enum-class-locations flags from madspace/generate_pyi.py).

[theoheimel]

Closing in favour of #291, which contains all commits of this branch (its head 18e0283 is an ancestor of feat-madspace-amplicol). The event streaming changes are described in the #291 description.