Add multithreading support to ddsim#1240
Conversation
Test Results 18 files 18 suites 10h 7m 13s ⏱️ Results for commit dcfd47d. ♻️ This comment has been updated with latest results. |
|
This is still a work in progress, but let me give some more background here as to the plan (the "more later" from the top).
It is likely that I will be doing some rewrites of the last commit (5bf711c) which is a bit too big for my taste and not as clear as one would like for non-squash merging. |
|
The status of this PR after #1417 is that we can now run MT gun simulations, like so: Main missing feature is ensuring |
|
The primary remaining difficulty I'm having here is with the event input files (I'm using HepMC3), which we create in: DD4hep/DDG4/python/DDSim/DD4hepSimulation.py Lines 444 to 446 in 6143d12 Simply making the GeneratorAction shared with To get around this, I went through setting up the necessary So, I'm open to input as to how best to have a single @MarkusFrankATcernch @andresailer Do you have any suggestions? |
|
Hi Wouter, This construct is not simple. Let's see what is necessary before the hacking starts... For Now what one probably wants is somehow a structure that this worker thread bound From what I see somehow the input stage needs then to be re-designed. In particular exchanging data between the actions building the queue of Having said all this, it looks necessary that the queue of
How to do something like this in a backwards compatible way is not so obvious to me and should be designed properly. |
|
So after coming back from the drawing board....
to an input stage which looks much rather like this (details are left out here):
Here you have
In principle this is not too difficult, but still quite more difficult that you have imagined upfront. In addition:
|
|
While I agree that the diagram you outline (with an event queue) is optimal for throughput performance, I wonder if there is another less optimal but still functional approach (which may have some threads waiting unnecessarily for some of the time). What I had originally anticipated was a pattern where there are multiple Geant4InputActions sequences, one per thread as now, but the action that reads from disk would have a shared reader that is locked. This would mimic the approach of the Geant4OutputAction where each thread has an output event action, but they share and lock the output file. |
|
@wdconinc Where one actually puts the Geant4EventQueue is sort of user-gusto here, but it must be located before the If we invest here, we should do it properly....even if it costs some development. |
See #1440, which exposes the properties correctly now. |
|
@wdconinc @MarkusFrankATcernch Hi Wouter and Markus, what is the status of including multithreading into ddsim with HEPMC3 file inputs? I am working on a large simulation, and having MT could potentially give a nice boost in throughput. If there are concrete places that I can contribute, please let me know. If the exposure of properties is handled, is this "simply" a matter of having ddsim handle file reads and writes correctly? Or are there more fundamental components to update? Again, keen to help where it makes sense! |
|
@murnanedaniel Depending on your use case, this already works fine (multi-threaded).
I have been using this for a month or two (hepmc3 input files) typically 8 threads and it scales without loss of efficiency at that (small) thread count. |
|
@wdconinc @murnanedaniel |
The scope of this PR has always been to make multi-threaded running work with ddsim. I don't know what other ways of running DDG4 people are using, but I'm not familiar with them. |
|
Hi @wdconinc. Thanks for the quick response. I have been trying to reproduce your working setup with input hepmc3 files, but get this error, when setting to 8 threads: I removed the elif python handler test that was causing the recursion, leading to (I believe) the real error, which is this: For context, here is the full output log: full_output.log |
|
Okay, I made a couple of quick tweaks and it does seem to work, which is really promising. Firstly, the tweaks I made are here: murnanedaniel@cbbe86e. I'm not sure if you agree with them, or why they are not necessary in your setup? Secondly, there is one final error appearing after the simulation that doesn't appear in the main DD4hep. You can see it at the end of the successful log file: full_output_mt.log |
|
@wdconinc Just following up here, since we will soon be running a large set of simulations with your MT changes. I just want to ensure that the small error that appears, and the tweaks that I needed to make, are still consistent with the working version of ddsim as you see it. On that same point - have you validated that MT sim of hepmc3 inputs produces more or less identical outputs to DD4hep main? |
|
Is there any plan to have this integrated? |
|
@BrieucF Whenever Wouter gives hist OK. |
|
I added what I have locally, which consists of mainly a number of new added tests that aim to compare single- and multi-threaded output. This requires a comparison script to allow for out-of-order event output (but still identical when comparing the event using an index map). There are some persistent differences between 2 and 4 threaded running. Those are still under investigation. |
This reverts commit e062d12.
In MT mode, creating the EventSeeder with shared=True caused the Geant4EventSeed to be instantiated on the master kernel, registering beginEvent on the master's event action sequence rather than the worker's. Since no master-side G4UserEventAction is ever created, the callback never fired and every MT event used the same random seed. Fix by using shared=False so each worker gets its own Geant4EventSeed with callbacks registered on the worker's event and stacking action sequences. Add a prepareEvent (stacking prepare) callback that re-seeds after GeneratePrimaries. File-based generators call event->SetEventID() inside GeneratePrimaries to assign the sequential file position, which differs from the Geant4-internal pre-assigned event ID seen by beginEvent in multi-threaded mode. Re-seeding after GeneratePrimaries guarantees reproducible physics across different thread counts.
Rename tests to t_ddsim_{generator}_mt{N}[_mt{M}_compare] scheme.
Rename st to mt1 throughout.
Remove duplicate and redundant tests:
- t_ddsim_mt_basic_2/4threads and t_ddsim_mt_single_thread are merged
into t_ddsim_gun_mt1/mt2/mt4 which now carry the PASS_REGULAR_EXPRESSION
thread-count checks and serve as inputs to both comparison pairs.
- t_ddsim_mt2_vs_mt4_field_* removed: SiD already has a solenoid field
so the regular gun tests exercise the same field path.
- t_ddsim_mt2_vs_mt4_gps_generate_* and _compare removed: they used
the regular DD4hep gun (-G), not GPS, making them redundant with the
gun comparison tests.
Add t_ddsim_gun_mt1_mt2_compare (single- vs multi-threaded gun).
The shared t_ddsim_gun_mt2 output is reused by both gun comparison tests.
Add t_ddsim_gps_mt2, t_ddsim_gps_mt4, t_ddsim_gps_mt2_mt4_compare, parallel to the existing G4Gun comparison tests. GPS batch tests use --macroFile for consistency; particle configuration uses Geant4 defaults since run_test.sh cannot pass macro-file contents in batch mode. Raise /run/beamOn in g4gun_test.mac and g4gps_test.mac from 3 to 10 to match the minimum event count used in all other tests.
The EDM4hepFileReader plugin lives in libDDG4EDM4HEPReader.so, a separate library from the output plugin libDDG4EDM4HEP.so. In MT mode the reader is created lazily inside Geant4InputAction::createReader() which executes on a worker thread. Loading a new shared library via the GaudiPluginService from a worker thread can fail because ROOT's internal state is not fully thread-safe at that point. Explicitly load libDDG4EDM4HEPReader on the master thread before PyDDG4.configure()/initialize() spawn the worker threads, so the factory is already registered when workers request it.
This reverts commit 95990dc.
…ribute Two issues when comparing DD4hep-native ROOT output (used as fallback when EDM4hep is not available): 1. Without loading libDDG4/libDDG4Plugins, ROOT has no dictionary for dd4hep::sim::Geant4Tracker::Hit etc. and cppyy cannot cast the vector<...Hit*> branch objects, crashing on every branch access. Fix: load both libraries before opening files. 2. DD4hep hit objects use 'energyDeposit' while the comparison chain only checked getEDep/EDep/getEnergy/energy, silently skipping energy comparison for all DD4hep-format hits. Fix: add 'energyDeposit' to the fallback chain.
When the run() method was refactored to inline the detector construction setup (instead of calling self.geometry.constructGeometry()), two things were lost: 1. The geometry debug/print flags from self.geometry (DebugMaterials, DebugElements, DebugVolumes, etc.) were not applied to the Geant4DetectorGeometryConstruction action. 2. The Geant4RegexSensitivesConstruction actions configured via SIM.geometry.regexSensitiveDetector were never registered, breaking the BoxOfStraws example and any detector using regex-matched sensitive volumes. Both are restored by propagating geometry properties to the ConstructGeo action and iterating over regexSensitiveDetector after ConstructSD.
|
The only remaining issue here is now the G4EventID that isn't a 'proper' branch but a direct leaf, and it fails in DDDigi. @jmcarcell Do you want to take a look? |
Scalar leaf branches (Int_t, Float_t etc.) have no ROOT class name and are not expected by downstream readers such as DDDigi's DigiROOTInput, which iterates the EVENT tree's object branches to convert hit collections. Writing G4EventID as a scalar branch in EVENT caused 'Unknown data type' FATAL errors in DDDigi. Fix by buffering G4EventIDs in commit() and writing them as a separate TTree 'G4EventIDs' (one entry per event) at file-close time. DDDigi opens only the 'EVENT' tree by name and is unaffected. The comparison script reads G4EventID from the companion tree for event-ID-based matching in MT mode; EDM4hep files continue to use EventHeader as before. Also clear m_sections fully at file close to prevent dangling TTree* pointers to trees owned by the (now closed) TFile.
The G4EventIDs TTree filled in commit() was never flushed to disk because m_sections was cleared before any Write() calls on it. TFile::Close() does not auto-flush TTrees; only m_tree->Write() was called explicitly, leaving G4EventIDs in memory only. Iterate m_sections (after erasing the main EVENT entry) and call Write() on each companion tree before clearing the map.
Passes all checks. Zarro boogs! |
|
Now, let's see what copilot thinks about all this... |
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds multi-threaded (MT) simulation validation by running ddsim with multiple thread counts and comparing outputs for physics-equivalence, alongside core MT determinism and ROOT I/O safety improvements.
Changes:
- Add CTest coverage for MT runs (gun, HepMC3, EDM4hep, G4Gun, GPS) plus file-to-file output comparisons.
- Introduce a ROOT-output comparison utility script and Geant4 macro inputs for G4Gun/GPS tests.
- Improve MT reproducibility and stability via per-thread random engines, event reseeding updates, ROOT I/O locking, and exposing PyDDG4 step-wise lifecycle calls.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| DDTest/python/compare_simulation_output.py | New Python comparator to validate ST vs MT output equivalence (branches, hits, particle properties). |
| DDTest/inputFiles/g4gun_test.mac | Adds a simple Geant4 Gun macro for MT/ST macroFile testing. |
| DDTest/inputFiles/g4gps_test.mac | Adds a simple Geant4 GPS macro for MT/ST macroFile testing. |
| DDTest/CMakeLists.txt | Adds many new MT CTest invocations and comparator-based validations. |
| DDG4/src/python/PyDDG4.cpp | Splits lifecycle into configure/initialize/run/terminate and renames legacy “all phases” runner to runAll. |
| DDG4/src/Geant4Random.cpp | Makes random instance thread-local and routes calls through a per-instance ROOT RNG member. |
| DDG4/src/Geant4Output2ROOT.cpp | Adds ROOT I/O mutexing, endRun-based closing, and a companion G4EventIDs tree for MT ordering. |
| DDG4/src/Geant4Kernel.cpp | Declares RunManagerType as a property for worker kernels. |
| DDG4/python/DDSim/Helper/Random.py | Adds per-worker random initialization and defers/duplicates EventSeeder per worker. |
| DDG4/python/DDSim/Helper/Geometry.py | Refactors geometry construction to only build geometry/regex SD steps (sensitives attached elsewhere). |
| DDG4/python/DDSim/DD4hepSimulation.py | Adds -j/--numberOfThreads, MT setup hooks, adjusted generator sharing, and step-wise PyDDG4 lifecycle usage. |
| DDG4/python/DDG4.py | Avoids shared output actions in ST and switches Geant4.run() to PyDDG4.runAll(). |
| DDG4/plugins/Geant4EventSeed.h | Adds prepare-stacking hook to reseed after GeneratePrimaries with final event ID. |
| DDG4/plugins/Geant4EventSeed.cpp | Implements reseeding in prepareEvent via G4EventManager current event. |
| DDG4/include/DDG4/Python/PyDDG4.h | Adds runAll/configure/initialize/terminate entry points. |
| DDG4/include/DDG4/Geant4Output2ROOT.h | Adds mutex + atomic endRun counter and overrides endRun. |
| DDG4/include/DDG4/Geant4Kernel.h | Exposes numThreads() accessor for kernel thread count. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| int PyDDG4::execute() { | ||
| Kernel& k = Kernel::instance(dd4hep::Detector::getInstance()); | ||
| return run(k); |
There was a problem hiding this comment.
This is intentional since we need to be able to call the phases in isolation, not only all together (with the new runAll which is the previous behavior.
'is not []' is an identity comparison that is always True regardless of whether the list is empty, since it compares against a freshly allocated list object. Use bare truthiness checks instead.
The inlined close body in beginRun() was identical to closeOutput() but couldn't call it due to the mutex already being held. Extract the core logic into a private closeOutputLocked() (called with s_rootMutex held) and have both closeOutput() and beginRun() use it.
Resetting the counter inside endRun() after closeOutput() leaves a window between close and reset where a new run could theoretically start and see stale counts. Reset in beginRun() under the mutex, before any events are processed for the new run, which is the natural boundary.
compare_vectors() silently returned True when neither (x,y,z) nor (X,Y,Z) coordinate attributes were found, potentially masking real mismatches for future hit types. Print a WARNING instead. Also check the return value of gSystem.Load(): a return value < 0 indicates failure. Emit a warning so the user knows DD4hep-native branches may not be readable, rather than hitting confusing errors later.
|
Now that the tests run multi-threaded, the issues like https://github.com/AIDASoft/DD4hep/actions/runs/27167948215/job/80199881284?pr=1240#step:4:6627 will pop up for older LCG releases due to Geant4's ongoing efforts at removing thread unsafe code (e.g. multiple threads writing simultaneously to a single static cache as in e.g. G4EmSaturation, or the ones listed in wdconinc/geant4#43 which still apply to 11.4.1). |
This PR adds multithreading support to ddsim, through the
-jflag.Backwards-incompatible effects (as the corner cases that they are):
__setupWorker, this means they may not be available in the same context as global actions. This means that during user physics list initialization one cannot query particle handlers which have not been created yet.BEGINRELEASENOTES
ENDRELEASENOTES