Parallelism and distribution

monoprop scales the same way on one node and on many: it partitions the operator into disjoint pieces and applies each gate to every partition in lock-step, exchanging only the terms that cross a partition boundary. Two axes compose:

• Across cores (default): one single-threaded partition per physical core, within a single process. This is on automatically, and can be configured with environment variables (see below).
• Across nodes (opt-in): MPI ranks, each of which partitions further across its own cores. Distributing the operator and its graph across ranks lets you simulate and variationally optimise systems larger than one node's memory.

Operator partitioning (single node)

By default monoprop splits the operator into one partition per physical core and gives each partition a pinned worker thread that runs it serially. Each partition keeps its own small, cache-resident term index; a gate is applied to all partitions at once, synchronised by a lightweight barrier, and anticommuting terms whose partner lives in another partition are resolved through a per-gate exchange.

A partition addresses its own terms with a 32-bit index, so one partition holds at most 2^32 (about 4.3 billion) terms. That is a few hundred gigabytes of resident operator in a single partition, so the ceiling is reached by running too few partitions rather than by problem size, and it is not a build option. An append that would cross it is refused before it happens — a RuntimeError naming the ceiling and the term counts, with the operator left as it was, rather than a silent wrap. The fix is more partitions or more MPI ranks — both divide the terms per partition — or a larger lower_atol (see Truncation and cutoffs).

Runtime environment variables

# Run 8 partitions instead of one-per-core:
export monoprop_NUM_THREADS=8

Where the operator's bytes are

operator_memory_breakdown() names the resident bytes of one rank's operator. total_bytes() sums the fields that are live at quiescence; the d_ keys are counts, subsets of a field, or quantities no resting field can hold, so folding one into the total would double-count.

The row store and the anticommutation index's dense columns are held as fixed-size chunks taken from a pool, not as vectors: growth appends a chunk, nothing is copied, and the only slack is the tail of the last chunk. Four keys describe that storage. d_terms_slack_bytes is that tail. d_pool_mapped_bytes is what the pools have mapped -- a pool maps whole arenas and keeps one as long as a single chunk in it is live, so it sits at or above what the chunk-counting fields price, and d_pool_free_chunk_bytes is the part of it not currently handed out. Those bytes are faulted only as they are written, but the mapping is what the kernel's high-water mark is charged for once they have been, which is why the ledger's total can sit below the resident set with nothing missing from any named field.

d_row_wide_rows, d_row_inline_width and d_row_restrides describe the row layout. A row is a popcount slot followed by its set positions, at one width paid for by every row; rows wider than that but still within the cutoff's structural bound go to a second tier at a fixed wider stride, and rows over the bound to a lossless side-map. The width is predicted from the model, so a wrong guess costs one restride -- the store re-lays every row at the bound between gates, keeping every row index -- and never a wrong answer. A nonzero d_row_restrides says the prediction was too narrow for this model; a wide-row count that is a large share of the term count says the same before it happens.

d_op_coeffs_slack_bytes is the most capacity the coefficient array held beyond its live rows at any point in the last propagate or build_graph call. The array grows at the row store's own 1.5x policy and is shrunk to fit when the call ends, which is the only moment a caller can read the ledger -- so it is reported as a high-water mark taken at the growth sites rather than as a measurement of the array as it stands, which would always be 0. Across partitions the marks are summed rather than maxed: the partitions grow together within a call, so the sum is the figure a per-process footprint wants, and it errs high.

Peak memory of large runs: pin glibc's mmap threshold

Every gate allocates and frees a few large transient buffers: the records it stages, sends, receives and answers. glibc's allocator reacts to the first free of such a buffer by raising its dynamic mmap threshold (up to 32 MiB) and its trim threshold, so from then on the widest gate's transients are served from the heap and kept on free lists the allocator cannot return to the kernel. On a 250 M-term Hubbard run that retention is about a fifth of the resident set. Pinning the threshold disables the escalation, so every transient above it is a private mapping that is unmapped on free:

export GLIBC_TUNABLES=glibc.malloc.mmap_threshold=1048576   # 1 MiB
mpirun -x GLIBC_TUNABLES ...                                # MPI ranks must inherit it

Measured on a 16-core workstation at 250 M terms (paired interleaved repetitions): peak resident high-water mark 0.95x in-process with 16 partitions and 0.91x (ranks summed) at 4 ranks x 4 partitions, for about 1 % more wall time; a 128 KiB threshold takes a further 2 % of memory for 2-4 % more time. The cost is page faults on the re-mapped transients, so the setting pays only when a gate's compute amortises them: use it at or above ~10^8 terms with several partitions, and not on small single-core runs, where it costs ~20 % time. Results are bit-identical either way -- this changes where bytes live, not what is computed.

Placement report

Building a propagator writes one COMMPLACE line per rank to stderr, naming the CPUs the launcher gave that rank and whether co-located ranks got disjoint masks. It is report-only — no placement decision reads it — and there is no knob: redirect stderr to drop it.

COMMPLACE rank=0 node_rank=0 node_size=2 masks=private cpus=64 node_cpus=128 cpu_list=0-63

masks is private when the co-located ranks' affinity masks are pairwise disjoint, shared when two ranks can land on the same CPU, alone when this rank is the only one on its host (which is not evidence a multi-rank launcher bound correctly), and unknown when a mask did not fit the exchanged window.

Pinning each partition to a core is not configurable: leaving placement to the launcher measured propagate[hubbard] 2.90x slower, so the disabled arm is gone.

MPI distribution (multi-node)

MPI partitions the operator and graph across ranks, composing with per-rank partitioning into one flat world of R × S partitions (R ranks, S partitions each). MPI communication is serialised through each rank's first partition, bracketed by the intra-rank barriers.

Single-node (MPI.COMM_SELF)

from mpi4py import MPI
sim = MajoranaPropagator(..., comm=MPI.COMM_SELF)

Multi-node (MPI.COMM_WORLD)

Replace the communicator and launch with mpiexec:

from mpi4py import MPI
sim = MajoranaPropagator(..., comm=MPI.COMM_WORLD)

mpiexec -n 8 uv run python your_script.py

A pure-MPI rank keeps its whole share in one partition unless monoprop_NUM_THREADS is set, so an MPI user who has not asked for threads gets one partition per rank.

Enabling MPI

MPI is off by default, so the prebuilt PyPI wheels run single-rank and the communicators above only distribute work after a from-source build with MPI enabled. See Building from source for the full build instructions.