Wire-Cell Signal Processing, Next Generation.

Goals

The SPNG package shall meet these goals:

• Perform calculations on CPU or GPU with configure-time switch.
• Process each plane independently.
• Pass arrays between functions while leaving them on a GPU device.
• Proper array padding to avoid deconvolution bleed-over / wrap-around.
• Support stream processing with no chunk boundary artifacts.

Array

The goals require basing the package on a new array / tensor type. Some of the requirements for the library providing this support are:

• Provide C++ multi-dimensional array / tensor data structure with at least single precision float element type.
• Element-wise arithmetic, large 1D and 2D FFT (decon), array mask operations (“region of interest finding”).
• Ability to run DL model inference a bonus.
• Support a run/config-time switch to place arrays on CPU or on GPU.
• Multiprocessing sharing a single GPU required, multi compute-units sharing a GPU (eg NVIDIA Triton) is a bonus.
• Support for CPU + NVIDIA GPU at least, but the more the merrier.
• Ability to pass handles to const arrays between disparate functions while keeping array data on a device with transparent, lazy CPU<–>GPU migration if/when needed.

Source

The spng source code simultaneously exists in two repositories: the stand-alone spng repo and the main wire-cell-toolkit repo (specifically in its “spng” branch).

In both cases the source must be provided as a spng/ sub-directory of a working directory of a wire-cell-toolkit repo clone. In either case, it is important to know that the connection between the two repos is limited to changes to the spng/ sub-directory as it resides in:

• spng repo “master” branch
• wire-cell-toolkit repo “spng” branch

This dual-source modal nature is historical and temporary. It is expected that the stand-alone spng repo will eventually become stale and all development will be done in the wire-cell-toolkit repository directly. Until then, this section describes how the developer may operate in either of the two source modes and then gives some simple commit push policy.

Local graft

In this mode, the developer will “graft” the stand-alone spng repo to local clone of wire-cell-toolkit repo as a spng/ sub-directory.

This mode is useful if the developer is working on a branch of the spng repo other than “master”.

$ git clone git@github.com:WireCell/wire-cell-toolkit.git
$ git clone git@github.com:wirecell/spng.git wire-cell-toolkit/spng

$ cd wire-cell-toolkit/spng
$ git switch my-feature-branch

In this mode, the developer may pick any branch of wire-cell-toolkit repo except the “spng” branch to form the base. The developer may work in any branch of the cloned spng repo. Any push/pull/commit for changes to the spng/ directory must be run from the spng/ subdir.

$ cd spng/
$ git <push|pull|commit> [...]
$ cd ..

In order for commits to the a spng repo feature branch to be used by others, they must be merged to spng repo “master” branch either via local git merge and git push or via GitHub PR. Afterwards, these new commits should be pulled to the wire-cell-toolkit repo “spng” branch as described below. Developers should periodically pull changes pushed to spng repo “master” branch.

Or, more simply, convert to the next mode:

WCT branch

This is expected to eventually be the only mode for working with SPNG source. In this mode, the developer need not know that spng/ has a git subrepo connection to the spng repo. Simply work on the “spng” branch:

$ git clone git@github.com:WireCell/wire-cell-toolkit.git
$ cd wire-cell-toolkit
$ git switch spng
$ (hack)
$ git <push|pull|commit> [...]

While the dual-source modality is maintained, the developer should also push the subrepo.

$ git subrepo push spng

The rest of this section summarizes the basic git subrepo maintenance. Initial creation:

$ git switch spng
$ git subrepo clone git@github.com:WireCell/spng.git spng

Pull updates in the spng repo “master” branch to the wire-cell-toolkit repo “spng” branch with:

$ git subrepo pull spng

Push updates in the wire-cell-toolkit repo “spng” branch to the spng repo “master” branch with:

$ git subrepo push spng

Independently, developers following the Local graft approach must push/pull spng repo “master” branch to put/get commits made by developers using this mode.

Commit push policy

This section categorizes two types of branches and describes simple policy specific to each.

Main branches

The spng repo “master” branch and wire-cell-toolkit repo “spng” branch are “main branches” and are intended to have identical content while the dual-source modality is maintained.

• You may push to a “main” branch as long as the code compiles.
• You should follow up by syncing the two “main” branches in the spng and wire-cell-toolkit repos.

Feature branches

Any spng repo branch other than “master” is a “feature branch and any wire-cell-toolkit branch other than “spng” which has SPNG as a git subrepo are “feature branches”.

• You may freely push to feature branches.
• Developers may share feature branches and should work out any additional policy for them as needed.

Building

See Source to prepare a source area. Then we treat spng/ as a normal WCT “subpackage” and configure the build something like:

$ TDIR=$PREFIX/lib/python3.11/site-packages/torch \
  ./wcb configure \
  --prefix=$PREFIX \
  --boost-mt \
  --boost-libs=$PREFIX/lib \
  --boost-include=$PREFIX/include \
  --with-jsonnet-libs=gojsonnet \
  --with-cuda-lib=/usr/lib/x86_64-linux-gnu,$PREFIX/targets/x86_64-linux/lib \
  --with-cuda-include=$PREFIX/targets/x86_64-linux/include/ \
  --with-libtorch=$TDIR \
  --with-libtorch-include=$TDIR/include,$TDIR/include/torch/csrc/api/include \
  --with-libtorch-lib=$TDIR/lib

$ ./wcb

Some contexts where PREFIX are defined:

• https://www.phy.bnl.gov/~bviren/wire-cell/docs/wcwc-howto.html#spack-view-wct
• https://github.com/brettviren/winch/blob/master/doc/spng.org

Note on Torch

The Torch package is “weird” in a few ways. You must direct the source to use not just two torch include areas but one that is inside the other. The locations of these C++ dependencies are themselves deep inside a Python installation area. And, that area is parameterized by the Python version.

If you have working Python runtime access to the Torch installation you can learn the base directory of the Torch installation with:

$ python -c 'import torch; print(torch.__path__[0]);'

Testing

Tests are in spng/tests/{doctest,test,check}_*.* following WCT’s testing support.

$ ./build/spng/wcdoctest-spng
$ ./build/spng/check_raytiling_speed [cpu|gpu] [yes|no] [seed]