Build instructions¶

Preface¶

This page covers topic of local deployment of na64sw software framework. Following common scenarios are considered (ordered by complexity):

No build at all. For those who have acces to CERN LXPLUS environemnt the simples possible option would be to exploit this software right away, from EOS share.
Build in an overriden environment (for instance on AFS or EOS at LXPLUS).
Running under the Docker container. This is neat solution for local builds, requiring minimal effort and compatible with most of the modern Linux/UNIX distributions (including MacOS).
Local build in a native system. This would require user to maintain compatibility of the involved packages by their own.

Each scenario is considered below. _For brevity, each section is accompanied with “TL;DR” section with working snippet only so user may copy/paste it in a shell without the need to go into details.

Scenario #1: LXPLUS¶

This usage scenario won’t require any setup except for source’ing a shell script:

$ source /afs/cern.ch/work/r/rdusaev/public/na64/sw/LCG_96b/x86_64-centos7-gcc62-opt/this-env.sh

This way have several restrictions as re-building the na64sw project is prohibited, but is enough to pass the Tutorial and perform simple analysis.

Scenario #2: Local LXPLUS build¶

For CERN LXPLUS Users we recommend to use an LCG_96 environment:

$ source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_96b x86_64-centos7-gcc62-opt

Clone the repo (see note below if you have no access to CERN GitLab):

$ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git

In order to be able to read the raw data files, build the p348-daq:

$ git clone https://gitlab.cern.ch/P348/p348-daq.git
$ cd p348-daq
$ ./build.sh
$ cd ..

Build this project itself with following commands:

$ mkdir na64sw.build
$ cd na64sw.build
$ cmake ../na64sw
$ make

You are now able to run the applications from your own build.

Scenario #3: Docker container¶

Docker daemon must be running and reachable with permissions to run following commands. Setting up and configuring this daemon is out of scope of this page, so please refer to your distro’s documentation. But in modern distros it is usually done with something like sudo apt install docker.

TL;DR¶

Run on your host:

$ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git
$ git clone ssh://git@gitlab.cern.ch:7999/P348/p348-daq.git
$ cd na64sw/presets/docker
$ docker build -t hepfarm4na64sw .        # takes a while, may require sudo
$ cd -
$ docker run -p5723 --volume $(readlink -f .):/var/src \
                    --volume /data:/data \
                    --volume /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY \
                    -ti hepfarm4na64sw

After last command you shall get into the container console. Command-line invitation shall turn into something like collector@9ad8cf6b3969 ~ $. Then:

$ sudo env-update
$ cd p348-daq/
$ ./build.sh         # takes a while
$ cd p348reco/conddb
$ make
$ cd
$ mkdir na64sw.build
$ cd na64sw.build
$ cmake ../na64sw -DCMAKE_BUILD_TYPE=Debug
$ make -j4           # -jN may be customized depending to fit your CPU

That’s it. After last command done you shall see na64sw-pipe and stuff appeared in a current dir.

Details on Docker build¶

Fetch¶

We shall start with fetching the project sources from remote repositories as we provide Dockerfile within our repo. We also need p348-daq repo around to explot some useful libs from it:

$ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git
$ git clone ssh://git@gitlab.cern.ch:7999/P348/p348-daq.git

Depending on your location and user credentials you may need to authenticate yourself in CERN’s GitLab.

Build Docker Image¶

Now get in the dir with Dockerfile (a manifest defining how to build a particular virtual environment) and run:

$ cd na64sw/presets/docker
$ docker build -t hepfarm4na64sw .

Depending on your system’s setting the last command may require sudo. It also will take a while since it will fetch around 8Gb of software packages from the network (mostly Geant4 and ROOT).

If everything went smooth, you can check that the Docker image built fine with command:

$ docker images

which output must contain something like

hepfarm4na64sw              latest     4618b664e386   2 hours ago   10.9GB

That’s it – the Docker image (virtualized environment with all the packages) is ready to go.

Getting into containerized environment (basic)¶

The Docker image in a nutshell is just a filesystem archive. To make use of it one have to start some running process. The simplest and most representative would be to run the BASH on it. The simplest (yet useless) command would be:

$ docker run -ti hepfarm4na64sw /bin/bash

You may play around in it, but soon will find that there is no common files between this (containerized) FS and your host system’s FS. Exit with exit command (or Control-d) and run BASH container again with:

$ docker run --volume $(readlink -f .):/var/src \
             -ti hepfarm4na64sw /bin/bash

This time /var/src in a container environment will have your current directory mounted. So all the content of your current dir will be available in a container. You may add mountpoints to the docker run invokation to forward your data shares and document folders into container and make it reachable from within.

For your convenience we provide here a command that appeared to satisfy common needs for analysis and software development. It also forwards X11 socket to provide capability of running GUI-based applications (Geant4 UI, ROOT TBrowser, etc):

$ docker run -p5723 --volume $(readlink -f .):/var/src \
                    --volume /data:/data \
                    --volume /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY \
                    -ti hepfarm4na64sw /bin/bash

It also exposes 5723 port that may be used by na64sw monitoring front-ends. Consider putting it into a shell script on your host for convenience.

Container Mountpoints¶

Assuming your data directory to be substituted instead of <your-local-data-dir> and p348-daq with na64sw projects dir instead of <your-local-repo-dir>, run:

$ docker run --rm --volume <your-local-data-dir>:/data \
                  --volume <your-local-repo-dir>:/var/src \
                  -ti hepfarm4ecal-edu /bin/bash

This command will leave you with containerizes interactive prompt for running the software. Note: if you are under SELinux, there might be need to append --volume ...:/var/src with :z postfix to allow editing.

Additionally, one may provide container with X11 display sockets by supplying the above invokation with --volume /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY. It is essential part if you would like to run any GUI application shipped within a container (ROOT, Geant4 Qt manager and so on).

Within the container your command-line prompt should look like collector@<hex-hash> ~ $.

Getting into containerized environment (advanced)¶

Above example runs Bourne Shell as a container, however elaborated usage scenaries often impliy few interactive sessions within a single environment (for instance, using IDE). For that purpose one may run SSH server as a container. We provide a template for customization in utils/run-remote-srv.sh.example.

Running Monitored processing¶

The na64sw-pipe util supports rudimentary remote monitoring procedure. To get representative excerpt over current processing, get the container’s IP with e.g.:

$ ifconfig eth0

the IP address of running container will appear after inet word. Then provide na64sw-pipe invokation with -D@5723 key, e.g.:

$ ./na64sw-pipe -D@5723 -r share/na64sw/run/default.yaml /data/cdr01002-002551.dat

One may run monitoring script on host with (say, container is running on IP 172.17.0.2:

$ bin/monitor.py tcp://172.17.0.2:5723

This will occupy current terminal session with periodically-refreshing excerpt on ongoing processing (number of discriminated events, elapsed time per handler, etc.).

Modifying Docker Image¶

For some particular tasks one may need extending the docker image described by our dockerfile. In a nutshell, this default image is based on some sort of custom Gentoo build (find the barebone image at crankone/hepfarm-amd64-dbg ) and supports modifications easily. So, do not hesitate to customize the Dockerfile for your needs.

Scenario #4: local build on native system¶

This scenario is needed if:

One would like to incorporate NA64SW into another software and container environment does not provide necessary features
Docker container failed to provide some useful features, like OpenGL (known issue for old graphic cards)

This way will require one to make sure some necessary packages are presented in a system.

Common case¶

Commands for installing packages are varying much from distro to distro. But one have to make sure that at least following packages are present in system (in their “development” version for RHEL or Debian!):

yaml-cpp
log4cpp
CMake
CERN ROOT is highly desirable but not necessary

Having those dependencies installed, one can do:

$ git clone ssh://git@gitlab.cern.ch:7999/P348/p348-daq.git
$ cd p348-daq/
$ ./build.sh         # takes a while
$ cd p348reco/conddb
$ make
$ cd -
$ git clone ssh://git@gitlab.cern.ch:7999/P348/tracking-tools.git
$ mkdir tracking-tools/build
$ cd tracking-tools/build
$ cmake ..
$ make -j4  # -jN may be customized depending to fit your CPU
$ cd ../examples/example1
$ cmake ..
$ make -j4
$ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git
$ mkdir na64sw.build
$ cd na64sw.build
$ cmake ../na64sw -DCMAKE_BUILD_TYPE=Debug
$ make -j4
$ cd ..

This will yield a NA64SW utils and libraries available at ./na64sw.build.

Local Build on Ubuntu¶

Full set of commands on Ubuntu, for instance. Will install basic system deps, ROOT by snap, then manually fetch, configure and install RAVE, GenFit, p348-daq and NA64SW. Packages supporting install targets will be configured and built by <prefix>/var/src, packages that don’t will be placed at <prefix>/opt. Assuming $NA64SWPRFX is your local prefix (say, /home/johndoe/projects/na64/software):

$ export NA64SWPRFX=$HOME/local-build  # or wherever you would like to deploy it
$ sudo apt install gfortran libtbb2 libexpat1-dev cmake libyamlcpp-dev \
        liblog4cpp5-dev liburiparser-dev libgsl-dev libeigen3-dev \
        libglew2.1 libgl2ps1.4 libftgl2 libafterimage-dev \
        libclhep-dev lib-gtest-dev libboost-dev
$ sudo snap install root-framework
$ sudo apt install
$ mkdir -p $NA64SWPRFX/var/src
$ cd $NA64SWPRFX/var/src
$ curl -LJO https://rave.hepforge.org/downloads/?f=rave-0.6.25.tar.gz
$ tar xvvf rave-0.6.25.tar.gz
$ cd rave-0.6.25
$ CXXFLAGS=-std=c++11 ./configure --disable-dependency-tracking --prefix=$NA64SWPRFX --disable-java
$ make -j4 install
$ cd ..
$ curl -LJO https://gitlab.cern.ch/-/snippets/2145/raw/master/GenFit2.patch
$ git clone https://github.com/GenFit/GenFit.git
$ cd GenFit
$ git apply --whitespace=fix ../GenFit2.patch
$ mv cmake/FindROOT.cmake cmake/FindROOT.cmake.bck
$ mkdir build
$ cd build
$ RAVEPATH=$NA64SWPRFX cmake .. \
    -DCMAKE_INSTALL_PREFIX=$NA64SWPRFX -DBUILD_TESTING=OFF \
    -DINCLUDE_OUTPUT_DIRECTORY=$NA64SWPRFX/include/genfit \
    -DRave_DIR=$NA64SWPRFX/share/rave/ -DCMAKE_CXX_FLAGS=-std=c++17
$ find . -type f -print0 | xargs -0 sed -i 's/root-framework\/[[:digit:]]\+\//root-framework\/current\//g'
$ make -j4 install
$ cd $NA64SWPRFX/var/src
$ git clone -b tracking-tools-nonuniformfield-999-0.install-targets https://gitlab.cern.ch/P348/tracking-tools.git
$ mkdir tracking-tools/build
$ cd tracking-tools/build
$ cmake .. -DCMAKE_INSTALL_PREFIX=$NA64SWPRFX -DGENFIT_INCLUDE_DIR=$NA64SWPRFX/include/genfit/ -DGENFIT_LIBRARY=$NA64SWPRFX/lib/libgenfit2.so
$ find . -type f -print0 | xargs -0 sed -i 's/root-framework\/[[:digit:]]\+\//root-framework\/current\//g'
$ make -j4 install
$ cd $NA64SWPRFX/opt
$ git clone -b dev/install-targets-compat ssh://git@gitlab.cern.ch:7999/P348/p348-daq.git p348-daq.current
$ cd p348-daq.current
$ ./build.sh
$ cd p348reco
$ ln -s ../../.. tracking-tools
$ ln -s ../../.. GenFit
$ make  # will fetch and build common calibration files
$ cd $NA64SWPRFX/var/src
$ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git
$ mkdir na64sw.build
$ cd na64sw.build
$ cmake ../na64sw -DCMAKE_INSTALL_PREFIX=$NA64SWPRFX \
    -DROOT_DIR=/snap/root-framework/current/usr/local/cmake
$ find . -type f -print0 | xargs -0 sed -i 's/root-framework\/[[:digit:]]\+\//root-framework\/current\//g'
$ make -j4 install

Note, that it is not necessary to keep the $NA64SWPRFX/opt/na64sw sources directory after installation. It is extremely useful if you plan to modify the core library (for instance, event structure) and in this case you may not need an installation step, actually.

Optionally, install millipede with:

$ git clone --depth 1 --branch V04-11-02 \
    https://gitlab.desy.de/claus.kleinwort/millepede-ii.git millipede
$ cd millipede
$ make PREFIX=$(readlink -f ../..) install

Dependencies¶

We’ll cover here dependencies in details. All of them must be available on the target system in their “development” version. For RHEL (Fedora or RedHat Linux) it is usually pkg name + “-devel” suffix installed with dnf or yum, for Debian-based (Ubuntu flavours) it is usually pkg name + “-dev” suffix and installed with apt. Gentoo distro will provide any package with its headers as usual (with emerge).

Mandatory vendor-specific dependencies¶

yaml-cpp – a YAML parser and emitter. Classes provided by this library is ubiquitously used to perform run-time configuration of calibration data, handlers, data allocators, etc. There are plans to either replace it, or to hide with some additional interfacing API to trace down YAML errors in a more convenient way.
log4cpp – a mature C++ message log library. This lib provides thread-safe and extensible capabilites for fast logging. It is lightweight, very mature, though some parts of it seems to be incompatible with recent compiler versions.
CMake, binutils and GCC/clang are needed to build the project.

Note on `log4cpp` versions¶

As for motivation on custom deployment on LXPLUS: log4cpp has throw() qualifiers and auto_ptr type that were deprecated in C++14 and were completely removed from C++17. LCG releases of higher versions require ROOT to be used in >=C++17 because of string_view so we have to either get rid of log4cpp or stick to C++14 or provide custom build with orocos flavour of log4cpp package which has this bug being fixed.

Optional Dependencies¶

These dependencies provide additional features.

CERN’s ROOT data analysis framework is widely used by standard handlers to depict the data with histograms
A GenFit for track reconstruction
CERN’s Geant4 framework used to simulate various setups of the experiment
ZeroMQ provides network communication
Google’s unit testing framework, the gtest provides unit testing facility
uriparser C library – a RFC 3986 compliant parsing and handling lib
A Doxygen processor to extract the documentation from C++ sopurce code and Sphinx + its Breathe extension to render it (note that you barely need it since the latest version is available online).

Also, there are some dependencies from NA64 software ecosystem that are not shipped within this project.

Obtaining the Source Code¶

For our task we typically need two repositories: na64sw and official NA64 reconstruction sources (p348reco-daq).

First, fetch this repository itself:

$ git clone https://bitbucket.org/CrankOne/na64-ecal-edu.git ecal-edu

For raw data reading of the NA64, the DaqDataDecoding library is an essential piece. It is a part of COMPASS (NA58, CERN, SPS) experiment software. The initial bundle managed by NA64 team can be obtained with:

$ git clone https://gitlab.cern.ch/P348/p348-daq.git

Most of the NA64SW utils may operate without this package, though practical usage would be then mostly restricted by some Monte-Carlo analysis.

Todo

Fetching the calibrations.

Configuring the source code¶

Todo

Document options for third-party software, debug and release builds, etc

Building the code¶

Todo

Document code-generation stages, cpu usage, refer to [project install structure](@ref na64sw-install-structure).

Installation¶

Todo

document prefix-based installation stage

Notes on CI/CD¶

Shortest way to assemble image for CI/CD on gitlab is to develop it using VM on OpenStack. Start with, say, CC7 image and do:

yum update
yum install -y yum-utils

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum makecache fast
yum install -y docker-ce

service docker start
docker ps # should output empty list of containers

cd wherever/lies/dockerfile
docker build . -f Dockerfile

To perform local build of packages compatible with LCG environment we need to get access to CVMFS inside a (CC7) container. To setup it on VM:

yum install locmap
locmap --enable cvmfs

Build instructions¶

Preface¶

Scenario #1: LXPLUS¶

Scenario #2: Local LXPLUS build¶

Scenario #3: Docker container¶

TL;DR¶

Details on Docker build¶

Fetch¶

Build Docker Image¶

Getting into containerized environment (basic)¶

Container Mountpoints¶

Getting into containerized environment (advanced)¶

Running Monitored processing¶

Modifying Docker Image¶

Scenario #4: local build on native system¶

Common case¶

Local Build on Ubuntu¶

Dependencies¶

Mandatory vendor-specific dependencies¶

Note on log4cpp versions¶

Optional Dependencies¶

Obtaining the Source Code¶

Configuring the source code¶

Building the code¶

Installation¶

Notes on CI/CD¶

Note on `log4cpp` versions¶