Build instructions
==================

Preface
-------

This page covers topic of local deployment of ``na64sw`` software framework.
Following common scenarios are considered (ordered by complexity):

1. :ref:`No build at all <lxplus nobuild>`. For those who have acces to CERN
   LXPLUS environemnt the simples possible option would be to exploit this
   software right away, from EOS share.
2. :ref:`Build in an overriden environment <local lxplus build>` (for instance
   on AFS or EOS at LXPLUS).
3. :ref:`Running under the Docker container <docker build>`. This is neat
   solution for local builds, requiring minimal effort and compatible with most
   of the modern Linux/UNIX distributions (including MacOS).
4. :ref:`Local build in a native system <native build>`. This would require user
   to maintain compatibility of the involved packages by their own.

Each scenario is considered below. _For brevity, each section is accompanied
with "TL;DR" section with working snippet only so user may copy/paste it in
a shell without the need to go into details.

.. _lxplus nobuild:

Scenario #1: LXPLUS
-------------------

This usage scenario won't require any setup except for source'ing a shell
script:

.. code-block:: shell

    $ source /afs/cern.ch/work/r/rdusaev/public/na64/sw/LCG_96b/x86_64-centos7-gcc62-opt/this-env.sh

This way have several restrictions as re-building the ``na64sw`` project is
prohibited, but is enough to pass the :ref:`tutorial` and perform
simple analysis.

.. _local lxplus build:

Scenario #2: Local LXPLUS build
-------------------------------

For CERN LXPLUS Users we recommend to use an ``LCG_96`` environment:

.. code-block:: shell

    $ source /cvmfs/sft.cern.ch/lcg/views/setupViews.sh LCG_96b x86_64-centos7-gcc62-opt

Clone the repo (see note below if you have no access to CERN GitLab):

.. code-block:: shell

    $ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git

In order to be able to read the raw data files, build the ``p348-daq``:

.. code-block:: shell

    $ git clone https://gitlab.cern.ch/P348/p348-daq.git
    $ cd p348-daq
    $ ./build.sh
    $ cd ..

Build this project itself with following commands:

.. code-block:: shell

    $ mkdir na64sw.build
    $ cd na64sw.build
    $ cmake ../na64sw
    $ make

You are now able to run the applications from your own build.

.. _docker build:

Scenario #3: Docker container
-----------------------------

Docker daemon must be running and reachable with permissions to run following
commands. Setting up and configuring this daemon is out of scope of this page,
so please refer to your distro's documentation. But in modern distros it
is usually done with something like ``sudo apt install docker``.

TL;DR
~~~~~

Run on your host:

.. code-block:: shell

    $ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git
    $ git clone ssh://git@gitlab.cern.ch:7999/P348/p348-daq.git
    $ cd na64sw/presets/docker
    $ docker build -t hepfarm4na64sw .        # takes a while, may require sudo
    $ cd -
    $ docker run -p5723 --volume $(readlink -f .):/var/src \
                        --volume /data:/data \
                        --volume /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY \
                        -ti hepfarm4na64sw

After last command you shall get into the container console. Command-line
invitation shall turn into something like ``collector@9ad8cf6b3969 ~ $``. Then:

.. code-block:: shell

    $ sudo env-update
    $ cd p348-daq/
    $ ./build.sh         # takes a while
    $ cd p348reco/conddb
    $ make
    $ cd
    $ mkdir na64sw.build
    $ cd na64sw.build
    $ cmake ../na64sw -DCMAKE_BUILD_TYPE=Debug
    $ make -j4           # -jN may be customized depending to fit your CPU

That's it. After last command done you shall see ``na64sw-pipe`` and stuff
appeared in a current dir.

Details on Docker build
~~~~~~~~~~~~~~~~~~~~~~~

Fetch
+++++

We shall start with fetching the project sources from remote repositories as we
provide ``Dockerfile`` within our repo. We also need ``p348-daq`` repo around to
explot some useful libs from it:

.. code-block:: shell

    $ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git
    $ git clone ssh://git@gitlab.cern.ch:7999/P348/p348-daq.git

Depending on your location and user credentials you may need to authenticate
yourself in CERN's GitLab.

Build Docker Image
++++++++++++++++++

Now get in the dir with ``Dockerfile`` (a manifest defining how to build a
particular virtual environment) and run:

.. code-block:: shell

    $ cd na64sw/presets/docker
    $ docker build -t hepfarm4na64sw .

Depending on your system's setting the last command may require ``sudo``. It also
will take a while since it will fetch around 8Gb of software packages from the
network (mostly Geant4 and ROOT).

If everything went smooth, you can check that the Docker image built fine with
command:

.. code-block:: shell

    $ docker images

which output must contain something like

.. code-block:: shell

    hepfarm4na64sw              latest     4618b664e386   2 hours ago   10.9GB

That's it -- the Docker image (virtualized environment with all the packages)
is ready to go.

Getting into containerized environment (basic)
++++++++++++++++++++++++++++++++++++++++++++++

The Docker image in a nutshell is just a filesystem archive. To make use of it
one have to start some running process. The simplest and most representative
would be to run the BASH on it. The simplest (yet useless) command would be:

.. code-block:: shell

    $ docker run -ti hepfarm4na64sw /bin/bash

You may play around in it, but soon will find that there is no common files
between this (containerized) FS and your host system's FS. Exit with ``exit``
command (or :kbd:`Control-d`) and run BASH container again with:

.. code-block:: shell

    $ docker run --volume $(readlink -f .):/var/src \
                 -ti hepfarm4na64sw /bin/bash

This time ``/var/src`` in a container environment will have your current
directory mounted. So all the content of your current dir will be available
in a container. You may add mountpoints to the ``docker run`` invokation to
forward your data shares and document folders into container and make it
reachable from within.

For your convenience we provide here a command that appeared to satisfy common
needs for analysis and software development. It also forwards X11 socket to
provide capability of running GUI-based applications (Geant4 UI, ROOT
``TBrowser``, etc):

.. code-block:: shell

    $ docker run -p5723 --volume $(readlink -f .):/var/src \
                        --volume /data:/data \
                        --volume /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY \
                        -ti hepfarm4na64sw /bin/bash

It also exposes ``5723`` port that may be used by ``na64sw`` monitoring front-ends.
Consider putting it into a shell script on your host for convenience.

Container Mountpoints
+++++++++++++++++++++

Assuming your data directory to be substituted instead of
``<your-local-data-dir>`` and ``p348-daq`` with ``na64sw`` projects dir instead
of ``<your-local-repo-dir>``, run:

.. code-block:: shell

    $ docker run --rm --volume <your-local-data-dir>:/data \
                      --volume <your-local-repo-dir>:/var/src \
                      -ti hepfarm4ecal-edu /bin/bash

This command will leave you with containerizes interactive prompt for running
the software. Note: if you are under SELinux, there might be need to append
``--volume ...:/var/src`` with ``:z`` postfix to allow editing.

Additionally, one may provide container with X11 display sockets by supplying
the above invokation with
``--volume /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=unix$DISPLAY``. It is
essential part if you would like to run any GUI application shipped within a
container (ROOT, Geant4 Qt manager and so on).

Within the container your command-line prompt should look like
``collector@<hex-hash> ~ $``.

Getting into containerized environment (advanced)
+++++++++++++++++++++++++++++++++++++++++++++++++

Above example runs Bourne Shell as a container, however elaborated usage
scenaries often impliy few interactive sessions within a single environment
(for instance, `using IDE`_).
For that purpose one may run SSH server as a container. We provide a template
for customization in ``utils/run-remote-srv.sh.example``.

.. _using IDE: https://www.jetbrains.com/help/clion/remote-projects-support.html

Running Monitored processing
++++++++++++++++++++++++++++

The ``na64sw-pipe`` util supports rudimentary remote
monitoring procedure. To get representative excerpt over current processing,
get the container's IP with e.g.:

.. code-block:: shell

    $ ifconfig eth0

the IP address of running container will appear after ``inet`` word. Then
provide ``na64sw-pipe`` invokation with ``-D@5723`` key, e.g.:

.. code-block:: shell

    $ ./na64sw-pipe -D@5723 -r share/na64sw/run/default.yaml /data/cdr01002-002551.dat

One may run monitoring script on host with (say, container is running on
IP ``172.17.0.2``:

.. code-block:: shell

    $ bin/monitor.py tcp://172.17.0.2:5723

This will occupy current terminal session with periodically-refreshing excerpt
on ongoing processing (number of discriminated events, elapsed time per handler,
etc.).

Modifying Docker Image
++++++++++++++++++++++

For some particular tasks one may need extending the docker image described by
our dockerfile. In a nutshell, this default image is based on some sort of
custom Gentoo build (find the barebone image at `crankone/hepfarm-amd64-dbg`_ )
and supports modifications easily. So, do not hesitate to customize the
Dockerfile for your needs.

.. _crankone/hepfarm-amd64-dbg: https://hub.docker.com/repository/docker/crankone/hepfarm-amd64-dbg

.. _native build:

Scenario #4: local build on native system
-----------------------------------------

This scenario is needed if:

- One would like to incorporate NA64SW into another software and container
  environment does not provide necessary features
- Docker container failed to provide some useful features, like OpenGL (known
  issue for old graphic cards)

This way will require one to make sure some necessary packages are presented in
a system.

Common case
~~~~~~~~~~~

Commands for installing packages are varying much from distro to distro.
But one have to make sure that at least following packages are present in
system (in their "development" version for RHEL or Debian!):

- yaml-cpp
- log4cpp
- CMake
- CERN ROOT is highly desirable but not necessary

Having those dependencies installed, one can do:

.. code-block:: shell

    $ git clone ssh://git@gitlab.cern.ch:7999/P348/p348-daq.git
    $ cd p348-daq/
    $ ./build.sh         # takes a while
    $ cd p348reco/conddb
    $ make
    $ cd -
    $ git clone ssh://git@gitlab.cern.ch:7999/P348/tracking-tools.git
    $ mkdir tracking-tools/build
    $ cd tracking-tools/build
    $ cmake ..
    $ make -j4  # -jN may be customized depending to fit your CPU
    $ cd ../examples/example1
    $ cmake ..
    $ make -j4
    $ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git
    $ mkdir na64sw.build
    $ cd na64sw.build
    $ cmake ../na64sw -DCMAKE_BUILD_TYPE=Debug
    $ make -j4
    $ cd ..

This will yield a NA64SW utils and libraries available at ``./na64sw.build``.

Local Build on Ubuntu
~~~~~~~~~~~~~~~~~~~~~

Full set of commands on Ubuntu, for instance. Will install basic
system deps, ROOT by ``snap``, then manually fetch, configure and install
RAVE, GenFit, p348-daq and NA64SW. Packages supporting install targets will
be configured and built by ``<prefix>/var/src``, packages that don't will be
placed at ``<prefix>/opt``. Assuming ``$NA64SWPRFX`` is your local
prefix (say, ``/home/johndoe/projects/na64/software``):

.. code-block:: shell

    $ export NA64SWPRFX=$HOME/local-build  # or wherever you would like to deploy it
    $ sudo apt install gfortran libtbb2 libexpat1-dev cmake libyamlcpp-dev \
            liblog4cpp5-dev liburiparser-dev libgsl-dev libeigen3-dev \
            libglew2.1 libgl2ps1.4 libftgl2 libafterimage-dev \
            libclhep-dev lib-gtest-dev libboost-dev
    $ sudo snap install root-framework
    $ sudo apt install
    $ mkdir -p $NA64SWPRFX/var/src
    $ cd $NA64SWPRFX/var/src
    $ curl -LJO https://rave.hepforge.org/downloads/?f=rave-0.6.25.tar.gz
    $ tar xvvf rave-0.6.25.tar.gz
    $ cd rave-0.6.25
    $ CXXFLAGS=-std=c++11 ./configure --disable-dependency-tracking --prefix=$NA64SWPRFX --disable-java
    $ make -j4 install
    $ cd ..
    $ curl -LJO https://gitlab.cern.ch/-/snippets/2145/raw/master/GenFit2.patch
    $ git clone https://github.com/GenFit/GenFit.git
    $ cd GenFit
    $ git apply --whitespace=fix ../GenFit2.patch
    $ mv cmake/FindROOT.cmake cmake/FindROOT.cmake.bck
    $ mkdir build
    $ cd build
    $ RAVEPATH=$NA64SWPRFX cmake .. \
        -DCMAKE_INSTALL_PREFIX=$NA64SWPRFX -DBUILD_TESTING=OFF \
        -DINCLUDE_OUTPUT_DIRECTORY=$NA64SWPRFX/include/genfit \
        -DRave_DIR=$NA64SWPRFX/share/rave/ -DCMAKE_CXX_FLAGS=-std=c++17
    $ find . -type f -print0 | xargs -0 sed -i 's/root-framework\/[[:digit:]]\+\//root-framework\/current\//g'
    $ make -j4 install
    $ cd $NA64SWPRFX/var/src
    $ git clone -b tracking-tools-nonuniformfield-999-0.install-targets https://gitlab.cern.ch/P348/tracking-tools.git
    $ mkdir tracking-tools/build
    $ cd tracking-tools/build
    $ cmake .. -DCMAKE_INSTALL_PREFIX=$NA64SWPRFX -DGENFIT_INCLUDE_DIR=$NA64SWPRFX/include/genfit/ -DGENFIT_LIBRARY=$NA64SWPRFX/lib/libgenfit2.so
    $ find . -type f -print0 | xargs -0 sed -i 's/root-framework\/[[:digit:]]\+\//root-framework\/current\//g'
    $ make -j4 install
    $ cd $NA64SWPRFX/opt
    $ git clone -b dev/install-targets-compat ssh://git@gitlab.cern.ch:7999/P348/p348-daq.git p348-daq.current
    $ cd p348-daq.current
    $ ./build.sh
    $ cd p348reco
    $ ln -s ../../.. tracking-tools
    $ ln -s ../../.. GenFit
    $ make  # will fetch and build common calibration files
    $ cd $NA64SWPRFX/var/src
    $ git clone ssh://git@gitlab.cern.ch:7999/P348/na64sw.git
    $ mkdir na64sw.build
    $ cd na64sw.build
    $ cmake ../na64sw -DCMAKE_INSTALL_PREFIX=$NA64SWPRFX \
        -DROOT_DIR=/snap/root-framework/current/usr/local/cmake
    $ find . -type f -print0 | xargs -0 sed -i 's/root-framework\/[[:digit:]]\+\//root-framework\/current\//g'
    $ make -j4 install

Note, that it is not necessary to keep the ``$NA64SWPRFX/opt/na64sw`` sources
directory after installation. It is extremely useful if you plan to modify the
core library (for instance, event structure) and in this case you may not need
an installation step, actually.

Optionally, install ``millipede`` with:

.. code-block:: shell

    $ git clone --depth 1 --branch V04-11-02 \
        https://gitlab.desy.de/claus.kleinwort/millepede-ii.git millipede
    $ cd millipede
    $ make PREFIX=$(readlink -f ../..) install

Dependencies
------------

We'll cover here dependencies in details. All of them must be available on the
target system in their "development" version. For RHEL (Fedora or RedHat Linux)
it is usually pkg name + "-devel" suffix installed with ``dnf`` or ``yum``, for
Debian-based (Ubuntu flavours) it is usually pkg name + "-dev" suffix and
installed with ``apt``. Gentoo distro will provide any package with its headers
as usual (with ``emerge``).

Mandatory vendor-specific dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

* yaml-cpp_ -- a YAML parser and emitter.
  Classes provided by this library is ubiquitously used to perform run-time
  configuration of calibration data, handlers, data allocators, etc. There are
  plans to either replace it, or to hide with some additional interfacing API
  to trace down YAML errors in a more convenient way.
* log4cpp_ -- a mature C++ message log
  library. This lib provides thread-safe and extensible capabilites for fast
  logging. It is lightweight, very mature, though some parts of it seems to be
  incompatible with recent compiler versions.
* CMake, binutils and GCC/clang are needed to build the project.

.. _yaml-cpp: https://github.com/jbeder/yaml-cpp
.. _log4cpp: http://log4cpp.sourceforge.net/

Note on ``log4cpp`` versions
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

As for motivation on custom deployment on LXPLUS: ``log4cpp`` has ``throw()`` qualifiers
and ``auto_ptr`` type that were deprecated in C++14 and were completely removed
from C++17. LCG releases of higher versions require ROOT to be used in >=C++17
because of ``string_view`` so we have to either get rid of log4cpp or stick
to C++14 or provide custom build with `orocos flavour`_ of log4cpp package
which has this bug being fixed.

.. _orocos flavour: https://github.com/orocos-toolchain/log4cpp

Optional Dependencies
~~~~~~~~~~~~~~~~~~~~~

These dependencies provide additional features.

* CERN's ROOT_ data analysis framework is widely used by
  standard handlers to depict the data with histograms
* A GenFit_ for track reconstruction
* CERN's Geant4_ framework used to simulate
  various setups of the experiment
* ZeroMQ_ provides network communication
* Google's unit testing framework, the gtest_ provides unit testing facility
* uriparser_ C library -- a RFC 3986 compliant
  parsing and handling lib
* A Doxygen_ processor to extract the documentation from C++ sopurce code and
  Sphinx_ + its Breathe_ extension to render it (note that you barely need it
  since the latest version is `available online`_).

.. _ROOT: https://root.cern.ch
.. _GenFit: https://github.com/GenFit/GenFit
.. _Geant4: https://geant4.web.cern.ch/
.. _ZeroMQ: https://zeromq.org/
.. _gtest: https://github.com/google/googletest
.. _uriparser: https://uriparser.github.io/
.. _Doxygen: http://www.doxygen.nl/
.. _Sphinx: https://www.sphinx-doc.org/en/master/
.. _Breathe: https://breathe.readthedocs.io/en/latest/
.. _available online: https://na64sw.docs.cern.ch/

Also, there are some dependencies from NA64 software ecosystem that are not
shipped within this project.

Obtaining the Source Code
-------------------------

For our task we typically need two repositories: ``na64sw`` and official NA64
reconstruction sources (``p348reco-daq``).

First, fetch this repository itself:

.. code-block:: shell

    $ git clone https://bitbucket.org/CrankOne/na64-ecal-edu.git ecal-edu

For raw data reading of the NA64, the ``DaqDataDecoding`` library is an essential
piece. It is a part of COMPASS (NA58, CERN, SPS) experiment software. The
initial bundle managed by NA64 team can be obtained with:

.. code-block:: shell

    $ git clone https://gitlab.cern.ch/P348/p348-daq.git

Most of the NA64SW utils may operate without this package, though practical
usage would be then mostly restricted by some Monte-Carlo analysis.

.. todo::

    Fetching the calibrations.

Configuring the source code
---------------------------

.. todo::

    Document options for third-party software, debug and release builds, etc

Building the code
-----------------

.. todo::

    Document code-generation stages, cpu usage, refer
    to [project install structure](@ref na64sw-install-structure).

Installation
------------

.. todo::

    document prefix-based installation stage


Notes on CI/CD
--------------

Shortest way to assemble image for CI/CD on gitlab is to develop it using
VM on OpenStack. Start with, say, CC7 image and do:

.. code-block:: shell

    yum update
    yum install -y yum-utils

    yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
    yum makecache fast
    yum install -y docker-ce

    service docker start
    docker ps # should output empty list of containers

    cd wherever/lies/dockerfile
    docker build . -f Dockerfile

To perform local build of packages compatible with LCG environment we need to
get access to CVMFS inside a (CC7) container. To setup it on VM:

.. code-block:: shell

    yum install locmap
    locmap --enable cvmfs