Switching Data Sources
======================

This section is devoted to three major data sources available in NA64SW
out of the box: default (DDD) data source, the MK/MC data format, and wrapper
around ``p348reco`` library.

A foreword on runtime extensions
--------------------------------

NA64sw was designed to be extensible at a runtime. That means the user
can add certain functionality without re-building the whole thing. Foreseen
*extension points* at which the user can extend NA64SW are:

* Event handlers (i.e. sub-programs that processes events)
* Data source (a sub-program that reads events to be propagated through the pipe)
* Calibration data types

We will cover some C/C++ internals in this tutorial later on, to demonstrate
how this can be implemented propgrammatically. For now the point is that the
input data format can be re-defined by some external module.

If you have read the command line reference of ``na64sw-pipe`` application
you've probably noted that:

* ``-m,--load-module`` option parameter is by default set to
  ``std-handlers,ddd``. This option defines the list of extension modules (also
  called *plug-ins*) loaded at the ``na64sw-pipe`` application startup
* ``-i,--input-cfg`` option parameter has default value like
  ``/some/where/source-ddd.yaml``. This option defines a configuration of the
  data source used by the application to interpret its input provided as
  positional command line argument(s).

By overriding these options one may switch to other than default (DDD) input
format and customize its configuration. Moreover, user can provide their own
extension module [#exts]_ to the pipeline application.

Default (DDD) data source
-------------------------

The DDD abbreviation stands for ``DaqDataDecoding`` library -- a standard
low-level data codec with C/C++ API, natively used for NA64 DAQ (inherited form
COMPASS).

This is our default input format, nothing special have to be done to make it
active.

It provides a very basic input for the treatment: pure DAQ digits. So,
initially an *event* does not have any data except for very basic values of
``rawData`` (like ``SADCHit/rawData``).

A reconstruction pipeline typically uses this DAQ digits information to
assemble clusters, calorimeter hits, track points and so on.

We've already considered some basic usage of the pipeline application with this
data source in previous chapters.

A wrapper over p348reco lib
---------------------------

The p348reco_ is a header-only reconstruction library used in
NA64, a somewhat official one. ``p348reco-wrapper`` interfaces this library
as a NA64SW's data source to provide acces to the reconstructed data.

.. _p348reco: https://gitlab.cern.ch/P348/p348-daq

Usage requirements:

* provide extension ``p348reco-wrapper`` to ``-m,--load-module`` option. This
  will make ``na64sw-pipe`` to *load* the module and make a wrapper available
  (it is not loaded by default).
* refer to its config (perhaps, by standard path) in ``-i,--input-config``
  option to make ``na64sw-pipe`` actually use the data source.

.. code-block:: shell

  $ na64sw-pipe -m p348reco-wrapper \
            -i $NA64SW_PREFIX/share/na64sw/source-p348reco.yaml \
            -r p348reco-example.yaml \
            /eos/experiment/na64/data/cdr/cdr01002-003292.dat

The ``p348reco-example.yaml`` pipeline config produces a set of plots similar
to one generated by the native p348reco's ``example.cc``.

For pipeline development it is important to
note that ``p348reco-wrapper`` data source is restricted to what the native
p348reco reconstruction routines generates as the input for their example
code -- e.g. *calorimeter hits* are not built as dedicated entities.
Practical usage of this source often implies common procedures that are
summed up in ``p348reco/appendix.yaml``. The set of plots is then defined
in ``p348reco/std-plots.yaml``. Both these configuration files are included
by ``p348reco-example.yaml``.

MK/MC data source
-----------------

MK/MC stands for "Mikhail Kirsanov's Monte-Carlo" data format. This is the
output format provided by `MC framework`_ commonly used in NA64. This format
is defined as a plain ASCII file with loose schema and variable sectioned
content.

.. _MC framework: https://gitlab.cern.ch/P348/na64-simulation/-/tree/master/geant4/simulation

It provides higher-level data (like calorimeter hits or track points) within
an event right from the "source". To use this data format on a file:

* provide extension ``mkmc`` to ``-m,--load-module`` option. This will make
  ``na64sw-pipe`` to *load* the module and make a format available
* refer to its config in ``-i,--input-config`` option to make ``na64sw-pipe``
  use this event data decoder

As follows (``-r`` and data file are given for instance):

.. code-block:: shell

    $ na64sw-pipe -m mkmc \
            -i $NA64SW_PREFIX/share/na64sw/source-mkmc.yaml \
            -r mkmc-example.yaml \
            /afs/cern.ch/work/m/mkirsano/public/simulation_data/CaloHits_W_mu150.d

.. todo::
    Add meaningful example to this section.
    If you have something practical and demonstrative, please contact me
    (Renat). I've tried some old (and illustrative) examples of
    ``na64-simulation`` and they seem to be broken for now (Mar, 2022).

.. rubric:: Footnotes

.. [#exts] Technically, an extension module is just a *shared object* file (``.so``
    lib in Linux) located in one of the paths listed in ``NA64SW_MODULES_PATH``
    environment variable. If this variable is not set, some default path is used.
    Typically it lists current dir (``.``) and an installation path
    (``/some/where/lib/na64sw-extensions/``). One may ``ls`` these paths to
    check out available extensions.