.. _troubleshooting:

Troubleshooting
===============

**(lxlplus) terminal malfunction after SIGTERM (ctrl+c)**

After interrupting ``na64sw-pipe`` with ``SIGTERM`` signal, terminal turns into
the "blind" state -- symbols typed on keyboard do not appear in the command
line.

To cure it one can (blindly) type a command:

.. code-block:: shell

    $ stty sane

That will re-set all the current command shell options including one
responsible for echoing symbols.

Particular reason of this trouble is not known. Apparently, one of the
libraries plays around with ANSI escape sequences. For the time only LXPLUS
builds seem to be affected.

**No network connection during docker build invocation**

On some systems, during execution of ``$ docker build -t hepfarm4ecal-edu .``
command an error may appear of the form: 

.. code-block:: shell

    !!! Error fetching binhost package info from 'http://hepsoft.crank.qcrypt.org/amd64.opt.20200222/' !!! <urlopen error [Errno 110] Connection timed out>

Indicating that build container is failed to fetch the pre-compiled package
from remote repository. Try to provide ``docker build`` command with the
``--network=host`` to disable docker network bridging during the build to
overcome this troble:

.. code-block:: shell

    $ docker build -t hepfarm4ecal-edu . --network=host

Alternatively, you might need to finely tune your docker networking. However,
this behavious seems to not affect the running container, so this
straightforward approach may be a better solution.

**An error with "Fix map file problem(s) first" prevents from reading raw data**

Verify that the data source configuration file provided to whatever application
is in use (usually with ``-i,--input-cfg``) provides correct value for parameter
named ``CORALMapsDir``.

**Annoying error messages of missing file entries appear on startup**

Messages of the form

.. code-block:: shell

    Error in cling::AutoLoadingVisitor::InsertIntoAutoLoadingState:
       Missing FileEntry for GblTrajectory.h
       requested to autoload type gbl::GblData

are related to known issue which ROOT `developers expected`_ to get rid only in 2019".
Nevertheless this messages most probably refer to GenFit headers and may
be cured for container with (consider to add it to collector's ``.bashrc``)

.. _developers expected: https://root-forum.cern.ch/t/error-in-cling-insertintoautoloadingstate/29347

.. code-block:: shell

    export ROOT_INCLUDE_PATH=/usr/include/genfit

and equivalent measure for other environments.

**Can't run app on production host because of libGLU failures**

If one can't run built apps on the OpenStack node (like, say, ``na64-dev``) or
in Docker container becuase of OpenGL missing libraries, e.g.:

.. code-block:: shell

    /afs/cern.ch/work/r/rdusaev/public/na64/sw/LCG_101/x86_64-centos7-gcc11-opt/bin/na64sw-ddd-pick-event-server: error while loading shared libraries: libGLU.so.1: cannot open shared object file: No such file or directory

Consider installing following packages:

.. code-block:: shell

    $ yum install mesa-libGLU libglvnd-devel

Even for simple applications, like event picking server these libraries might
still be required because of NA64sw util library tightly bound to cohesive
ROOT libs. Perhaps, this is to be be fixed at some point...

**Can't run app on production host because of ROOT discovery failures**

If one can't run built apps on the OpenStack node (like, say, ``na64-dev``)
becuase of obscure ROOT message saying something like:

.. code-block:: shell

    error: entry with relative path at the root level is not discoverable
    { 'name': '', 'type': 'directory',
          ^~
    Error in modulemap.overlay!

Consider installing following `deps on the node`_:

.. _deps on the node: https://github.com/root-project/root/issues/8157

.. code-block:: shell

    $ sudo yum install git make cmake3 gcc-c++ gcc binutils libX11-devel libXpm-devel libXft-devel libXext-devel python openssl-devel

It is unclear what confuses ROOT in particular.