This was an old post of mine that was on IBM QRadar's developerworks. The post seems to be gone, so I'm posting here for an archive. This is probably out of date, and would not work anymore (written in 2017)

(Cross-Posted from my DeveloperWorks post on https://developer.ibm.com/qradar/2017/01/27/running-background-services-in-qradar-apps/)

The QRadar Application Framework allows you to run complex applications within a QRadar environment. By default, the framework creates and manages a Flask web application instance for you. For a lot of apps, that is all that is required to build a rich and fully-functional QRadar integration. Sometimes, you want to run background services in your application. Maybe it's a database, a background processing script, or a complex machine learning engine. Getting your application set up to install those services is pretty easy once you get the hang of it!  NOTE: In QRadar 7.3.0 and up, the correct way to do this is through Named Services.

The QRadar Application Framework uses supervisord as a process control system. supervisord manages your flask server by default, but it also allows you to create configuration files for all the services that you would like to run and will manage starting those instances when your application starts. supervisord will also manage restarting any service that happens to crash and allows for you to define logic around how to restart services that you would like to keep running. Before we dive into adding your own background services, there are a couple of files and conventions you should be aware of:

  • In your app, anything in src_deps/init/ordering.txt will be executed first before anything else happens in your application. If you need to copy any configuration files or run any commands, ordering.txt is where you define your required actions. Each line in this file is read as a shell command. This means that anything you can do in the shell you can do in the file.
  • If you want to include arbitrary files in your container, just drop them in to the file path src_deps/init and you can move them where you want to using the src_deps/init/ordering.txt file.
  • Any dependencies you need to have installed for your background service (including your background service itself) must be in the src_deps/rpms folder. The QRadar Application framework runs on CentOS 6.6 in the following QRadar versions: 7.2.6, 7.2.7, and 7.2.8. Therefore, your services must be compiled for Centos 6.6 and you should also include the entire dependency tree of your services as .rpms. This ensures that everything your service needs to run is installed.

For this tutorial, let's create an instance of RabbitMQ that runs in our container. RabbitMQ is a popular message broker that can manage sending messages between services you may have. This is great for designing an application that contains many background services that may need to connect with each other.

Step 1 - Create config files

The first thing you need to do is create a supervisord configuration file for each background service you would like to start. These files are created in the src_deps/init folder of your application. You can see all the configuration options for a supervisord program in their docs. Let's create a configuration for RabbitMQ: src_deps/init/rabbitmq.conf

[program:rabbitmq]
command=/bin/sh /opt/rabbitmq.sh
user=root
autostart=true
autorestart=true
HOME="/var/lib/rabbitmq/",
RABBITMQ_PID_FILE="/var/run/rabbitmq/pid"

The configuration above is pretty self-explanatory. First, we define a command to run when this service starts - /bin/sh /opt/rabbitmq.sh. This shell script is pretty simple, as it starts rabbitmq and handles any restarts or crashes that may happen. We want our RabbitMQ service to start automatically and restart if it crashes. We set a home directory for the data to live in, then set a file where the PID for RabbitMQ is written. That way if we need to find the service later, we can do so without any complicated logic. Let's take a look at our /opt/rabbitmq.sh:

#!/usr/bin/env bash
# call "rabbitmqctl stop" when exiting
trap "{ echo Stopping rabbitmq; rabbitmqctl stop; exit 0 }" TERM

echo Starting rabbitmq
rabbitmq-server &

# from docs: When Bash receives a signal for which a
# trap has been set while waiting for a command to
# complete, the trap will not be executed until the
# command completes.
#
# This is why we use & and wait here. Idea taken from:
# http://veithen.github.io/2014/11/16/sigterm-propagation.html
PID=$!
wait $PID

Pretty easy to understand! Now that we have our configuration file and our shell script to start RabbitMQ, let's make sure they go in the right location!

Step 2 - Copy config files into the right location

Now, we have created our configuration file (.conf) for RabbitMQ and the shell script (.sh) to start it. These two files are in src_deps/init. This file gets mounted inside the container to /src_deps/init. Let's copy these files into our container in a more logical location in our filesystem. We can do this by adding some commands in src_deps/init/ordering.txt:

cp /src_deps/init/rabbitmq.conf /etc/rabbitmq.conf
cp /src_deps/init/rabbitmq.sh /opt/rabbitmq.sh

Step 3 - Alert supervisord to our new service

Now, let's extend the default QRadar supervisord configuration to include our own services. Just like before we are going to create our configuration in src_deps/init, then move it to where it needs to be later: src_deps/init/supervisord.conf

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)

[supervisord]
logfile=/store/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB       ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10          ; (num of main logfile rotation backups;default 10)
loglevel=info               ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=true             ; (start in foreground if true;default false)
minfds=1024                 ; (min. avail startup file descriptors;default 1024)
minprocs=200                ; (min. avail process descriptors;default 200)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket

[program:startflask]
command=/start_flask.sh
autorestart=true

[include]
files = /etc/rabbitmq.conf

You'll notice we added an include section to the bottom where we pointed to where our rabbitmq.conf file will end up in our container. Let's then add a line to our src_deps/init/ordering.txt to move this new supervisord configuration file into place:

cp /src_deps/init/supervisord.conf /etc/supervisord.conf

Step 4 - Add rpms to your application

The final step is adding all the rpms to src_deps/rpms. You can grab the rpms for RabbitMQ compiled for Centos 6.6 here: https://ibm.box.com/s/my0p9ebtz0gm4s3chyqnjvb3w5n9fa16

Conclusion

You're now set up to run a background service in your QRadar application! You can pretty easily extrapolate the instructions here for any background service that you want to run. There are a couple important tips to keep in mind though:

  1. Your background services, if they listen on ports, are only accessible from within your application
  2. Each background service you run consumes memory. Keep your applications as slim as you can!