Syslog To Intermediate Log Format (ILF)

The Syslog to ILF translator converts unix generated syslogs or audit logs to ILF so it can be processed by HBL. The translator has various features, such as being able to read from a file or a pipe, being able to parse a directory of logs, and the ability to condense auditd logs to consolidate events.

The syslog translator was designed to handle the standard headers provided by the syslog and audit log daemons. If you are looking to translate logs that do not use these headers or are incompatible, you can enable the string parsing feature the syslog translator has. This feature removes the explicit header matching expression, and uses one regex to parse the entire log. See more information in the string parsing and Log Headers sections.

Getting Started

clone the repo and its submodules

git clone https://github.com/mitre/syslog_to_ilf
cd syslog_to_ilf
git submodule update --init --recursive

you can check proper installation of the submodules by listing the contents of the directory.

$ ls lib/libilf
README.md                    readerwritercircularbuffer.h
atomicops.h                  readerwriterqueue.h
ilf.h                        resources
parser.h                     test

---

Installation

The hi-redis and redis-plus-plus dependencies can be installed locally by running the ./build-deps.sh script. Please make sure to check the output for any errors, as the binaries are being built from source, and have not been tested on every system.

Curl has been added to the list of dependencies to support publishing of ILF to couchdb. See the installation guide for a more information regarding installing the pre-requisite libraries.

Building the Code

There are two pre-processor directives that can be specified when building cmake to alter translator function. They are:

USE_TLS - enables TLS encrypted publishing for redis. STRING_PARSE - Disregards the "logtype" and makes no assumption on log headers.

To enable either of these options, pass them as arguments to the cmake call.

Note: Using TLS mode requires openssl to be installed and the dependencies to be built for TLS!. See the TLS section.

For more information on using the STRING_PARSE mode see the string parse configuration section.

cmake -S . -B build -DUSE_TLS=ON -DSTRING_PARSE=ON

Note these are optional arguments, for normal operation they can be left out.

cmake -S . -B build

After building with cmake, we can generate the executables with make

make -C build -j $(nproc)

---

Run the translator

Since the translator installs libraries locally, the paths for these libraries are in "non-default" locations for the linker, which tries to resolve these paths at run-time. This can be fixed by setting the $LD_LIBRARY_PATH environment variable, which is the primary purpose of why a run script was created. This solves the issue of having to define env vars before invoking the binary. Any command line arguments passed to the start script will be forwarded to the translator executable.

./startTranslator.sh

To better support its various use-cases, the syslog translator uses argument parsing to determine what action it should take. Currently these are the existing valid arguments:

-m      mode:      "file" or "pipe"
-l      log type:  "syslog" or "audit"
-s      source:    path/to/valid/data/source (can be file, pipe, or a directory)
-o      output:    output_file_name (The output of the translator is redirected 
                   from publishing to redis to writing to a file when an
                   output path is specified.)
-d      directory: Enables reading of directory which contains audit.log's 
-c      combine:   Enables the combination of audit logs from auditd
-b      baseline:  Puts the translator in baseline mode. The translator
                   will look to parse a directory of audit log files you 
                   specified with the source option (uses threads).
-u      UUID:      Assign a RFC4122 stringified UUID to the translator.
                   This ID will get published in the translator heartbeat
                   messages.
-t      tag:       Give the translator a name to be printed out while doing
                   its translation.

example execution with argument's:

./startTranslator.sh -m file -l syslog -s /var/log/messages -d -c -o output.ilf

Providing a large list of command line arguments can be annoying when running the translator repeatedly with minor tweaks, or having to type the command from scratch. For this reason the config.json file was created to act as the 'default' parameters passed to the translator at runtime. Command line arguments are still in effect, and will override these default configuration values.

Top priority	Second Priority
Command-line arguments	config.json

---

TLS / SSL

The translator and its dependencies can be built with TLS support for publishing to redis.

The following commands will ensure the translator is ready to run in TLS mode:

# Build the dependant libraries in TLS mode
./build-deps.sh TLS

# Build the translator in TLS mode
cmake -S . -B build -DUSE_TLS=ON
make -C build -j $(nproc)

# Now you are ready to start the translator. Make sure
# "CACERT" in the cfg/config.txt contains the path to the ca certificate.
./startTranslator.sh

Testing

After following the build steps, we can run the translator_tester executable with the proper environment variables set by calling the runTest.sh script.

Note: if you do not have gtest installed, this executable will not be generated.

./runTest.sh build/translator_tester

---

Configuration

There exists multiple configuration files for the Syslog To ILF translator that live in the cfg/ directory.

File	Description
config.txt	- Supply redis information - Filter logs
config.json	A repeatable way to set command-line options for different translate modes and operation
attribute_config_*.json	The way in which the translator can transform a log into ILF. Fields from syslogs or audit logs are mapped to user defined fields with the help of REGEX expressions.

The second configuration file is called attribute_config.json which the translator uses to parse the logs. The json file contains a json array of json elements, each element have keys and values that contain information about a specific syslog log we are trying to translate, most notably the regex key. The given value for the regex key is a regex expression which the translator will look to detect a match for a given log. If a match occurs, the groups contained by parentheses () are extrapolated and assigned names based on the attribute names field. It is important for the number of attribute names to match the number of groups in the regex expression otherwise an error will occur.

note: The regex expression in the json file requires an additional escape character (\) as the expression is contained in quotes and entered as a string.

Profiling

using perf

# record CPU cycles of the translator
sudo perf record -g ./syslog_translator

# change permissions so you can read it
sudo chmod 666 perf.data

# display the results
perf report -g

Troubleshooting

No such file or directory "ilf.h"

This is a simple issue caused by the submodules not getting cloned properly.

Please try updating the submodules to remedy this issue.

git submodule update --init

No such file or directory "<curl/curl.h>"

This error occurs when the build system cannot find the curl header file on the system. Please check that curl is installed in the system. See install curl for instructions.

Error trying to build in TLS mode.

make                                                                      ─╯
[  9%] Building CXX object CMakeFiles/SyslogLib.dir/src/syslog_translator.cpp.o
/Users/cniebling/Documents/SEAL/repos/syslog_to_ilf/src/syslog_translator.cpp:64:39: error: no member named 'enabled' in 'sw::redis::tls::TlsOptions'
        _redis_connection_options.tls.enabled = true;
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
/Users/cniebling/Documents/SEAL/repos/syslog_to_ilf/src/syslog_translator.cpp:65:39: error: no member named 'cacert' in 'sw::redis::tls::TlsOptions'
        _redis_connection_options.tls.cacert = "/Users/cniebling/Documents/SEAL/repos/syslog_to_ilf/self-signed.hbl-testenv.mitre.org.crt";
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
2 errors generated.
make[2]: *** [CMakeFiles/SyslogLib.dir/src/syslog_translator.cpp.o] Error 1
make[1]: *** [CMakeFiles/SyslogLib.dir/all] Error 2
make: *** [all] Error 2

This error is indicating that the TLS specific API calls are invalid. There could, be two major things causing this. The hiredis and redis++ dependencies were, not built in TLS mode. Or, CMake is finding another hiredis / redis++ installation on the system before it finds the libraries in the custom path. To fix this, you can try specifying -DCMAKE_PREFIX_PATH=/your/path while calling cmake for the translator's build. Or you could try removing the system install if you have the ability to do so.

License

This software is licensed under the Apache 2.0 license.

Public Release

Note

Approved for Public Release; Distribution Unlimited. Public Release Case Number 24-3939.