redis_publish_perf

redis_publish.py implemented in C++ with the intention of minimizing time delays from code execution.

Build

Clone redis_publish_perf:

git clone --recurse-submodules https://github.com/mitre/redis_publish_perf.git
cd redis_publish_perf

Hiredis

Hiredis is required to run redis_publish_perf. Install Hiredis as follows:

cd hiredis
make
sudo make install

redis_publish_perf

Build redis_publish_perf:

cd ..
make

Run

redis_publish_perf takes a required path to a file containing Intermediate Log Format (ILF)s as an argument in addition to optional arguments for configuring its behavior.

Run redis_publish_perf with examples/seconds.ilf:

./redis_publish examples/seconds.ilf

Delay for 3x the time difference between ILF timestamps:

./redis_publish examples/seconds.ilf -m 3

The -m argument multiplies the time difference between ILFs (e.g., if -m is 2, double the delay time, or if -m is 0.5, split the delay time in half).

Override ILF timestamps and delay for 100 milliseconds between ILFs:

./redis_publish examples/nanos.ilf -d 100ms

The -d argument is formatted as a positive integer followed by a unit of time. Valid units of time include s, ms, us, and ns. If no unit of time is provided, then seconds is used by default.

Print help message:

./redis_publish -h

Considerations

redis_publish_perf was designed to minimize the delay of code execution and better preserve the fidelity of ILFs originally as they were seen. However, depending on the time difference between ILFs in a given file, this fidelity may be worse or better preserved. Specifically, if you observe that the time difference between ILFs is on the scale of tens of microseconds or less, then redis_publish_perf may introduce delays (also on the scale of tens of microseconds).

There may be some room for reducing these delays in the future, but unfortunately, the process of publishing to Redis is relatively costly, so any optimizations will likely be moot unless said optimizations apply directly to improving communication with Redis.

Additionally, when dealing with time differences on the scale of microseconds, there are certain factors out of our control that may introduce delays. Firstly, system clocks can suffer from jitter and latency, which will add deviations on the scale of microseconds to measured time. Also, we are, to an extent, at the mercy of the scheduling algorithm of the operating system. It could well be that our instance of redis_publish_perf gets knocked off a core and doesn't get executed for several milliseconds. You may be able to mitigate the effects of scheduling with chrt:

sudo chrt -f 99 ./redis_publish <args>

-f will change the scheduling policy to FIFO, and 99 is the maximum priority for the FIFO policy. See more about the FIFO scheduling policy in sched(7).

If your system fails to find Hiredis when running redis_publish_perf with chrt, then run the following:

sudo LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH chrt -f 99 ./redis_publish <args>

Another lesser note worth mentioning is that redis_publish_perf may not be able to interpret timestamps in all ILFs. Below are the currently supported timestamp formats:

• Unix epoch, which may be represented in seconds, milliseconds, microseconds, or nanoseconds (e.g., 1645812608).
• Floating point Unix epoch represented in seconds (e.g., 1645812608.825).
• ISO 8601, which must be formatted as YYYY-MM-DD hh:mm::ss, where ss may be either an integer or floating point number.

License

This software is licensed under the Apache 2.0 license.

Public Release

Note

Approved for Public Release; Distribution Unlimited. Public Release Case Number 24-3939.