Catch2 Test Generator for MagAOX Flatlogs

Background

The MagAO-X ecosystem is largely based on C++. Type-safe logging requirements play a key role in the ecosystem's robustness. The MagAO-X logging system is based on Flatfile (created by Facebook), and is controlled through the Flatlog classes within MagAO-X. Individual entries within Flatlogs may each contain different data and be of a different type. The types are defined as C++ classes generated from the Flatfile configuration. Lastly, MagAO-X installations may each utilize different classes for their logging needs.

While Flatlogs is an excellent method of logging variadic data of multiple types, testing the flatlog classes is difficult. Entry classes need to be tested singularly with generated data, and also need to be tested together in longer sequences of logs.

To this end, the Catch2 Test Generator will automatically review the class definitions in the Flatfile .fbs schema files. Once analyzed, the generator will create singular C++ tests using Catch2, compile, run, and report their results. In addition, the generator will also generate a combined test of multiple "random" types and compile, run and report their results.

Getting started

The test generator is composed of two files, generateTemplatedCatch2Tests.py and catch2TestTemplate.jinja2. When run, Catch2 test files for each flatlog type will be created.

How To Run

Pre-requisites

Python must be installed with version >= 3.9. This is checked by the script. (As of this writing, the generator has been tested with Python 3.12.)
jinja2 must be installed. The script will check for jinja2 and automatically install it if not present. It can be installed manually with the following command:

python3 -m pip install Jinja2

It is assumed that any libraries and dependencies for the flatlog types have already been installed.

Generating the tests

The generator can be run by either running the python script directly, or calling the Makefile target. They are functionally equivalent.

To run the generator, do:

python3 ./generateTemplatedCatch2Tests.py

or

make generated_tests

Upon success, a folder titled generated_tests will be created, containing generated tests with the title <log_type>_generated_tests.cpp.

Running the tests

To build and run the tests, do:

make do_catch2_test

If the generated test files are not present, they will be generated by the Makefile.

Notes/Caveats:

At the start of the python script, the previous generated test files will be deleted. Because random values are used for test values, the values will change between runs of the generator. A random seed can be provided as a command line option with -s. For example:

python3 ./generateTemplatedCatch2Tests.py -s <seed>

To have incrementing test values, use the -i option:

python3 ./generateTemplatedCatch2Tests.py -i

If no option is provided, the generator defaults to randomly generated values.
To handle subtle differences in field names in the .fbs and .hpp files, the generator reads field names from both .fbs and .hpp names and uses the correct name when appropriate. The caveat to this is that the order in which those names appear MUST correspond between the two files.
This script detects a 'base' type if it does not have eventCode and defaultLevel in the .hpp file. It is noted in these log types that they cannot be used directly. The base types found are:
- empty_log
- flatbuffer_log
- software_log**
- string_log
- saving_state_change (not explicitly noted, but inferred)
It is assumed that these base types do not explicitly require tests, but will be tested through their children types.

**software_log is a special case since the base type is contained in the same file as its child types. The template uses LOG_NOTICE for all software logs.
Tables can be nested in the root-type table in .fbs files. However, the sub-tables must be must be defined before the _fb table definition. See telem_stdcam.fbs for a working example. It is expected that the title of the root-type table has the suffix '_fb'.
The generated_tests folder will be deleted in addition to the default clean target with make really_clean.
If field types of the .fbs file and the .hpp file don't match exactly, then the generator will use the .hpp field names in the generator. This is only used for software_log, because the amount of fields vary. In other words, if the amount of fields is different between the .fbs and .hpp file, the field names MUST match.

Entropy Test Generator

In order to test flatlogs further, 'entropy' tests can be generated. These tests consist of a number of flatlogs being created with random values, and then each field is verified one-by-one.

The entropy generator is composed of entropyTestTemplate.jinja2 and generateEntropyTests.py. Both files should be in the same directory as types/.

How to Run

Pre-requisites

The Catch2 Test Generator described above must be run before entropy tests can be generated, since entropy tests include these test files.

The easiest way to generate, compile, and run the entropy test is through the makefile target:

make do_entropy_test <OPTIONS>

The generated file is written to gen_entropy_tests/generated_test_e<ENTROPY>_n<NTYPES>.cpp

For example, if e=3 and n=12, the cpp file is titled generated_test_e3_n12.cpp

and the test executable is generated_test_e3_n12.

Options

s : random seed
n : number of distinct flatlog types to use.
e : entropy level. The number of total flatlogs that will be tested is e * n
t : specific types of flatlogs to use, in quotes, comma-separated. The types will be randomly selected if this option is not used.

Note: It is not required to provide these options on the commandline. Their default values are stored as variables in the makefile, and can be set there as well.

Example usage

make do_entropy_test

make do_entropy_test s=123 n=25 e=2

make do_entropy_test n=5 e=5 t="config_log, git_state"

Notes/Caveats

If a test is generated with options e and nthat already exist in gen_entropy_tests/, the existing file will be overwritten.
The makefile creates the compiled executable generated_entropy_test_e<E>_n<N>. Because this file includes other Catch2 tests, they all will be run when the executable is run. In order to run just the entropy test, you must specify the scenario:

./generated_entropy_test_e<E>_n<N> "Scenario: test_e<E>_n<N>"