TDengine/tests/pytest/crash_gen

User's Guide to the Crash_Gen Tool

Introduction

To effectively test and debug our TDengine product, we have developed a simple tool to exercise various functions of the system in a randomized fashion, hoping to expose maximum number of problems, hopefully without a pre-determined scenario.

Features

This tool can run as a test client with the following features:

1. Any number of concurrent threads
2. Any number of test steps/loops
3. Auto-create and writing to multiple databases
4. Ignore specific error codes
5. Write small or large data blocks
6. Auto-generate out-of-sequence data, if needed
7. Verify the result of write operations
8. Concurrent writing to a shadow database for later data verification
9. User specified number of replicas to use, against clusters

This tool can also use to start a TDengine service, either in stand-alone mode or cluster mode. The features include:

1. User specified number of D-Nodes to create/use.

Preparation

To run this tool, please ensure the followed preparation work is done first.

1. Fetch a copy of the TDengine source code, and build it successfully in the build/ directory
2. Ensure that the system has Python3.8 or above properly installed. We use Ubuntu 20.04LTS as our own development environment, and suggest you also use such an environment if possible.

Simple Execution as Client Test Tool

To run the tool with the simplest method, follow the steps below:

1. Open a terminal window, start the taosd service in the build/ directory (or however you prefer to start the taosd service)
2. Open another terminal window, go into the tests/pytest/ directory, and run ./crash_gen.sh -p -t 3 -s 10 (change the two parameters here as you wish)
3. Watch the output to the end and see if you get a SUCCESS or FAILURE

That's it!

Running Server-side Clusters

This tool also makes it easy to test/verify the clustering capabilities of TDengine. You can start a cluster quite easily with the following command:

$ cd tests/pytest/
$ rm -rf ../../build/cluster_dnode_?; ./crash_gen.sh -e -o 3   # first part optional

The -e option above tells the tool to start the service, and do not run any tests, while the -o 3 option tells the tool to start 3 DNodes and join them together in a cluster. Obviously you can adjust the the number here. The rm -rf command line is optional to clean up previous cluster data, so that we can start from a clean state with no data at all.

Behind the Scenes

When the tool runs a cluster, it users a number of directories, each holding the information for a single DNode, see:

$ ls build/cluster*
build/cluster_dnode_0:
cfg  data  log

build/cluster_dnode_1:
cfg  data  log

build/cluster_dnode_2:
cfg  data  log

Therefore, when something goes wrong and you want to reset everything with the cluster, simple erase all the files:

$ rm -rf build/cluster_dnode_*

Addresses and Ports

The DNodes in the cluster all binds the the 127.0.0.1 IP address (for now anyway), and uses port 6030 for the first DNode, and 6130 for the 2nd one, and so on.

Testing Against a Cluster

In a separate terminal window, you can invoke the tool in client mode and test against a cluster, such as:

$ ./crash_gen.sh -p -t 10 -s 100 -i 3

Here the -i option tells the tool to always create tables with 3 replicas, and run all tests against such tables.

Additional Features

The exhaustive features of the tool is available through the -h option:

$ ./crash_gen.sh -h
usage: crash_gen_bootstrap.py [-h] [-a] [-b MAX_DBS] [-c CONNECTOR_TYPE] [-d] [-e] [-g IGNORE_ERRORS] 
    [-i NUM_REPLICAS] [-k] [-l] [-m] [-n]
    [-o NUM_DNODES] [-p] [-r] [-s MAX_STEPS] [-t NUM_THREADS] [-v] [-w] [-x]

TDengine Auto Crash Generator (PLEASE NOTICE the Prerequisites Below)
---------------------------------------------------------------------
1. You build TDengine in the top level ./build directory, as described in offical docs
2. You run the server there before this script: ./build/bin/taosd -c test/cfg

optional arguments:
  -h, --help            show this help message and exit
  -a, --auto-start-service
                        Automatically start/stop the TDengine service (default: false)
  -b MAX_DBS, --max-dbs MAX_DBS
                        Maximum number of DBs to keep, set to disable dropping DB. (default: 0)
  -c CONNECTOR_TYPE, --connector-type CONNECTOR_TYPE
                        Connector type to use: native, rest, or mixed (default: 10)
  -d, --debug           Turn on DEBUG mode for more logging (default: false)
  -e, --run-tdengine    Run TDengine service in foreground (default: false)
  -g IGNORE_ERRORS, --ignore-errors IGNORE_ERRORS
                        Ignore error codes, comma separated, 0x supported (default: None)
  -i NUM_REPLICAS, --num-replicas NUM_REPLICAS
                        Number (fixed) of replicas to use, when testing against clusters. (default: 1)
  -k, --track-memory-leaks
                        Use Valgrind tool to track memory leaks (default: false)
  -l, --larger-data     Write larger amount of data during write operations (default: false)
  -m, --mix-oos-data    Mix out-of-sequence data into the test data stream (default: true)
  -n, --dynamic-db-table-names
                        Use non-fixed names for dbs/tables, for -b, useful for multi-instance executions (default: false)
  -o NUM_DNODES, --num-dnodes NUM_DNODES
                        Number of Dnodes to initialize, used with -e option. (default: 1)
  -p, --per-thread-db-connection
                        Use a single shared db connection (default: false)
  -r, --record-ops      Use a pair of always-fsynced fils to record operations performing + performed, for power-off tests (default: false)
  -s MAX_STEPS, --max-steps MAX_STEPS
                        Maximum number of steps to run (default: 100)
  -t NUM_THREADS, --num-threads NUM_THREADS
                        Number of threads to run (default: 10)
  -v, --verify-data     Verify data written in a number of places by reading back (default: false)
  -w, --use-shadow-db   Use a shaddow database to verify data integrity (default: false)
  -x, --continue-on-exception
                        Continue execution after encountering unexpected/disallowed errors/exceptions (default: false)