Testing resource agents

10.1. Testing with ocf-tester

The resource agents repository (and hence, any installed resource agents package) contains a utility named ocf-tester. This shell script allows you to conveniently and easily test the functionality of your resource agent.

ocf-tester is commonly invoked, as root, like this:

ocf-tester -n <name> [-o <param>=<value> ... ] <resource agent>
  • <name> is an arbitrary resource name.
  • You may set any number of <param>=<value> with the -o option, corresponding to any resource parameters you wish to set for testing.
  • <resource agent> is the full path to your resource agent.

When invoked, ocf-tester executes all mandatory actions and enforces action behavior as explained in Section 5, “Resource agent actions”.

It also tests for optional actions. Optional actions must behave as expected when advertised, but do not cause ocf-tester to flag an error if not implemented.

ocf-tester does not initiate “dry runs” of actions, nor does it create resource dummies of any kind. Instead, it exercises the actual resource agent as-is, whether that may include opening and closing databases, mounting file systems, starting or stopping virtual machines, etc. Use with care.

For example, you could run ocf-tester on the foobar resource agent as follows:

# ocf-tester -n foobartest \
             -o superfrobnicate=true \
             -o datadir=/tmp \
Beginning tests for /home/johndoe/ra-dev/foobar...
* Your agent does not support the notify action (optional)
* Your agent does not support the reload action (optional)
/home/johndoe/ra-dev/foobar passed all tests

10.2. Testing with ocft

ocft is a testing tool for resource agents. The main difference to ocf-tester is that ocft can automate creating complex testing environments. That includes package installation and arbitrary shell scripting.

10.2.1. ocft components

ocft consists of the following components:

  • A test case generator (/usr/sbin/ocft) — generates shell scripts from test case configuration files
  • Configuration files (/usr/share/resource-agents/ocft/configs/) —  a configuration file contains environment setup and test cases for one resource agent
  • The testing scripts are stored in /var/lib/resource-agents/ocft/cases/, but normally there is no need to inspect them

10.2.2. Customizing the testing environment

ocft modifies the runtime environment of the resource agent either by changing environment variables (through the interface defined by OCF) or by running ad-hoc shell scripts which can for instance change permissions of a file or unmount a file system.

10.2.3. How to test

You need to know the software (resource) you want to test. Draw a sketch of all interesting scenarios, with all expected and unexpected conditions and how the resource agent should react to them. Then you need to encode these conditions and the expected outcomes as ocft test cases. Running ocft is then simple:

# ocft make <RA>
# ocft test <RA>

The first subcommand generates the scripts for your test cases whereas the second runs them and checks the outcome.

10.2.4. ocft configuration file syntax

There are four top level options each of which can contain one or more sub-options.

CONFIG (top level option)

This option is global and influences every test case.

  • AgentRoot (sub-option)
AgentRoot /usr/lib/ocf/resource.d/xxx

Normally, we assume that the resource agent lives under the heartbeat provider. Use AgentRoot to test agent which is distributed by another vendor.

  • InstallPackage (sub-option)
InstallPackage package [package2 [...]]

Install packages necessary for testing. The installation is skipped if the packages have already been installed.

  • HangTimeout (sub-option)
HangTimeout secs

The maximum time allowed for a single RA action. If this timer expires, the action is considered as failed.

SETUP-AGENT (top level option)
  bash commands

If the RA needs to be initialized before testing, you can put bash code here for that purpose. The initialization is done only once. If you need to reinitialize then delete the /tmp/.[AGENT_NAME]_set stamp file.

CASE (top level option)
CASE "description"

This is the main building block of the test suite. Each test case is to be described in one CASE top level option.

One case consists of several suboptions typically followed by the RunAgent suboption.

  • Var (sub-option)
Var VARIABLE=value

It is to set up an environment variable of the resource agent. They usually appear to be OCF_RESKEY_xxx. One point is to be noted is there is no blank by both sides of “=”.

  • Unvar (sub-option)

Remove the environment variable.

  • Include (sub-option)
Include macro_name

Include statements in macro_name. See below for description of CASE-BLOCK.

  • Bash (sub-option)
Bash bash_codes

This option is to set up the environment of OS, where you can insert BASH code to customize the system randomly. Note, do not cause unrecoverable consequences to the system.

  • BashAtExit (sub-option)
BashAtExit bash_codes

This option is to recover the OS environment in order to run another test case correctly. Of cause you can use Bash option to recover it. However, if mistakes occur in the process, the script will quit directly instead of running your recovery codes. If it happens, you ought to use BashAtExit which can restore the system environment before you quit.

  • RunAgent (sub-option)
RunAgent cmd [ret_value]

This option is to run resource agent. “cmd” is the parameter of the resource agent, such as “start, status, stop …”. The second parameter is optional. It will compare the actual returned value with the expected value when the script has run recourse agent. If differs, bugs will be found.

It is also possible to execute a suboption on a remote host instead of locally. The protocol used is ssh and the command is run in the background. Just add the @<ipaddr> suffix to the suboption name. For instance:

Bash@ date

would run the date program. Remote commands are run in background.

NB: Not clear how can ssh be automated as we don’t know in advance the environment. Perhaps use “well-known” host names such as “node2”? Also, if the command runs in the background, it’s not clear how is the exit code checked. Finally, does Var@node make sense? Or is the current environment somehow copied over? We probably need an example here.

Need examples in general.

CASE-BLOCK (top level option)
CASE-BLOCK macro_name

The CASE-BLOCK option defines a macro which can be Include+d in any +CASE. All CASE suboptions are valid in CASE-BLOCK.