Fault Injection Framework

Fault is defined as a point of interest in the source code with an associated action to be taken when that point is hit during execution. Fault points are defined using SIMPLE_FAULT_INJECTOR() macro or by directly invoking the FaultInjector_TriggerFaultIfSet() function. A fault point is identifed by a name. This module provides an interface to inject a pre-defined fault point into a running PostgreSQL database by associating an action with the fault point. Action can be error, panic, sleep, skip, infinite_loop, etc.

SQL based tests can make use of the "inject_fault()" interface to simulate complex scenarios that are otherwise cumbersome to automate.

For example,

select inject_fault('checkpoint', 'error');

The above command causes the next checkpoint to fail with elog(ERROR). The 'checkpoint' fault is defined in CreateCheckPoint() function in xlog.c. Note that the fault is set to trigger only once by default. Subsequent checkpoints will not be affected by the above fault.

select inject_fault('checkpoint', 'status');

The above command checks the status of the fault. It reports the number of times the fault has been triggered during execution and whether it has completed. Faults that are completed will no longer trigger.

select wait_until_triggered_fault('checkpoint', 1);

The above command blocks until the checkpoint fault is triggered once.

select inject_fault('checkpoint', 'reset');

The above command removes the fault, such that no action will be taken when the fault point is reached during execution. A fault can be set to trigger more than once. For example:

select inject_fault_infinite('checkpoint', 'error');

This command causes checkpoints to fail until the fault is removed.

More detailed interface

A more detailed version of the fault injector interface accepts several more paramters. Let us assume that a fault named "heap_insert" has been defined in function heap_insert() in backend code, like so:

--- a/src/backend/access/heap/heapam.c +++ b/src/backend/access/heap/heapam.c @@ -1875,6 +1875,13 @@ heap_insert(Relation relation, HeapTuple tup, CommandId cid, Buffer vmbuffer = InvalidBuffer; bool all_visible_cleared = false;

+#ifdef FAULT_INJECTOR

• FaultInjector_TriggerFaultIfSet(

•   "heap_insert",
  

•   "" /* database name */,
  

•   RelationGetRelationName(relation));
  

+#endif +

A SQL test may want to inject "heap_insert" fault such that inserts into a table named "my_table" fail for first 10 tuples.

select inject_fault( 'heap_insert', 'error', '' /* database name /, 'my_table' / table name /, 1 / start occurrence /, 10 / end occurrence /, 0 / */);

The above command injects heap_insert fault such that the inserting transaction will abort with elog(ERROR) when the code reaches the fault point, only if the relation being inserted to has the name 'my_table'. Moreover, the fault will stop triggering after 10 tuples have been inserted into the my_table. The 11th transaction to insert into my_table will continue the insert as usual.

Fault actions

Fault action is specified as the type parameter in inject_fault() interface. The following types are supported.

error elog(ERROR)

fatal elog(FATAL)

panic elog(PANIC)

sleep sleep for specified amount of time, use extraArg in seconds

infinite_loop block until the query is canceled or terminated

suspend block until the fault is removed

resume resume backend processes that are blocked due to a suspend fault

skip do nothing (used to implement custom logic that is not supported by predefined actions)

reset remove a previously injected fault

segv crash the backend process due to SIGSEGV

interrupt simulate cancel interrupt arrival, such that the next interrupt processing cycle will cancel the query

finish_pending similar to interrupt, sets the QueryFinishPending global flag

status return a text datum with details of how many times a fault has been triggered, the state it is currently in. Fault states are as follows:

  "set" injected but the fault point has not been reached during
  execution yet.

  "tiggered" the fault point has been reached at least once during
  execution.

  "completed" the action associated with the fault point will no
  longer be taken because the fault point has been reached maximum
  number of times during execution.