15. Status – Here be dragons

Most users never need to understand the contents of the status section and can be happy with the output from crm_mon.

However for those with a curious inclination, this section attempts to provide an overview of its contents.

15.1. Node Status

In addition to the cluster’s configuration, the CIB holds an up-to-date representation of each cluster node in the status section.

A bare-bones status entry for a healthy node cl-virt-1

<node_state id="1" uname="cl-virt-1" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member">
 <transient_attributes id="1"/>
 <lrm id="1"/>
</node_state>

Users are highly recommended not to modify any part of a node’s state directly. The cluster will periodically regenerate the entire section from authoritative sources, so any changes should be done with the tools appropriate to those sources.

Authoritative Sources for State Information
CIB Object Authoritative Source
node_state pacemaker-controld
transient_attributes pacemaker-attrd
lrm pacemaker-execd

The fields used in the node_state objects are named as they are largely for historical reasons, to maintain compatibility with older versions.

Node Status Fields
Field Description
id Unique identifier for the node. Corosync-based clusters use a numeric counter.
uname

The node’s name as known by the cluster

in_ccm

Is the node a member at the cluster communication later? Allowed values: true, false.

crmd

Is the node a member at the pacemaker layer? Allowed values: online, offline.

crm-debug-origin

The name of the source function that made the most recent change (for debugging purposes).

join

Does the node participate in hosting resources? Allowed values: down, pending, member. banned.

expected

Expected value for join.

The cluster uses these fields to determine whether, at the node level, the node is healthy or is in a failed state and needs to be fenced.

15.2. Transient Node Attributes

Like regular Node Attributes, the name/value pairs listed in the transient_attributes section help to describe the node. However they are forgotten by the cluster when the node goes offline. This can be useful, for instance, when you want a node to be in standby mode (not able to run resources) just until the next reboot.

In addition to any values the administrator sets, the cluster will also store information about failed resources here.

A set of transient node attributes for node cl-virt-1

<transient_attributes id="cl-virt-1">
  <instance_attributes id="status-cl-virt-1">
     <nvpair id="status-cl-virt-1-pingd" name="pingd" value="3"/>
     <nvpair id="status-cl-virt-1-probe_complete" name="probe_complete" value="true"/>
     <nvpair id="status-cl-virt-1-fail-count-pingd:0.monitor_30000" name="fail-count-pingd:0#monitor_30000" value="1"/>
     <nvpair id="status-cl-virt-1-last-failure-pingd:0" name="last-failure-pingd:0" value="1239009742"/>
  </instance_attributes>
</transient_attributes>

In the above example, we can see that a monitor on the pingd:0 resource has failed once, at 09:22:22 UTC 6 April 2009. [1].

We also see that the node is connected to three pingd peers and that all known resources have been checked for on this machine (probe_complete).

15.3. Operation History

A node’s resource history is held in the lrm_resources element (a child of the lrm element). The information stored here includes enough information for the cluster to stop the resource safely if it is removed from the configuration section. Specifically, the resource’s id, class, type and provider are stored.

A record of the apcstonith resource

<lrm_resource id="apcstonith" type="fence_apc_snmp" class="stonith"/>

Additionally, we store history entries for certain actions.

Attributes of an lrm_rsc_op element
Field Description
id

Identifier for the history entry constructed from the resource ID, action name, and operation interval.

call-id

A node-specific counter used to determine the order in which actions were executed.

operation

The action name the resource agent was invoked with.

interval

The frequency, in milliseconds, at which the operation will be repeated. One-time execution is indicated by 0.

op-status

The execution status of this action. The meanings of these codes are internal to Pacemaker.

rc-code

The resource agent’s exit status for this action. Refer to the Resource Agents chapter of Pacemaker Administration for how these values are interpreted.

last-rc-change

Machine-local date/time, in seconds since epoch, at which the action first returned the current value of rc-code. For diagnostic purposes.

exec-time

Time, in milliseconds, that the action was running for. For diagnostic purposes.

queue-time

Time, in seconds, that the action was queued for in the local executor. For diagnostic purposes.

crm_feature_set

The Pacemaker feature set used to record this entry.

transition-key

A concatenation of the action’s graph action number, the graph number, the expected result and the UUID of the controller instance that scheduled it. This is used to construct transition-magic (below).

transition-magic

A concatenation of op-status, rc-code and transition-key. Guaranteed to be unique for the life of the cluster (which ensures it is part of CIB update notifications) and contains all the information needed for the controller to correctly analyze and process the completed action. Most importantly, the decomposed elements tell the controller if the history entry was expected and whether it failed.

op-digest

An MD5 sum representing the parameters passed to the action. Used to detect changes to the configuration, to restart resources if necessary.

crm-debug-origin

The origin of the current values. For diagnostic purposes.

15.3.1. Simple Operation History Example

A monitor operation (determines current state of the apcstonith resource)

<lrm_resource id="apcstonith" type="fence_apc_snmp" class="stonith">
  <lrm_rsc_op id="apcstonith_monitor_0" operation="monitor" call-id="2"
    rc-code="7" op-status="0" interval="0"
    crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
    op-digest="2e3da9274d3550dc6526fb24bfcbcba0"
    transition-key="22:2:7:2668bbeb-06d5-40f9-936d-24cb7f87006a"
    transition-magic="0:7;22:2:7:2668bbeb-06d5-40f9-936d-24cb7f87006a"
    last-rc-change="1239008085" exec-time="10" queue-time="0"/>
</lrm_resource>

In the above example, the action is a non-recurring monitor operation often referred to as a “probe” for the apcstonith resource.

The cluster schedules probes for every configured resource on a node when the node first starts, in order to determine the resource’s current state before it takes any further action.

From the transition-key, we can see that this was the 22nd action of the 2nd graph produced by this instance of the controller (2668bbeb-06d5-40f9-936d-24cb7f87006a).

The third field of the transition-key contains a 7, which indicates that the cluster expects to find the resource inactive. By looking at the rc-code property, we see that this was the case.

As that is the only action recorded for this node, we can conclude that the cluster started the resource elsewhere.

15.3.2. Complex Operation History Example

Resource history of a pingd clone with multiple entries

<lrm_resource id="pingd:0" type="pingd" class="ocf" provider="pacemaker">
  <lrm_rsc_op id="pingd:0_monitor_30000" operation="monitor" call-id="34"
    rc-code="0" op-status="0" interval="30000"
    crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
    transition-key="10:11:0:2668bbeb-06d5-40f9-936d-24cb7f87006a"
    last-rc-change="1239009741" exec-time="10" queue-time="0"/>
  <lrm_rsc_op id="pingd:0_stop_0" operation="stop"
    crm-debug-origin="do_update_resource" crm_feature_set="3.0.1" call-id="32"
    rc-code="0" op-status="0" interval="0"
    transition-key="11:11:0:2668bbeb-06d5-40f9-936d-24cb7f87006a"
    last-rc-change="1239009741" exec-time="10" queue-time="0"/>
  <lrm_rsc_op id="pingd:0_start_0" operation="start" call-id="33"
    rc-code="0" op-status="0" interval="0"
    crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
    transition-key="31:11:0:2668bbeb-06d5-40f9-936d-24cb7f87006a"
    last-rc-change="1239009741" exec-time="10" queue-time="0" />
  <lrm_rsc_op id="pingd:0_monitor_0" operation="monitor" call-id="3"
    rc-code="0" op-status="0" interval="0"
    crm-debug-origin="do_update_resource" crm_feature_set="3.0.1"
    transition-key="23:2:7:2668bbeb-06d5-40f9-936d-24cb7f87006a"
    last-rc-change="1239008085" exec-time="20" queue-time="0"/>
  </lrm_resource>

When more than one history entry exists, it is important to first sort them by call-id before interpreting them.

Once sorted, the above example can be summarized as:

  1. A non-recurring monitor operation returning 7 (not running), with a call-id of 3
  2. A stop operation returning 0 (success), with a call-id of 32
  3. A start operation returning 0 (success), with a call-id of 33
  4. A recurring monitor returning 0 (success), with a call-id of 34

The cluster processes each history entry to build up a picture of the resource’s state. After the first and second entries, it is considered stopped, and after the third it considered active.

Based on the last operation, we can tell that the resource is currently active.

Additionally, from the presence of a stop operation with a lower call-id than that of the start operation, we can conclude that the resource has been restarted. Specifically this occurred as part of actions 11 and 31 of transition 11 from the controller instance with the key 2668bbeb.... This information can be helpful for locating the relevant section of the logs when looking for the source of a failure.

[1]You can use the standard date command to print a human-readable version of any seconds-since-epoch value, for example date -d @1239009742.