Product SiteDocumentation Site

Pacemaker 1.1

Configuration Explained

An A-Z guide to Pacemaker's Configuration Options

Edition 1

Andrew Beekhof

Primary author 
Red Hat

Dan Frîncu

Romanian translation 

Philipp Marek

Style and formatting updates. Indexing. 
LINBit

Tanja Roth

Utilization chapter Multi-Site Clusters chapter 
SUSE

Lars Marowsky-Bree

Multi-Site Clusters chapter 
SUSE

Yan Gao

Utilization chapter Multi-Site Clusters chapter 
SUSE

Thomas Schraitle

Utilization chapter Multi-Site Clusters chapter 
SUSE

Legal Notice

Copyright © 2009-2011 Andrew Beekhof.
The text of and illustrations in this document are licensed under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA")[2].
In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
In addition to the requirements of this license, the following activities are looked upon favorably:
  1. If you are distributing Open Publication works on hardcopy or CD-ROM, you provide email notification to the authors of your intent to redistribute at least thirty days before your manuscript or media freeze, to give the authors time to provide updated documents. This notification should describe modifications, if any, made to the document.
  2. All substantive modifications (including deletions) be either clearly marked up in the document or else described in an attachment to the document.
  3. Finally, while it is not mandatory under this license, it is considered good form to offer a free copy of any hardcopy or CD-ROM expression of the author(s) work.
Abstract
The purpose of this document is to definitively explain the concepts used to configure Pacemaker. To achieve this, it will focus exclusively on the XML syntax used to configure the CIB.
For those that are allergic to XML, Pacemaker comes with a cluster shell; a Python based GUI exists, too, however these tools will not be covered at all in this document[1], precisely because they hide the XML.
Additionally, this document is NOT a step-by-step how-to guide for configuring a specific clustering scenario. Although such guides exist, the purpose of this document is to provide an understanding of the building blocks that can be used to construct any type of Pacemaker cluster.


[1] I hope, however, that the concepts explained here make the functionality of these tools more easily understood.

Table of Contents

Preface
1. Document Conventions
1.1. Typographic Conventions
1.2. Pull-quote Conventions
1.3. Notes and Warnings
2. We Need Feedback!
1. Read-Me-First
1.1. The Scope of this Document
1.2. What Is Pacemaker?
1.3. Types of Pacemaker Clusters
1.4. Pacemaker Architecture
1.4.1. Internal Components
2. Configuration Basics
2.1. Configuration Layout
2.2. The Current State of the Cluster
2.3. How Should the Configuration be Updated?
2.4. Quickly Deleting Part of the Configuration
2.5. Updating the Configuration Without Using XML
2.6. Making Configuration Changes in a Sandbox
2.7. Testing Your Configuration Changes
2.8. Interpreting the Graphviz output
2.8.1. Small Cluster Transition
2.8.2. Complex Cluster Transition
2.9. Do I Need to Update the Configuration on all Cluster Nodes?
3. Cluster Options
3.1. Special Options
3.2. Configuration Version
3.3. Other Fields
3.4. Fields Maintained by the Cluster
3.5. Cluster Options
3.6. Available Cluster Options
3.7. Querying and Setting Cluster Options
3.8. When Options are Listed More Than Once
4. Cluster Nodes
4.1. Defining a Cluster Node
4.2. Describing a Cluster Node
4.3. Adding a New Cluster Node
4.3.1. Corosync
4.3.2. Heartbeat
4.4. Removing a Cluster Node
4.4.1. Corosync
4.4.2. Heartbeat
4.5. Replacing a Cluster Node
4.5.1. Corosync
4.5.2. Heartbeat
5. Cluster Resources
5.1. What is a Cluster Resource
5.2. Supported Resource Classes
5.2.1. Open Cluster Framework
5.2.2. Linux Standard Base
5.2.3. Legacy Heartbeat
5.2.4. STONITH
5.3. Properties
5.4. Resource Options
5.5. Setting Global Defaults for Resource Options
5.6. Instance Attributes
5.7. Resource Operations
5.7.1. Monitoring Resources for Failure
5.8. Setting Global Defaults for Operations
5.8.1. When Resources Take a Long Time to Start/Stop
5.8.2. Multiple Monitor Operations
5.8.3. Disabling a Monitor Operation
6. Resource Constraints
6.1. Scores
6.1.1. Infinity Math
6.2. Deciding Which Nodes a Resource Can Run On
6.2.1. Options
6.2.2. Asymmetrical "Opt-In" Clusters
6.2.3. Symmetrical "Opt-Out" Clusters
6.2.4. What if Two Nodes Have the Same Score
6.3. Specifying in which Order Resources Should Start/Stop
6.3.1. Mandatory Ordering
6.3.2. Advisory Ordering
6.4. Placing Resources Relative to other Resources
6.4.1. Options
6.4.2. Mandatory Placement
6.4.3. Advisory Placement
6.5. Ordering Sets of Resources
6.6. Collocating Sets of Resources
7. Receiving Notification for Cluster Events
7.1. Configuring Email Notifications
7.2. Configuring SNMP Notifications
8. Rules
8.1. Node Attribute Expressions
8.2. Time/Date Based Expressions
8.2.1. Date Specifications
8.2.2. Durations
8.3. Using Rules to Determine Resource Location
8.3.1. Using score-attribute Instead of score
8.4. Using Rules to Control Resource Options
8.5. Using Rules to Control Cluster Options
8.6. Ensuring Time Based Rules Take Effect
9. Advanced Configuration
9.1. Connecting from a Remote Machine
9.2. Specifying When Recurring Actions are Performed
9.3. Moving Resources
9.3.1. Manual Intervention
9.3.2. Moving Resources Due to Failure
9.3.3. Moving Resources Due to Connectivity Changes
9.3.4. Resource Migration
9.4. Reusing Rules, Options and Sets of Operations
9.5. Reloading Services After a Definition Change
10. Advanced Resource Types
10.1. Groups - A Syntactic Shortcut
10.1.1. Properties
10.1.2. Options
10.1.3. Instance Attributes
10.1.4. Contents
10.1.5. Constraints
10.1.6. Stickiness
10.2. Clones - Resources That Get Active on Multiple Hosts
10.2.1. Properties
10.2.2. Options
10.2.3. Instance Attributes
10.2.4. Contents
10.2.5. Constraints
10.2.6. Stickiness
10.2.7. Resource Agent Requirements
10.2.8. Notifications
10.2.9. Proper Interpretation of Notification Environment Variables
10.3. Multi-state - Resources That Have Multiple Modes
10.3.1. Properties
10.3.2. Options
10.3.3. Instance Attributes
10.3.4. Contents
10.3.5. Monitoring Multi-State Resources
10.3.6. Constraints
10.3.7. Stickiness
10.3.8. Which Resource Instance is Promoted
10.3.9. Resource Agent Requirements
10.3.10. Notifications
10.3.11. Proper Interpretation of Notification Environment Variables
11. Utilization and Placement Strategy
11.1. Background
11.2. Utilization attributes
11.3. Placement Strategy
11.4. Allocation Details
11.4.1. Which node is preferred to be chosen to get consumed first on allocating resources?
11.4.2. Which resource is preferred to be chosen to get assigned first?
11.5. Limitations
11.6. Strategies for Dealing with the Limitations
12. Protecting Your Data - STONITH
12.1. Why You Need STONITH
12.2. What STONITH Device Should You Use
12.3. Configuring STONITH
12.3.1. Example
13. Status - Here be dragons
13.1. Node Status
13.2. Transient Node Attributes
13.3. Operation History
13.3.1. Simple Example
13.3.2. Complex Resource History Example
14. Multi-Site Clusters and Tickets
14.1. Abstract
14.2. Challenges for Multi-Site Clusters
14.3. Conceptual Overview
14.3.1. Components and Concepts
14.4. Configuring Ticket Dependencies
14.5. Managing Multi-Site Clusters
14.5.1. Managing Tickets Manually
14.5.2. Managing Tickets via a Cluster Ticket Registry
14.6. For more information
A. FAQ
B. More About OCF Resource Agents
B.1. Location of Custom Scripts
B.2. Actions
B.3. How are OCF Return Codes Interpreted?
B.3.1. Exceptions
C. What Changed in 1.0
C.1. New
C.2. Changed
C.3. Removed
D. Installation
D.1. Choosing a Cluster Stack
D.2. Enabling Pacemaker
D.2.1. For Corosync
D.2.2. For Heartbeat
E. Upgrading Cluster Software
E.1. Version Compatibility
E.2. Complete Cluster Shutdown
E.2.1. Procedure
E.3. Rolling (node by node)
E.3.1. Procedure
E.3.2. Version Compatibility
E.3.3. Crossing Compatibility Boundaries
E.4. Disconnect and Reattach
E.4.1. Procedure
E.4.2. Notes
F. Upgrading the Configuration from 0.6
F.1. Preparation
F.2. Perform the upgrade
F.2.1. Upgrade the software
F.2.2. Upgrade the Configuration
F.2.3. Manually Upgrading the Configuration
G. init-Script LSB Compliance
H. Sample Configurations
H.1. An Empty Configuration
H.2. A Simple Configuration
H.3. An Advanced Configuration
I. Further Reading
J. Revision History
Index

List of Figures

1.1. Active/Passive Redundancy
1.2. Shared Failover
1.3. N to N Redundancy
1.4. Conceptual Stack Overview
1.5. The Pacemaker Stack
1.6. Internal Components
4.1. The result of using crm_attribute to specify which kernel pcmk-1 is running
6.1. Ordered Set
6.2. Two Sets of Unordered Resources
6.3. Three Resources Sets
6.4. Another Three Resources Sets
9.1. How the cluster translates the pingd constraint
12.1. Obtaining a list of STONITH Parameters
13.1. A bare-bones status entry for a healthy node called cl-virt-1
13.2. Example set of transient node attributes for node "cl-virt-1"
13.3. A record of the apcstonith resource
13.4. A monitor operation (determines current state of the apcstonith resource)
13.5. Resource history of a pingd clone with multiple jobs

List of Tables

3.1. Configuration Version Properties
3.2. Properties Controlling Validation
3.3. Properties Maintained by the Cluster
3.4. Cluster Options
5.1. Properties of a Primitive Resource
5.2. Options for a Primitive Resource
5.3. Properties of an Operation
6.1. Options for Simple Location Constraints
6.2. Properties of an Ordering Constraint
6.3. Properties of a Collocation Constraint
8.1. Properties of a Rule
8.2. Properties of an Expression
8.3. Properties of a Date Expression
8.4. Properties of a Date Spec
9.1. Environment Variables Used to Connect to Remote Instances of the CIB
9.2. Extra top-level CIB options for remote access
9.3. Common Options for a 'ping' Resource
10.1. Properties of a Group Resource
10.2. Properties of a Clone Resource
10.3. Clone specific configuration options
10.4. Environment variables supplied with Clone notify actions
10.5. Properties of a Multi-State Resource
10.6. Multi-state specific resource configuration options
10.7. Additional constraint options relevant to multi-state resources
10.8. Role implications of OCF return codes
10.9. Environment variables supplied with Master notify actions
13.1. Authoritative Sources for State Information
13.2. Node Status Fields
13.3. Contents of an lrm_rsc_op job.
B.1. Required Actions for OCF Agents
B.2. Optional Actions for OCF Agents
B.3. Types of recovery performed by the cluster
B.4. OCF Return Codes and their Recovery Types
E.1. Summary of Upgrade Methodologies
E.2. Version Compatibility Table

List of Examples

2.1. Sample output from crm_mon
2.2. Sample output from crm_mon -n
3.1. Deleting an option that is listed twice
4.1. Example cluster node entry
5.1. An example LSB resource
5.2. An example OCF resource
5.3. An example Heartbeat resource
5.4. An LSB resource with cluster options
5.5. An example OCF resource with instance attributes
5.6. Displaying the metadata for the Dummy resource agent template
5.7. An OCF resource with a recurring health check
5.8. An OCF resource with custom timeouts for its implicit actions
5.9. An OCF resource with two recurring health checks, performing different levels of checks - specified via OCF_CHECK_LEVEL.
5.10. Example of an OCF resource with a disabled health check
6.1. Example set of opt-in location constraints
6.2. Example set of opt-out location constraints
6.3. Example of two resources that prefer two nodes equally
6.4. Example of an optional and mandatory ordering constraint
6.5. An example colocation constraint
6.6. An example anti-colocation constraint
6.7. An example advisory-only colocation constraint
6.8. A chain of ordered resources
6.9. A chain of ordered resources expressed as a set
6.10. A group resource with the equivalent ordering rules
6.11. Ordered sets of unordered resources
6.12. Advanced use of set ordering - Three ordered sets, two of which are internally unordered
6.13. A chain of collocated resources
6.14. The equivalent colocation chain expressed using resource_sets
6.15. A group resource with the equivalent colocation rules
6.16. Using colocation sets to specify a common peer.
6.17. A colocation chain where the members of the middle set have no inter-dependencies and the last has master status.
8.1. True if now is any time in the year 2005
8.2. Equivalent expression.
8.3. 9am-5pm, Mon-Friday
8.4. 9am-6pm, Mon-Friday, or all day saturday
8.5. 9am-5pm or 9pm-12pm, Mon-Friday
8.6. Mondays in March 2005
8.7. A full moon on Friday the 13th
8.8. Prevent myApacheRsc from running on c001n03
8.9. Prevent myApacheRsc from running on c001n03 - expanded version
8.10. A sample nodes section for use with score-attribute
8.11. Defining different resource options based on the node name
8.12. Change resource-stickiness during working hours
9.1. Specifying a Base for Recurring Action Intervals
9.2. An example ping cluster resource, checks node connectivity once every minute
9.3. Don't run on unconnected nodes
9.4. Run only on nodes connected to three or more ping nodes; this assumes multiplier is set to 1000.
9.5. Prefer the node with the most connected ping nodes
9.6. A more complex example of choosing a location based on connectivity
9.7. Referencing rules from other constraints
9.8. Referencing attributes, options, and operations from other resources
9.9. The DRBD Agent's Control logic for Supporting the reload Operation
9.10. The DRBD Agent Advertising Support for the reload Operation
9.11. Parameter that can be changed using reload
10.1. An example group
10.2. How the cluster sees a group resource
10.3. Example constraints involving groups
10.4. An example clone
10.5. Example constraints involving clones
10.6. Example notification variables
10.7. Monitoring both states of a multi-state resource
10.8. Example constraints involving multi-state resources
10.9. Manually specifying which node should be promoted
12.1. Sample STONITH Resource
D.1. An example Corosync configuration file
D.2. Example options for an IPv6 environment
D.3. Configuration fragment for enabling Pacemaker under Corosync
D.4. Configuration fragment for enabling Pacemaker under Heartbeat
H.1. An empty configuration
H.2. 2 nodes, some cluster options and a resource
H.3. groups and clones with stonith