Configure MariaDB HA Pacemaker Cluster with DRBD on RHEL/Centos7

MariaDB HA Cluster Configuration

Today In this article We are going to explain two node Active/Passive MariaDB HA Cluster using Pacemaker. Previously We have covered the installation and configuration of DRBD and basic HA Cluster. Refer below links for more information;

Setting up the Basic High-Availability Cluster on Centos/RHEL 7.3

Setting Up DRBD on CentOS/RHEL 7.3

My Setup:

Configure the MariaDB HA Setup:

1) Installation and configuration of basic HA Cluster.

Previously We have already covered the Installation and configuration of basic HA Cluster. refer below link;

Configure the Basic HA Cluster Using pcs.

2) Configure the DRBD Setup.

We have covered the configuration of DRBD setup, refer the below link.

Setting up the DRBD setup.

In Previously article, We have mounted the DRBD device on primary node (/Drbdshared) and create some files on the file-system.

[master ~]# cd /Drbdshared/
[master Drbdshared]# ls
file1   file11  file13  file15  file17  file19  file20  file4  file6  file8
file10  file12  file14  file16  file18  file2   file3   file5  file7  file9
[master Drbdshared]#

3) Installation of Mariadb Packages.

We have to install the MariaDB Packages on the both nodes.

[master ~]# yum install  mariadb-server mariadb -y

On Node 2 ;

[client ~]# yum install  mariadb-server mariadb -y

Ensure that MariaDB service is disabled, as it will managed by pacemaker;

[master ~]# systemctl is-enabled mariadb
disabled
[master ~]#

On Node 2;

[client ~]# systemctl is-enabled mariadb
disabled
[client ~]#

Now manually start the MariaDB service on one of the cluster nodes.

[master ~]# systemctl start mariadb

We can install a fresh MariaDB database with the “mysql_install_db” command on /Drbdshared directory using mysql user.

Sample Output:

[master ~]# mysql_install_db --datadir=/Drbdshared/ --user=mysql
Installing MariaDB/MySQL system tables in '/Drbdshared/' ...
170630  6:09:55 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 26857 ...
OK
Filling help tables...
170630  6:09:55 [Note] /usr/libexec/mysqld (mysqld 5.5.52-MariaDB) starting as process 26866 ...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system




. . . . . .

Corporation Ab. You can contact us about this at sales@mariadb.com.
Alternatively consider joining our community based development effort:
http://mariadb.com/kb/en/contributing-to-the-mariadb-project/

[master ~]#

Now, If you want to Set a password for the mariaDB root user for secure installation, refer below command; In our case we are not set the password for root user.

[master ~]# mysql_secure_installation

Note: If your’s SElinux enable, we need to give the same SELinux policy as the MariaDB datadir, use the “semange” command to add file context.

[master ~]# semanage fcontext -a -t mysqld_db_t "/Drbdshared/(/.*)?"
[master ~]# restorecon -Rv /Drbdshared/

Stop the database service.

[master ~]# systemctl stop mariadb
[master ~]# systemctl status mariadb
● mariadb.service - MariaDB database server
   Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Jun 30 05:40:42 master.itcare.com mariadb-prepare-db-dir[25824]: http://dev.mysql.com
Jun 30 05:40:42 master.itcare.com mariadb-prepare-db-dir[25824]: Support MariaDB development by buying support/new features from MariaDB
Jun 30 05:40:42 master.itcare.com mariadb-prepare-db-dir[25824]: Corporation Ab. You can contact us about this at sales@mariadb.com.
Jun 30 05:40:42 master.itcare.com mariadb-prepare-db-dir[25824]: Alternatively consider joining our community based development effort:
Jun 30 05:40:42 master.itcare.com mariadb-prepare-db-dir[25824]: http://mariadb.com/kb/en/contributing-to-the-mariadb-project/
Jun 30 05:40:42 master.itcare.com mysqld_safe[25903]: 170630 05:40:42 mysqld_safe Logging to '/var/log/mariadb/mariadb.log'.
Jun 30 05:40:42 master.itcare.com mysqld_safe[25903]: 170630 05:40:42 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
Jun 30 05:40:44 master.itcare.com systemd[1]: Started MariaDB database server.
Jun 30 06:16:03 master.itcare.com systemd[1]: Stopping MariaDB database server...
Jun 30 06:16:04 master.itcare.com systemd[1]: Stopped MariaDB database server.
[master ~]#

And now unmounted the DRBD file-system.

[master ~]# df -h /Drbdshared
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      9.8G   34M  9.8G   1% /Drbdshared
[root@master ~]# umount /Drbdshared

4) Configure the cluster resource for DRBD.

When you are using the pcs command, you can use the “-f” option to save a configuration change to a file without affecting the active CIB. If you have previously configured a cluster and there is already an active CIB, you use the following command to save the raw xml a file.

[master ~]# pcs cluster cib clustdatabase_cfg

Now We have to create a cluster resource for the DRBD Device named as “Database_mysql”.

[master ~]# pcs -f clustdatabase_cfg resource create Database_mysql ocf:linbit:drbd \
  drbd_resource=shared_drbd \
  op monitor interval=30s

We have create the clone resource named as “databasebothnode” which allow the resource to run on both cluster nodes at the same time.

[master ~]# pcs -f clustdatabase_cfg resource master databasebothnode Database_mysql \
 master-max=1 master-node-max=1 \
 clone-max=2 clone-node-max=1 \
 notify=true ordered=false

Where,

master-max: How many copies of the resource can be promoted to master status and the default value is 1.
master-node-max: How many copies of the resource can be promoted to master status on a single node and the default value is 1.
clone-max: How many copies of the resource to start. Defaults to the number of nodes in the cluster.
clone-node-max: How many copies of the resource can be started on a single node.The default value is 1.
notify: When stopping or starting a copy of the clone, tell all the other copies before hand and when the action was successful. Allowed values:  false,true. The default value is false.

The ordering and the configuration logic for Cluster resources as show below;

Pacemaker Cluster resources  Start sequence: fsDRBD_mysql –> MariaDB_New –> MariaDB_VIP,
Pacemaker Cluster resources  Stop sequence: MariaDB_VIP –> MariaDB_New –> fsDRBD_mysql.

Now create the cluster resource for the file-systems named as fsDRBD_mysql.

[master ~]# pcs -f clustdatabase_cfg resource create fsDRBD_mysql Filesystem \
 device="/dev/drbd1" \
 directory="/Drbdshared" \
 fstype="xfs"

Then tell the cluster using “constraint colocation” option the clone resource i.e  “databasebothnode” must be run on the same node as the file-system resource.

[master ~]# pcs -f clustdatabase_cfg constraint colocation add fsDRBD_mysql with databasebothnode \
  INFINITY with-rsc-role=Master

Now the clone resource must be started before the file-system resource.

[master ~]#  pcs -f clustdatabase_cfg constraint order promote databasebothnode then start fsDRBD_mysql

Create the cluster resource named as “MariaDB_New” for the MariaDB service.

[master ~]# pcs -f clustdatabase_cfg resource create MariaDB_New systemd:mariadb \
 op start timeout=60s \
 op stop timeout=60s \
 op monitor interval=20s timeout=30s

Where,

timout: How long to wait before declaring the action has failed.
interval: How frequently (in seconds) to perform the operation. Default value is 0, meaning never.

Then Tell the cluster that the MariaDB service must be run on the same node as the “fsDRBD_mysql” file-system resource.

[master ~]# pcs -f clustdatabase_cfg constraint colocation add MariaDB_New with fsDRBD_mysql INFINITY

Now the “fsDRBD_mysql” resource must be started before the MariaDB resource.

[master ~]# pcs -f clustdatabase_cfg constraint order  fsDRBD_mysql then MariaDB_New

Note: We have to add one parameter on “my.cnf” configuration on both nodes.

#vim /etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
log-error=/var/log/mariadb/mariadb.log
pid-file=/var/run/mariadb/mariadb.pid
bind_address=172.16.1.20
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

~
~
~
:wq

Important Commands:

You can Check the configuration which are saved on “clustdatabase_cfg” cib file.

[master ~]# pcs -f clustdatabase_cfg  resource show
 Asterick_VIP01    (ocf::heartbeat:IPaddr2):    Started NODE1
 Master/Slave Set: databasebothnode [Database_mysql]
     Stopped: [ NODE1 NODE2 ]
 fsDRBD_mysql    (ocf::heartbeat:Filesystem):    Stopped
 MariaDB    (systemd:mariadb):    Stopped
 MariaDB_New    (systemd:mariadb):    Stopped

Suppose if you want to delete one of resource which are added by mistakenly, you can use below command;

[master ~]# pcs -f clustdatabase_cfg  resource delete MariaDB
[master ~]# pcs -f clustdatabase_cfg  resource show
 Asterick_VIP01    (ocf::heartbeat:IPaddr2):    Started NODE1
 Master/Slave Set: databasebothnode [Database_mysql]
     Stopped: [ NODE1 NODE2 ]
 fsDRBD_mysql    (ocf::heartbeat:Filesystem):    Stopped
 MariaDB_New    (systemd:mariadb):    Stopped

We need to add ip address on which mariaDB service will run. We create a resource named as “MariaDB_VIP” for IP Address.

pcs -f clustdatabase_cfg resource create MariaDB_VIP ocf:heartbeat:IPaddr2 \
 ip=172.16.1.20 cidr_netmask=24 \
 op monitor interval=30s

The virtual IP “MariaDB_VIP” resource must be run on the same node as the MariaDB resource and ensure that all other resources are already started before we can connect to the virtual IP.

[master ~]# pcs -f clustdatabase_cfg constraint colocation add MariaDB_VIP with MariaDB_New INFINITY
[master ~]# pcs -f clustdatabase_cfg constraint order MariaDB_New then MariaDB_VIP

Finally We can commit changes now and check cluster status;

Few Steps are perform when we committed the configuration.

1) Pacemaker will start DRBD on the both cluster nodes.
2) Then Pacemaker will select one node for promotion to the DRBD Primary role.
3) Then it will mount the DRBD filesystem and start the MariaDB service on the primary node.

[master ~]# pcs cluster cib-push clustdatabase_cfg
[master ~]# pcs status
Cluster name: AsteriskCluster
Stack: corosync
Current DC: NODE1 (version 1.1.15-11.el7_3.4-e174ec8) - partition with quorum
Last updated: Mon Jul  3 09:17:47 2017        Last change: Mon Jul  3 07:58:00 2017 by root via cibadmin on NODE1

2 nodes and 5 resources configured

Online: [ NODE1 NODE2 ]

Full list of resources:

 Asterick_VIP01    (ocf::heartbeat:IPaddr2):    Started NODE1
 Master/Slave Set: databasebothnode [Database_mysql]
     Masters: [ NODE1 ]
     Slaves: [ NODE2 ]
 fsDRBD_mysql    (ocf::heartbeat:Filesystem):    Started NODE1
 MariaDB_New    (systemd:mariadb):    Started NODE1
MariaDB_VIP    (ocf::heartbeat:IPaddr2):    Started NODE1
Daemon Status:
  corosync: active/disabled
  pacemaker: active/disabled
  pcsd: active/enabled
[master ~]#

You can login on Database, using below command.

[master ~]# mysql -h 172.16.1.20 -u root 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 5.5.52-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]>

We successfully login on MariaDB Server.
Note: If you set the password for Database then use “-p” option for login.

Hope this post will help Linux/Unix beginners. Please share you feedback and Comments. Stay tune for more updates with ittroubleshooter.in …!!!

Leave a Reply

Be the First to Comment!

Notify of
avatar
wpDiscuz