Setting Up DRBD on CentOS/Rhel 7.3

DRBD

DRBD stand for “Distributed Replicated Block Device”. It is a distributed storage system (or mirrors block devices) over multiple different hosts. In DRBD the data is replicated below the file-system at the block layer over TCP/IP. We can mirrored the block device like hard disks, partitions, RAID devices, logical volumes, etc.

My Setup:

  • We are using two disks(size 10GB) on both nodes.
  • Set the selinux on permissive mode.
  • Allow the port no. 7788 on firewalld.
  • We have two nodes that are reachable by below Networks:
    10.1.1.0/24  : Cluster heartbeat vlan.
    172.16.1.0/24 : LAN with access to the Internal LAN network and Internet.

Configure the Basic DRBD Setup:

1) Install the drbd module Packages.

First, Import the ELRepo package with signing key on both nodes & enable the repository.

[master ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[client ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

Update (or install) the ELRepo Package on both nodes.

[master ~]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
Retrieving http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:elrepo-release-7.0-2.el7.elrepo  ################################# [100%]
[master ~]#

Node 2 Sample Output:

[client ~]# rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
Retrieving http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:elrepo-release-7.0-2.el7.elrepo  ################################# [100%]
[client ~]#

Then install the DRBD kernel module with dependencies. Install below Packages on the both nodes.

[master ~]# yum install  kmod-drbd84 drbd84-utils
Loaded plugins: fastestmirror
base                                                                                         | 3.6 kB  00:00:00     
elrepo                                                                                       | 2.9 kB  00:00:00     
extras                                                                                       | 3.4 kB  00:00:00     
updates                                                                                      | 3.4 kB  00:00:00     
(1/3): extras/7/x86_64/primary_db                                                            | 168 kB  00:00:03     
(2/3): elrepo/primary_db                                                                     | 415 kB  00:00:03     
(3/3): updates/7/x86_64/primary_db                                                           | 5.7 MB  00:00:18     
Loading mirror speeds from cached hostfile
 * base: mirrors.viethosting.com
 * elrepo: mirrors.netix.net
 * extras: mirrors.viethosting.com
 * updates: mirrors.aluhost.com
Resolving Dependencies
--> Running transaction check
---> Package drbd84-utils.x86_64 0:8.9.8-1.el7.elrepo will be installed
---> Package kmod-drbd84.x86_64 0:8.4.9-1.el7.elrepo will be installed

. . . . . . .

 kmod-drbd84            x86_64               8.4.9-1.el7.elrepo                  elrepo                     209 k

Transaction Summary
====================================================================
Install  2 Packages

Total download size: 615 k
Installed size: 1.8 M
Is this ok [y/d/N]: y
Downloading packages:

To Avoid the issues of SELinux we set the selinux on  “Permissive” mode and we are going to remit the DRBD processes from SELinux control.

[master ~]# getenforce 
Permissive
[master ~]# semanage permissive -a drbd_t
[master ~]#

Perform same steps on node 2.

[client ~]# getenforce 
Permissive
[client ~]# semanage permissive -a drbd_t
[client ~]#

2) Configure the Partition for DRBD on both node.

Let’s make a Partition, Right now look out the partition of the both servers. We have 10 GB volume group (i.e drbd_vg) on both the node.

[master ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  cl        1   2   0 wz--n-  7.00g     0 
  drbd_vg   1   0   0 wz--n- 10.00g 10.00g
[master ~]#

On node 2:

[client ~]# vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  cl        1   2   0 wz--n-  7.00g     0 
  drbd_vg   1   0   0 wz--n- 10.00g 10.00g
[client ~]#

Create a new 10GB logical volume for DRBD on node 1:

[master ~]# lvcreate -L 10000M -n drbd_lv drbd_vg
  Logical volume "drbd_lv" created.
[master ~]# lvdisplay drbd_vg
  --- Logical volume ---
  LV Path                /dev/drbd_vg/drbd_lv
  LV Name                drbd_lv
  VG Name                drbd_vg
  LV UUID                zAL3Y9-a8Xs-akHz-LujS-zQyP-C0XC-WnWMys
  LV Write Access        read/write
  LV Creation host, time master.itcare.com, 2017-06-19 01:25:13 -0400
  LV Status              available
  # open                 0
  LV Size                9.77 GiB
  Current LE             2500
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2
   
[master ~]#

Similarly We create the 10GB logical volume for DRBD on node 2:

[client ~]# lvcreate -L 10000M -n drbd_lv drbd_vg
  Logical volume "drbd_lv" created.
client ~]# lvdisplay drbd_vg
  --- Logical volume ---
  LV Path                /dev/drbd_vg/drbd_lv
  LV Name                drbd_lv
  VG Name                drbd_vg
  LV UUID                YOxoTO-yW3C-Po0I-1DJo-QVem-6riu-tFQVDU
  LV Write Access        read/write
  LV Creation host, time client.itcare.com, 2017-06-19 01:29:49 -0400
  LV Status              available
  # open                 0
  LV Size                9.77 GiB
  Current LE             2500
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2
   
[client ~]#

3) Configuration of DRBD.

Before configuring the drbd First take the backup of  “global_common.conf” file.

[master ~]# cp -v /etc/drbd.d/global_common.conf  /etc/drbd.d/global_common.conf.bak
‘/etc/drbd.d/global_common.conf’ -> ‘/etc/drbd.d/global_common.conf.bak’
[master ~]#

Node 2:

[client ~]# cp -v /etc/drbd.d/global_common.conf  /etc/drbd.d/global_common.conf.bak
‘/etc/drbd.d/global_common.conf’ -> ‘/etc/drbd.d/global_common.conf.bak’
[client ~]#

Now, We have a resource named “shared_drbd” which uses /dev/drbd_vg/drbd_lv as the lower-level device, and is configured with internal meta data.

#vim /etc/drbd.d/global_common.conf

# DRBD is the result of over a decade of development by LINBIT.
# In case you need professional services for DRBD or have
# feature requests visit http://www.linbit.com

global {
    usage-count yes;
    # minor-count dialog-refresh disable-ip-verification
    # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600;
}
resource shared_drbd {
protocol C;
handlers {
        # These are EXAMPLE handlers only.
        # They may have severe implications,
        # like hard resetting the node under certain circumstances.
        # Be careful when chosing your poison.
         pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
         pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
         local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
        # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
         split-brain "/usr/lib/drbd/notify-split-brain.sh root";
        # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
        # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
        # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;   
}
    startup {
        # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
wfc-timeout  60;
degr-wfc-timeout 120;     
}
    options {
        # cpu-mask on-no-data-accessible
    }
    disk {
        # size on-io-error fencing disk-barrier disk-flushes
        # disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
on-io-error   detach;
#fencing resource-and-stonith;    
}
    net {
        # protocol timeout max-epoch-size max-buffers unplug-watermark
        # connect-int ping-int sndbuf-size rcvbuf-size ko-count
        # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
        # after-sb-1pri after-sb-2pri always-asbp rr-conflict
        # ping-timeout data-integrity-alg tcp-cork on-congestion
        # congestion-fill congestion-extents csums-alg verify-alg
        # use-rle
allow-two-primaries no;
  after-sb-0pri discard-zero-changes;
  after-sb-1pri discard-secondary;
  after-sb-2pri disconnect;
  rr-conflict disconnect;
}
  on master.itcare.com {
    device    /dev/drbd1;
    disk      /dev/drbd_vg/drbd_lv;
    address   172.16.1.1:7789;
  meta-disk internal;
  }
  on client.itcare.com {
    device    /dev/drbd1;
    disk      /dev/drbd_vg/drbd_lv;
    address   172.16.1.2:7789;
    meta-disk internal;
  }
}

Note: The hostname needed identical to the node name so that make sure your DNS Resolution working fine. We defined the Hostname and IP Address in “/etc/hosts” file(On both the nodes), as shown below;

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.16.1.1        master.itcare.com   #Physical IP Address of NODE1
172.16.1.2        client.itcare.com   #Physical IP Address of NODE2
10.1.1.1          NODE1               #HeartBeat IP Addr of NODE1
10.1.1.2          NODE2               #HeartBeat IP Addr of NODE2

The resource uses “7789/tcp” port for its network connections,So we enable the ports on firewall on both nodes.

[master ~]# firewall-cmd --permanent --add-port=7789/tcp
success
[master ~]# firewall-cmd --reload
success
[master ~]#

Node2:

[client ~]# firewall-cmd --permanent --add-port=7789/tcp
success
[client ~]# firewall-cmd --reload
success
[client ~]#

Now, Copy the DRBD configured file (i.e global_common.conf) to node 2.

[master ~]# scp -r /etc/drbd.d/global_common.conf root@172.16.1.2:/etc/drbd.d/
The authenticity of host '172.16.1.2 (172.16.1.2)' can't be established.
ECDSA key fingerprint is c4:b2:7a:a4:c4:6c:cf:5f:8f:c2:8b:89:b3:3d:63:6d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.1.2' (ECDSA) to the list of known hosts.
root@172.16.1.2's password: 
global_common.conf                                                                                             100% 2592     2.5KB/s   00:00    
[master ~]#

Now, we need to initialize the DRBD meta data storage on both servers.

[master ~]# drbdadm create-md shared_drbd
initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
success
[master ~]#

Similarly command execute on second node.

[client ~]# drbdadm create-md shared_drbd
initializing activity log
NOT initializing bitmap
Writing meta data...
New drbd meta data block successfully created.
[client ~]#

Bring up the DRBD resource;

[master ~]# drbdadm up shared_drbd

Node2:

[client ~]# drbdadm up shared_drbd

For data consistency, We need to execute the “drbdadm” command on the PRIMARY node.

[master ~]# drbdadm primary --force shared_drbd

Check if Device disk initial synchronization to complete (100%) and check to confirm you are on primary node.

[master ~]# drbd-overview 
 1:shared_drbd/0  SyncSource Primary/Secondary UpToDate/Inconsistent 
    [======>.............] sync'ed: 37.8% (6228/9996)M                 
[master ~]#

Note: By using the both commands(i.e drbd-overview, cat /proc/drbd) we can check the status of synchronization of block device between multiple nodes.

[master ~]# cat /proc/drbd 
version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by akemi@Build64R7, 2016-12-04 01:08:48

 1: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
    ns:3980964 nr:0 dw:0 dr:3982004 al:0 bm:0 lo:0 pe:2 ua:0 ap:0 ep:1 wo:f oos:6260384
    [======>.............] sync'ed: 38.9% (6112/9996)M
    finish: 0:02:31 speed: 41,336 (31,332) K/sec
[master ~]#

After completing 100% shows the below output.

[master ~]# cat /proc/drbd 
version: 8.4.9-1 (api:1/proto:86-101)
GIT-hash: 9976da086367a2476503ef7f6b13d4567327a280 build by akemi@Build64R7, 2016-12-04 01:08:48

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:10239648 nr:0 dw:0 dr:10240704 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
[master ~]#

Now Create filesystem on DRBD device, this command will perform on primary node.

[master ~]# mkfs.xfs /dev/drbd1
meta-data=/dev/drbd1             isize=512    agcount=4, agsize=639978 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=2559912, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[master ~]#

We are mounting the DRBD device on primary node and create some files on the filesystem.

[master ~]# mkdir /Drbdshared
[client ~]# mkdir /Drbdshared
[master ~]# mount /dev/drbd1 /Drbdshared/
[master ~]# df -hT /Drbdshared
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/drbd1     xfs   9.8G   33M  9.8G   1% /Drbdshared
[master ~]# cd /Drbdshared
[master Drbdshared]# ls
[master Drbdshared]# touch file{1..20}
[master Drbdshared]# ls
file1   file11  file13  file15  file17  file19  file20  file4  file6  file8
file10  file12  file14  file16  file18  file2   file3   file5  file7  file9
[master Drbdshared]#

Warning: Note that All data you write on “/Drbdshared” directory will be synced to node2. Remember that you don’t need to mount the DRBD Disk on other node.

4) Test the data actually syncing between the nodes.

To test the data syncing between the nodes we need to perform below steps. First we need to umount the DRBD (/Drbdshared) directory and Demote the primary server to the secondary role.

[master ~]# umount /Drbdshared
[master ~]# drbdadm secondary shared_drbd

Now on the secondary machine (i.e Node2) promote it to the primary role and mount the DRBD filesystem on /Drbdshared directory and using “ls” command check the data.

[client ~]# drbdadm primary --force shared_drbd 
[client ~]# mount /dev/drbd1 /Drbdshared/
[client ~]# df -h /Drbdshared
Filesystem           Size  Used Avail Use% Mounted on
/dev/drbd1           9.8G   33M  9.8G   1% /Drbdshared
[client ~]#cd /Drbdshared/
[root@client Drbdshared]# ls
file1   file11  file13  file15  file17  file19  file20  file4  file6  file8
file10  file12  file14  file16  file18  file2   file3   file5  file7  file9
[client Drbdshared]#

Some Tips:

When you occurred below error, perform below steps on primary node and note that don’t need to mount the DRBD Disk on other node (secondary).

Error:

1: Failure: (127) Device minor not allocated
additional info from kernel:
unknown minor
Command 'drbdsetup-84 primary 1 --force' terminated with exit code 10

Resolution Steps:

[master ~]# drbdadm up shared_drbd
Marked additional 12 MB as out-of-sync based on AL.
[master ~]# drbdadm primary --force shared_drbd
[master ~]# mount /dev/drbd1 /Drbdshared/
[master ~]# df -h /Drbdshared/
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      9.8G   33M  9.8G   1% /Drbdshared

Hope this post will help Linux/Unix beginners. Please share you feedback and Comments. Stay tune for more updates with ittroubleshooter.in …!!!

Leave a Reply

Be the First to Comment!

Notify of
avatar
wpDiscuz