HowTo: Two way file sync between three or more servers

Posted on June 27, 2014 Updated on July 2, 2014

I was working in a project have multiple server, just assume that more than two, all those webservers are under loadbalancer and which doesn’t having centralize storage. So one thing I can do is sync files between the servers.

The first priority to avoid unwanted process and sync the file only if any updates happen, this action will effect all the servers.

I used the tool called Lsync + Unison

Lsyncd

Source : https://code.google.com/p/lsyncd/

Lsyncd watches a local directory trees event monitor interface (inotify or fsevents). It aggregates and combines events for a few seconds and then spawns one (or more) process(es) to synchronize the changes. By default this is rsync. Lsyncd is thus a light-weight live mirror solution that is comparatively easy to install not requiring new filesystems or blockdevices and does not hamper local filesystem performance.

Unison 

Source : http://olex.openlogic.com/packages/unison

Unison is a file-synchronization tool for Unix and Windows. It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other. License GPLv3

Please follow the implementation steps below for all the servers, because each server should check for updates

Server1 : 192.168.1.51

Server2: 192.168.1.52

Server3: 192.168.1.53

Shared folder is same on all the server /home/syncfuser/fileupload

SSH port : 10022

Web root writable user : syncfuser

Installation

Add additional Centos repository here I used CentOS 6.4 64bit. you may get different version from the give link

http://wiki.centos.org/AdditionalResources/Repositories/RPMForge

[root@srv-51 ~]# rpm -Uvh http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
 Retrieving http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
 warning: /var/tmp/rpm-tmp.iORM9p: Header V3 DSA/SHA1 Signature, key ID 6b8d79e6: NOKEY
 Preparing... ########################################### [100%]
 1:rpmforge-release ########################################### [100%]

[root@srv-51 ~]# rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt

Installing  Lsycd , unison

[root@srv-51 ~]# wget http://lsyncd.googlecode.com/files/lsyncd-2.0.6.tar.gz

[root@srv-51 ~]# tar -zxvf lsyncd-2.0.6.tar.gz

[root@srv-51 ~]# cd lsyncd-2.0.6 ; make ; make install

[root@srv-51 ~]# yum install pkgconfig lua.x86_64 lua-devel.x86_64 lua-static.x86_64 gcc unison -y

[root@srv-51 ~]# touch /var/log/lsyncd/{lsyncd,lsyncd-status}.log

[root@srv-51 ~]# vi /etc/init.d/lsyncd

 #!/bin/bash
 #
 # chkconfig: - 85 15
 # description: Lightweight inotify based sync daemon
 #
 # processname:  lsyncd
 # config:       /etc/lsyncd.conf
 # config:       /etc/sysconfig/lsyncd
 # pidfile:      /var/run/lsyncd.pid
# Source function library
 . /etc/init.d/functions
# Source networking configuration.
 . /etc/sysconfig/network
# Check that networking is up.
 [ "$NETWORKING" = "no" ] && exit 0
LSYNCD_OPTIONS="-pidfile /var/run/lsyncd.pid /etc/lsyncd.conf"
if [ -e /etc/sysconfig/lsyncd ]; then
 . /etc/sysconfig/lsyncd
 fi
RETVAL=0
prog="lsyncd"
 thelock=/var/lock/subsys/lsyncd
start() {
 [ -f /etc/lsyncd.conf ] || exit 6
 echo -n $"Starting $prog: "
 if [ $UID -ne 0 ]; then
 RETVAL=1
 failure
 else
 daemon /usr/local/bin/lsyncd $LSYNCD_OPTIONS
 RETVAL=$?
 [ $RETVAL -eq 0 ] && touch $thelock
 fi;
 echo
 return $RETVAL
 }
stop() {
 echo -n $"Stopping $prog: "
 if [ $UID -ne 0 ]; then
 RETVAL=1
 failure
 else
 killproc lsyncd
 RETVAL=$?
 [ $RETVAL -eq 0 ] && rm -f $thelock
 fi;
 echo
 return $RETVAL
 }
reload(){
 echo -n $"Reloading $prog: "
 killproc lsyncd -HUP
 RETVAL=$?
 echo
 return $RETVAL
 }
restart(){
 stop
 start
 }
condrestart(){
 [ -e $thelock ] && restart
 return 0
 }
case "$1" in
 start)
 start
 ;;
 stop)
 stop
 ;;
 restart)
 restart
 ;;
 reload)
 reload
 ;;
 condrestart)
 condrestart
 ;;
 status)
 status lsyncd
 RETVAL=$?
 ;;
 *)
 echo $"Usage: $0 {start|stop|status|restart|condrestart|reload}"
 RETVAL=1
 esac
exit $RETVAL

chmod +x /etc/init.d/lsyncd

Before start configuring lsync please make sure that password less login is enabled for all the servers.

Create a new conf file /etc/lsyncd.conf

[root@srv-51 ~]# vi /etc/lsyncd.conf

settings = {
 logfile = "/var/log/lsyncd/lsyncd.log",
 statusFile = "/var/log/lsyncd/lsyncd-status.log",
 maxDelays = 3
 }
 runUnison2 = {
 maxProcesses = 1,
 delay = 3,
 onAttrib = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.52:10022//home/syncfuser/fileupload",
 onCreate = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.52:10022//home/syncfuser/fileupload",
 onDelete = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.52:10022//home/syncfuser/fileupload",
 onModify = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.52:10022//home/syncfuser/fileupload",
 onMove = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.52:10022//home/syncfuser/fileupload",
 }
runUnison3 = {
 maxProcesses = 1,
 delay = 3,
 onAttrib = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload",
 onCreate = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload",
 onDelete = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload",
 onModify = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload",
 onMove = "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload",
 }
sync{ runUnison2, source="/home/syncfuser/fileupload" }
sync{ runUnison3, source="/home/syncfuser/fileupload" }

Please use -confirmbigdel=false if you have clear Idea about what you are doing, which will give the power to remove the file forcefully even if the directory is to be empty.

With out -confirmbigdel=false will stop syncing (crash lsync) for the scenario ( directory going to be empty) . It will protect the file from accidentally removal commad rm -rf * , if it is a server you don’t need to worry about it  because the file removal is handled by the application.
Sample logs for file syncs .

[root@srv-51 fileupload]# cat > samplefile-sv1.txt
This is the sample file from server1.
^C

tail -f /var/log/lsyncd/lsyncd.log

Fri Jun 27 18:41:44 2014 Normal: Event Delete spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false  /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload"
Fri Jun 27 18:41:44 2014 Normal: Event Delete spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false  /home/syncfuser/fileupload ssh://syncfuser@192.168.1.51:10022//home/syncfuser/fileupload"
Contacting server...
Contacting server...
Connected [//srv-51//home/syncfuser/fileupload -> //srv-52//home/syncfuser/fileupload]
Looking for changes
  Waiting for changes from server
Reconciling changes
Connected [//srv-52//home/syncfuser/fileupload -> //srv-53//home/syncfuser/fileupload]
Looking for changes
  Waiting for changes from server
Nothing to do: replicas have been changed only in identical ways since last sync.
Fri Jun 27 18:41:44 2014 Normal: Retrying Delete on /home/syncfuser/fileupload//.unison.samplefile-sv1.txt.f629b5a5e1d6f2942bd1ec2ad54122b6.unison.tmp = 0
Fri Jun 27 18:41:44 2014 Normal: Event Create spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false  /home/syncfuser/fileupload ssh://syncfuser@192.168.1.51:10022//home/syncfuser/fileupload"
Reconciling changes
Nothing to do: replicas have been changed only in identical ways since last sync.
Fri Jun 27 18:41:44 2014 Normal: Retrying Delete on /home/syncfuser/fileupload//.unison.samplefile-sv1.txt.f629b5a5e1d6f2942bd1ec2ad54122b6.unison.tmp = 0
Fri Jun 27 18:41:44 2014 Normal: Event Create spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false  /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload"
Contacting server...
Contacting server...
Connected [//srv-51//home/syncfuser/fileupload -> //srv-52//home/syncfuser/fileupload]
Looking for changes
  Waiting for changes from server
Reconciling changes
Nothing to do: replicas have not changed since last sync.
Fri Jun 27 18:41:44 2014 Normal: Retrying Create on /home/syncfuser/fileupload//samplefile-sv1.txt = 0
Connected [//srv-52//home/syncfuser/fileupload -> //srv-53//home/syncfuser/fileupload]
Looking for changes
  Waiting for changes from server
Reconciling changes
Nothing to do: replicas have not changed since last sync.
Fri Jun 27 18:41:44 2014 Normal: Retrying Create on /home/syncfuser/fileupload//samplefile-sv1.txt = 0
Fri Jun 27 18:41:54 2014 Normal: Event Delete spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false  /home/syncfuser/fileupload ssh://syncfuser@192.168.1.51:10022//home/syncfuser/fileupload"
Fri Jun 27 18:41:54 2014 Normal: Event Delete spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false  /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload"
Contacting server...
Contacting server...
Connected [//srv-52//home/syncfuser/fileupload -> //srv-53//home/syncfuser/fileupload]
Connected [//srv-51//home/syncfuser/fileupload -> //srv-52//home/syncfuser/fileupload]
Looking for changes
  Waiting for changes from server
Looking for changes
  Waiting for changes from server
Reconciling changes
Reconciling changes
Nothing to do: replicas have been changed only in identical ways since last sync.
Fri Jun 27 18:41:54 2014 Normal: Retrying Delete on /home/syncfuser/fileupload//.unison.samplefile-sv1.txt.f629b5a5e1d6f2942bd1ec2ad54122b6.unison.tmp = 0
Fri Jun 27 18:41:54 2014 Normal: Event Create spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false  /home/syncfuser/fileupload ssh://syncfuser@192.168.1.51:10022//home/syncfuser/fileupload"
Nothing to do: replicas have been changed only in identical ways since last sync.
Fri Jun 27 18:41:54 2014 Normal: Retrying Delete on /home/syncfuser/fileupload//.unison.samplefile-sv1.txt.f629b5a5e1d6f2942bd1ec2ad54122b6.unison.tmp = 0
Fri Jun 27 18:41:54 2014 Normal: Event Create spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false  /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload"
Contacting server...
Contacting server...
Connected [//srv-51//home/syncfuser/fileupload -> //srv-52//home/syncfuser/fileupload]
Connected [//srv-52//home/syncfuser/fileupload -> //srv-53//home/syncfuser/fileupload]
Looking for changes
  Waiting for changes from server
Looking for changes
Reconciling changes
Nothing to do: replicas have not changed since last sync.
Fri Jun 27 18:41:54 2014 Normal: Retrying Create on /home/syncfuser/fileupload//samplefile-sv1.txt = 0
  Waiting for changes from server
Reconciling changes
Nothing to do: replicas have not changed since last sync.
Fri Jun 27 18:41:54 2014 Normal: Retrying Create on /home/syncfuser/fileupload//samplefile-sv1.txt = 0

[root@srv-52 fileupload]# cat >> samplefile-sv1.txt

File edited from server2.
^C
Log:

Fri Jun 27 18:43:54 2014 Normal: Event Modify spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.53:10022//home/syncfuser/fileupload"
Fri Jun 27 18:43:54 2014 Normal: Event Modify spawns shell "export HOME=/root ; /usr/bin/unison -batch -confirmbigdel=false /home/syncfuser/fileupload ssh://syncfuser@192.168.1.51:10022//home/syncfuser/fileupload"
Contacting server...
Contacting server...
Connected [//srv-51//home/syncfuser/fileupload -> //srv-52//home/syncfuser/fileupload]
Connected [//srv-52//home/syncfuser/fileupload -> //srv-53//home/syncfuser/fileupload]
Looking for changes
Looking for changes
 Waiting for changes from server
 Waiting for changes from server
Reconciling changes
changed ----> samplefile-sv1.txt 
local : changed file modified on 2014-06-27 at 18:43:50 size 65 rw-r--r--
srv-53 : unchanged file modified on 2014-06-27 at 18:41:51 size 38 rw-r--r--
Propagating updates
UNISON 2.40.63 started propagating changes at 18:43:54.25 on 27 Jun 2014
[BGN] Updating file samplefile-sv1.txt from /home/syncfuser/fileupload to //srv-53//home/syncfuser/fileupload
Reconciling changes
changed ----> samplefile-sv1.txt 
local : changed file modified on 2014-06-27 at 18:43:50 size 65 rw-r--r--
srv-51 : unchanged file modified on 2014-06-27 at 18:41:51 size 38 rw-r--r--
Propagating updates
UNISON 2.40.63 started propagating changes at 18:43:54.25 on 27 Jun 2014
[BGN] Updating file samplefile-sv1.txt from /home/syncfuser/fileupload to //srv-51//home/syncfuser/fileupload
[END] Updating file samplefile-sv1.txt
UNISON 2.40.63 finished propagating changes at 18:43:54.25 on 27 Jun 2014
Saving synchronizer state
[END] Updating file samplefile-sv1.txt
UNISON 2.40.63 finished propagating changes at 18:43:54.25 on 27 Jun 2014
Saving synchronizer state
Synchronization complete at 18:43:54 (1 item transferred, 0 skipped, 0 failed)
Fri Jun 27 18:43:54 2014 Normal: Retrying Modify on /home/syncfuser/fileupload//samplefile-sv1.txt = 0
Synchronization complete at 18:43:54 (1 item transferred, 0 skipped, 0 failed)
Fri Jun 27 18:43:54 2014 Normal: Retrying Modify on /home/syncfuser/fileupload//samplefile-sv1.txt = 0

[root@srv-53 fileupload]# rm -rf samplefile-sv1.txt
[root@srv-53 fileupload]# ll
total 0

Log :

Connected [//srv-51//home/syncfuser/fileupload -> //srv-53//home/syncfuser/fileupload]
Connected [//srv-52//home/syncfuser/fileupload -> //srv-53//home/syncfuser/fileupload]
Looking for changes
 Waiting for changes from server
Looking for changes
Reconciling changes
deleted ----> samplefile-sv1.txt 
local : deleted
srv-51 : unchanged file modified on 2014-06-27 at 18:53:14 size 0 rw-r--r--
Propagating updates
UNISON 2.40.63 started propagating changes at 18:53:31.85 on 27 Jun 2014
[BGN] Deleting samplefile-sv1.txt from //srv-51//home/syncfuser/fileupload
 Waiting for changes from server
[END] Deleting samplefile-sv1.txt
UNISON 2.40.63 finished propagating changes at 18:53:31.85 on 27 Jun 2014
Saving synchronizer state
Reconciling changes
deleted ----> samplefile-sv1.txt 
local : deleted
srv-52 : unchanged file modified on 2014-06-27 at 18:53:14 size 0 rw-r--r--
Propagating updates
UNISON 2.40.63 started propagating changes at 18:53:31.85 on 27 Jun 2014
[BGN] Deleting samplefile-sv1.txt from //srv-52//home/syncfuser/fileupload
[END] Deleting samplefile-sv1.txt
UNISON 2.40.63 finished propagating changes at 18:53:31.85 on 27 Jun 2014
Saving synchronizer state
Synchronization complete at 18:53:31 (1 item transferred, 0 skipped, 0 failed)
Fri Jun 27 18:53:31 2014 Normal: Retrying Delete on /home/syncfuser/fileupload//samplefile-sv1.txt = 0
Synchronization complete at 18:53:31 (1 item transferred, 0 skipped, 0 failed)
Fri Jun 27 18:53:31 2014 Normal: Retrying Delete on /home/syncfuser/fileupload//samplefile-sv1.txt = 0

Now You have unique file in all the servers.