CoDaXe Blog

After much research into available backup solutions for windows I could not find one that could do what I wanted (essentially sync files and folders across machines and have changes replicated in all directions so if a change occurred on machine 1 and another on machine 2 both changes would be reflected in both machines instead of one change overwriting the other in a traditional one directional backup solutions). A popular service which serves this purpose is Dropbox but it runs on the cloud and that’s enough for you to look the other way. I’m not a fan of the cloud and neither should you, keep your files safe locally and encrypted.

A linux based command line utility called rsync was the first milestone in finding a solution in terms of efficiency as it is extremely fast in replicating updates across however the problem was this was also one directional.
The final solution came when I discovered a tool called unison which is based on rsync but provides this bidirectional functionality. Unison and rsync are both unix based tool so we will use cygwin on windows to use them.

1. Install cygwin on Machine 1 and Machine 2

Download the installer from http://www.cygwin.com. Follow the instructions until you get to the “Select Packages” pages. On this page select specific packages listed below. Simply search for them in the search bar at the top and when they appear on the list click on “Skip” it will change it to “Keep” which will install them.

Step 1-5) Next

Step 6) Select a proxy, any really

Step 7) Wait for download

Step 8 ) Press OK

Step 9) Select the following packages

(required) net > openssh
(required) utils > unison (2.40 or higher)
(recommended) tools > tcp_wrappers
(recommended) editors > vim
(recommended)  net > rsync

Step 10) Next

Step 11) Wait for install

Step 12) Finish

You might get a popup on Windows 7 like below. You may safely ignore it and press “This program installed correctly”. This is due to the nature of the cygwin installer which is in essence just an extractor with optional registry insertion features.

2. Setup SSH Server on Machine 1 (pick machine 1 to be your server, pick the computer that is most reliable or on the most)

Step 1) Right click on the shortcut created by the installer and select “Run as administrator”

Step 2) Type

$ mkdir /var/empty
$ chgrp Administrators /var/{run,log,empty}
$ chown Administrators /var/{run,log,empty}
$ chmod 775 /var/{run,log}
$ chmod 755 /var/empty

Step 3) Type the following commands

$ ssh-host-config
$ Should privilege seperation be used? (yes/no) yes
$ new local account 'sshd'? (yes/no) yes
$ Do you want to install sshd as a service? (yes/no) yes
$ Enter the value of CYGWIN for the daemon: [] ntsec tty binmode server nodosfilewarning
$ Do you want to use a different name? (yes/no) no
$ Create new privileged user account 'cyg_server'? (yes/no)? yes
# type here any password you wish
$ Please enter the password: ______________
$ Reenter: ______________

Step 4) Start the ssh server

$ net start sshd
# note as seen in the screenshot you will see the ssh try to start and automatically stop. This happens because you need to give proper permission to the user you specified during ssh-host-config which in our case was the default 'cyg_server' this is the user which needs to own the appropriate directories so it is fixed with chown cyg_server /var/empty'
$ chown cyg_server /var/empty
$ chmod 755 /var/empty
$ net start sshd
# At this point you should be all set and have a working up and running ssh server on windows.

2. Configure SSH Key Authentification on Machine 2 and Machine 1

Step 1) On Machine 2 generate ssh keys

$ ssh-keygen -t rsa
  Generating public/private rsa key pair.
# type yes to select default location to store the keys
$ Enter file in which to save the key (/home/DX2/.ssh/id_rsa): yes
# when asked for a password press enter do not type in anything it will make your life easier
$ Enter passphrase (empty for no passphrase):
$ Enter same passphrase again:
 Your identification has been saved in yes.
 Your public key has been saved in yes.pub.
 The key fingerprint is:
 5e:cc:85:43:b5:08:6b:b7:5f:04:41:7e:4d:39:cb:93 DX@DX
 The key's randomart image is:
 +--[ RSA 2048]----+
|        . .o=.  o|
|         + + o = |
|        o = + + =|
|       . + + o E |
|        S =   . .|
|       . . . .   |
|        .   .    |
|                 |
|                 |
+-----------------+

Step 2) Organize your ssh keys on Machine 2

$ mkdir -p ~/.ssh/ids/192.168.1.110/DX
# 192.168..1.110 should be the hostname of your server / DX should be the username on the server</pre>
$ touch ~/.ssh/config
$ vi ~/.ssh/config
# now press i (enters editing mode) then type in the following line by line (without the # sign)
# IdentityFile ~/.ssh/ids/%h/%r/id_rsa
# IdentityFile ~/.ssh/ids/%h/id_rsa
# IdentityFile ~/.ssh/id_rsa
# now press [escape] then : then wq then [enter]
# finally move the generated files earlier to this directory
# do this by running
$ mv ~/.ssh/id_* ~/.ssh/ids/192.168.1.110/DX/

Step 3) Add your public key to Machine 1

Now copy the content from Machine 2 of ~/.ssh/ids/192.168.1.110/DX/id_rsa.pub(C:\cygwin\home\DX\.ssh\ids\192.168.1.110\DX\id_rsa.pub) and paste it on Machine 1 inside ~/.ssh/authorized_keys (C:\cygwin\home\DX\.ssh\authorized_keys)

Step 4) Test your ssh connection

# you are now all set with your connection, to test this you can do an
$ ssh -v DX@192.168.1.110
# this should log you in to your remote server, if there is an error the -v option should tell you what's wrong

3. Before your first sync copy your files manually from Machine 2 to Machine 1

Step 1) You first want to copy the files before using the bidirectional unison tool as it will be quite slow otherwise. You can use scp command for that purpose.

# for instance if you want to sync (client) C:\myfolder to (client) C:\myfolder first perform the below command to quickly make an exact duplicate copy
$ scp -r /cygwin/c/myfolder DX@192.168.1.110:/cygwin/c/myfolder

You can otherwise duplicate the directory any other way most convenient for you.

4. Write your first bidirectional unison script on Machine 2

Step 1) Go to your home directory and create a .unison directory if it does not exist

$ mkdir ~/.unison

Step 2) Create a file common.prf in your .unison directory

$ touch ~/.unison/base.prf

Step 3) Edit base.prf either through vi in the terminal or in your favorite text editor and add the following lines

# helps out speed on Windows
fastcheck = true

# backup related
backup         = Name *
backuplocation = central
backupdir      = Backups
maxbackups     = 37

# log file
logfile        = unison.log

# don't synchronize permissions
perms = 0

# place new files at the top of the list
sortnewfirst = true

# turn on ssh compression
rshargs = -C
sshargs = -C

# ignore files from sync
ignore = Name Thumbs.db
ignore = Name *~
ignore = Name *.tmp

Step 4) Create your sync script by creating a mysync.prf

$ touch ~/.unison/mysync.prf

Then add the following lines to it

include base.prf
# the local path to the directory to sync (/cygdrive/ needs to be there, just focus on the path after cygdrive)
root = /cygdrive/c/myfolder/
# the remote user, host and path to directory to do the bidirectional sync
root = ssh://DX@192.168.1.110//cygdrive/c/myfolder/

Step 5) Congratulations! You are now all set to do your bidirectional sync. I would recommend the first time around to do a manual run by ommitting the flag -batch so it will prompt you to answer questions. Once you get comfortable with the process you can add -batch again

# the following command will update modified files in both directions, if you want no prompt add -batch at the end
$ unison mysync
Contacting server...
Connected [//DX//cygdrive/c/myfolder -> //DX2//cygdrive/c/myfolder]
Looking for changes
Waiting for changes from server
No updates to propagate

Upcoming Subjects
- advanced unison configuration, specify merge application for unison to use
- star topology network, sync it on all your machines
- network connection triggered sync
- ssh security