[problem]
You have an admin host (ADMIN), which can see two servers (hostA and hostB) behind firewalls.
You are not allowed to scp/rdist, etc from hostA or B to ADMIN. Nor are you allowed to scp/rdist, etc between hostA and B.
ADMIN can scp, etc though to both hosts. Now in this real life example, updates are made directly to hostA – but not hostB. So they need to be synchronized somehow, looking at the real files and performing a pull to ADMIN from hostA and then a push to hostB.
But then we only want new files …
[/problem]
[solution]
In this script at the example, I pull from hostA and push to hostB. But only “look” for files that have updated, since last time I looked. 🙂
[/solution]
[example]
It starts off just setting variables, creating temporary directories on ADMIN host.
Then at line 26 I obtain a list of files from hostA, that have been updated since last time I ran this script.
At line 29 I spin around this list and copy the given file to the temp directory (line 34). Notice I use ${file##*/} this strips off the path – so I just get the file (i.e. bob.gif not /dirABCD/…./bob.gif).
I then copy this file (line 36) back out to the full path on hostB. Build a test and check the file exists on hostB – lines 38 – 42. Then remove the temporary copy of the file (line 44).
Spin back round for the next file at line 45. Lastly cleaning up.
Whilst this is not as efficient as rdist (opening one ssh connection), it will only copy updated files. So not a tar of the whole directory, etc. Plus by pepper potting between hostA and hostB – there is a few seconds delay between each file update – so should n’t overload either host.
1:#!/bin/zsh
2:
3:tempdir="/tmp/sync.$$"
4:filelist="files.lst"
5:basedir="/doc_root"
6:touchfile="/tmp/.sync"
7:dtg=$(perl -M'English' -e 'print $BASETIME."n";')
8:report="/tmp/sync.$dtg"
9:
10:
11:[ -d $tempdir ] && {
12: echo "$0: $tempdir already exists - exiting ..."
13: exit 1
14:}
15:
16:mkdir $tempdir || {
17: echo "$0: mkdir $tempdir failed - exiting ..."
18: exit 1
19:}
20:
21:cd $tempdir || {
22: echo "$0: $tempdir not accessible - exiting ..."
23: exit 1
24:}
25:
26:/bin/ssh hostA /bin/find $basedir -type f -a
-newer $touchfile > $filelist 2> /dev/null
27:#/bin/ssh hostA /bin/find $basedir -type f -a
-mtime -1 > $filelist 2> /dev/null
28:
29:for file in $(<$filelist)
30:do
31: echo "copying $file"
32: echo "copying $file" >> $report
33:
34: /bin/scp hostA:$file . 2> /dev/null
35:
36: /bin/scp ${file##*/} hostB:$file 2> /dev/null
37:
38: check=$(/bin/ssh hostB /bin/ls -ld "${file}" 2>&1)
39:
40: [[ $(echo $check | grep -c "No such") -gt 0 ]] && {
41: echo "$0:$dtg:$file failed" >> $report
42: }
43:
44: /bin/rm ${file##*/}
45:done
46:
47:/bin/ssh hostA /bin/touch $touchfile 2> /dev/null
48:
49:/bin/rm $filelist
50:
51:cd /tmp
52:
53:rmdir $tempdir
54:
55:exit 0
See reference to download this code.
[/example]
[reference]
[tags]synchronized files, pull and push files, unix scp, scp, UNIX Coding School[/tags]
[/reference]
