Month: December 2007

[problem]

You want to find a pattern in a file and have the line displayed.

[/problem]

[solution]

I “discovered” this undocument (well in all the doco I’ve ever read), pretty much by accident.

Basically find produces a list of files (type f), in the current directory and supplies them individually to the grep command. Ordinarily if grep is supplied a single file, it just returns the pattern. You can suffix grep with -n to also return the line number and -l to return file names without the pattern.

But you cannot (AFAIK) force grep to return the filename and the pattern, from a list of files supplied by find. Of course you could use xargs, to produce a similar mechanism – but that is yet another process to fire up. 🙂

Also you could call a script, instead of grep – which uses something like sed to substitute the beginning of the line with filename – sed “s/^/$1: /” for example.

See the example.

[/solution]

[example]

Anyway, I’m quite happy just using this little secret. 🙂 Having used it many times:

find . -type f -exec grep -i "pattern" {} /dev/null ;

[/example]

[reference]

[tags]find undocumented secret, Unix Coding School[/tags]

• Linux Man Pages – find command
• Linux Man Pages – grep command
• Linux Man Pages – sed command

[/reference]

[problem]

Ever need to look at a file, but have not been able to – due to hidden control characters, etc.

[/problem]

[solution]

Or just needed to see spaces and end of line characters, comparing files, etc. Optical dump or more specifically the od command, can be extremely useful for display ascii characters. See the example.

[/solution]

[example]

The basic syntax looks like this:

od -c filename
Heres a demo:
echo "hi
> world
> " | od -c
0000000 h i n w o r l d n n
0000012
Also useful if wanted to look at a binary file, without breaking your terminal. 🙂 Or looking through encryption.

Also an easy converting of ascii to hex:
dd if=/dev/urandom count=1 bs=128 | od -x
1+0 records in
1+0 records out
0000000 5715 4ac5 e457 9c7b 013b 94f8 cdb7 36b7
0000020 1ae0 a650 e236 c4e3 589f 4af2 dddd f67c
0000040 f722 58b3 7e4e 1de4 6de4 36fb 2c8a 6864
0000060 c682 f951 f966 108f b9dd eaca aefd 8a8a
0000100 da45 7e0f a3ae 350f b0b8 a1f8 1dbe aa4c
0000120 aad8 7002 f0ec 2a94 076a 70f9 0d6f cf44
0000140 1cb2 88b6 9db2 258c 6b4e 08f7 bc41 e6e2
0000160 5ec1 101a f849 b50c 322a 43cf 1520 7dba
0000200

[/example]

[reference]

[tags]optical dump, binary format, Unix Coding School[/tags]

• Linux Man Pages – od command

[/reference]

[problem]

1. You want to compare 2 streams, with byte offset and line numbers?
2. Pull off content with a byte offset?
3. Display lines before or after a pattern?
4. Send an email – upon job failure?

[/problem]

[solution]

The shell is a powerful beast! Anything is possible … with a little know-how. 🙂 See the examples.

[/solution]

[example]

1. md5sum TARBALL | awk '{print $1;}' | cmp - TARBALL.md5
   # cmp good for comparing output with file contents, or diffing files. Outputs byte offset and line number. md5sum creates a hash (checksum) of file.
2. dd bs=1 skip=106 if=infile
   # disk to disk copy, example outputs infile with offset of 106 bytes (block size of 1 byte, skip 106 bytes).
3. Ever needed to display a line before or after a pattern. Yep sure you can use Perl, but this is a bit simplier to work out.

   See the code here

   See a run through screen shot here

   Usage – ppgrep #1 #2 pattern file – where #1 is the number of lines prior to the pattern (could be 0) and #2 is number of lines post pattern.

4. Sending email upon UNIX cron failure

[/example]

[reference]

[tags]dd, cmp, md5sum, grep, sed, Unix Coding School[/tags]

• Linux Man Pages – dd command
• Linux Man Pages – cmp command
• Linux Man Pages – md5sum command
• Linux Man Pages – grep command
• Linux Man Pages – sed command
• Linux Man Pages – crontab command

[/reference]

[problem]

1. You want to generate a self-signed certificate?
2. Encrypt a message with triple des
3. View base 64 encoded data
4. Generate and use random data
5. Test ciphers
6. Monitor certificate expiry dates

[/problem]

[solution]

openssl to the rescue! 🙂 See examples.

[/solution]

[example]

1. openssl code to generate self-signed cert
2. encrypt with tripledes
3. openssl base64 [ -d ]
   # encrypted with base64.
4. Useful fortransfering control/binary content – like this:

   dd bs=1 count=512 if=/dev/urandom | openssl base64

   Also useful for reading mail

   openssl base64 -d /var/spool/mqueue/qfk3GAlBMn018552

5. Using openssl s_client to test ciphers supported, with automated script – then loop sites and email.
6. Using openssl s_client to extract expiry dates, with automated script – then loop sites and email.

[/example]

[reference]

[tags]openSSL, dd, UNIX, SSL Certificates, PKI, Encryption, Generate Random Data, Unix Coding School[/tags]

• [ Linux Man Pages – openSSL ]
• [ Linux Man Pages – dd ]

[/reference]

[problem]

You want to send an attachment, from UNIX command line or from within a shell script

[/problem]

[solution]

uuencode works a treat with Solaris. Not sure on RH Linux, etc.

[/solution]

[example]


cat filename | uuencode wat_u_want_attatch_2b_called.ext | mail -s "this is the subject and here is wat u want attach 2b called" email_addr


Simple as that, it will send the filename as an attachment to a blank email, with your subject.

Here is a demo:

cat index.php | uuencode index.php | mail -s "test uuencode" [email protected]

To do multiple attachments, simply cat files – then uuencode them into a temp file. Like this:


cat file1.doc | uuencode attach1.doc > /tmp/send1.doc

Repeat this for say file2 into attach2 and send2. Then to send them all do this:

cat /tmp/send[0-9].doc | mail -s "your subj" youraddr

[/example]

[reference]

[tags]uuencode, Unix Coding School[/tags]

• Linux Man Pages – uuencode command
• Linux Man Pages – mail command
• Linux Man Pages – cat command

[/reference]

[problem]

You want to make some backups with tar.

[/problem]

[solution]

tar is very functional and has many uses.

[/solution]

[example]

If you are lucky enough to have GNU tar (tar –version will return GNU if so), then you can compress and backup at the same time. 🙂

tar zcvf /tmp/filename.tgz .

This backups and zips the contents of the current directory, to filename.tgz in tmp directory.

If not – no drama. This will backup and pipe through content to gzip, which then zips it.

tar cvf - . | gzip --best > /tmp/filename.tgz

Recreate a directory:

mkdir new

tar -C old -cf - . | tar -C new -xvf

Restore over the network:

ssh remote_hostname "(tar -cf - remote_directory)" | tar -xvf -

For example:

ssh bree "(tar -cf - Perl_Bin)" | tar -xvf -

[/example]

[reference]

[tags]tar, backups, restores, Unix Coding School[/tags]

• Linux Man Pages – tar command
• Linux Man Pages – ssh command
• Linux Man Pages – gzip command

[/reference]

[problem]

You want to forward all traffic from a port on your box, to another host/port combination.

[/problem]

[solution]

The command in the example will port forward, from the local box – any traffic sent to port 8081, will be sent to port 8080 on www.example.com:

[/solution]

[example]

ssh -x -g -L 8081:www.example.com:8080 localhost

Don’t forget, if you need to open LINUX (FC4) firewall, do this (as root):

vi /etc/sysconfig/iptables # add following line then save, exit
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8081 -j ACCEPT
/etc/init.d/iptables restart

[/example]

[reference]

[tags]Port forwarding, SSH, Unix Coding School[/tags]

• Linux Man Pages – crontab command
• Linux Man Pages – ssh command
• Linux Man Pages – vim command
• Linux Man Pages – iptables command

[/reference]

[problem]

You want to receive an email upon cron job failure.

[/problem]

[solution]

The example code can be wrapped around a normal cronjob, to send an email if the cron fails.

[/solution]

[example]

Add to beginning:

bash -c '( msg=`

Add to the end:

2>&1 > /tmp/logfail.log`; if [ ! -z "$msg" ] ; then echo "$msg" |

/bin/mail -s "`/bin/uname -n`:$LOGNAME:`/bin/date`: YOUR MESSAGE" YOUR_MAIL_LIST; fi )'

Demo

# # # # # bash -c '( msg=`YOUR COMMAND 2>&1 > /dev/null`; if [ ! -z "$msg" ] ; then echo "$msg" |/bin/mail -s "`/bin/uname -n`:$LOGNAME:`/bin/date`: YOUR MESSAGE" YOUR_MAIL_LIST; fi )'

Example of generating an email upon cron failure

Another example

[/example]

[reference]

[tags]bash, cron email, Unix Coding School[/tags]

• Linux Man Pages – crontab command
• Linux Man Pages – bash command
• Linux Man Pages – mail command
• Linux Man Pages – uname command
• Linux Man Pages – date command

[/reference]

[problem]

You want to display line lengths – then average line length of file.

[/problem]

[solution]

Snip using awk – see example.

[/solution]

[example]


awk ' { thislen=length($0); printf("%-5s %d\n", NR, thislen); totlen+=thislen}
END { printf("average: %d\n", totlen/NR); } ' filename

[/example]

[reference]

gawk, Unix Coding School

• Linux Man Pages – gawk command

[/reference]

[problem]

Had to find out how to allow a range with iptables recently, whilst setting up Samba and NFS.

[/problem]

[solution]

Always take a backup first, then vi /etc/sysconfig/iptables. In the example I am allowing access to all ports between 600 and 699 – for udp and tcp traffic.

[/solution]

[example]

# grep 600 /etc/sysconfig/iptables
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 600:699 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 600:699 -j ACCEPT
Then run /etc/init.d/iptables restart

You can test if this connection is now viable, by using my Perl port testing code under Network connections – Perl Coding School

Other gotchas with samba and nfs, was fiddling with selinux and ensuring portmap was running, etc. Happy to explain further, feel free to post a comment with your questions.

[/example]

[reference]

[tags]iptables, Unix Coding School[/tags]

• Linux Man Pages – iptables command
• Linux Man Pages – vim command

[/reference]