Archives for : unix

sqlite 10 sec guide

Problem

You want to get up and running with sqlite



Solution

Install sqlite3 on linux, via yum: yum install sqlite3

Then just run sqlite dbname



Example



% sqlite3 testdata.db
SQLite version 3.5.9
Enter ".help" for instructions
sqlite> create table memos(text,priority INTEGER);
sqlite> insert into memos values('deliver project description',10);
sqlite> insert into memos values('lunch with Christine',100);
sqlite> select * from memos;
deliver project description|10
lunch with Christine|100
sqlite>





starting binary – library file not found

Problem

Getting an error about not being able to find libraries (file not found), when starting up a binary – such as httpd.



Solution

You need to identify the shared libraries required by your binary. Ordinarly we use the ldd command for this and amended LD_LIBRARY_PATH to find the libs – see example below.

You might also want to get the shared libs of a running process – with pldd or need to dig into a lib/binary.



Example


# find dependencies for a binary


% ldd /usr/bin/httpd
libpng.so.2 => /usr/local/lib/libpng.so.2
libcrypt_i.so.1 => /usr/lib/libcrypt_i.so.1
libresolv.so.2 => /usr/lib/libresolv.so.2

# demo of how ldd walks the values in LD_LIBRARY_PATH


% unset LD_LIBRARY_PATH

% ldd /usr/bin/httpd
libpng.so.2 => (file not found)
libcrypt_i.so.1 => /usr/lib/libcrypt_i.so.1
libresolv.so.2 => /usr/lib/libresolv.so.2

# find files installed via a package on Solaris


% grep libpng.so.2 /var/sadm/install/contents | head -1
/usr/sf/lib/libpng.so=libpng.so.2 s none SUNWpng

Another way to see which libraries are being picked up, i.e. if a binary works on one system and not another:


% truss -t'open' -f date
21667: open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT
21667: open("/usr/lib/libc.so.1", O_RDONLY) = 3
21667: open("/usr/lib/libdl.so.1", O_RDONLY) = 3
...

Getting the dependent shared libraries of a running process (especially where vars might have changed since startup):


% pldd 24832
24832: /usr/bin/httpd
/usr/local/lib/libpng.so.2.1.0.5
/usr/lib/libcrypt_i.so.1
/usr/lib/libresolv.so.2
...

Seeing more detail via the headers within an object file (use objdump on linux):


% elfdump /usr/bin/httpd | grep NEEDED
[0] NEEDED 0x15755 libpng.so.2
[1] NEEDED 0x15761 libcrypt_i.so.1
[2] NEEDED 0x15771 libresolv.so.2
...

See configure parameters passed to a binary during compilation (can also be useful for getting a usage out a binary, that does not appear by default or with –h, etc):





Stripping out newlines from string

Problem

Different versions of UNIX handle the stripping of newlines in a string differently. Obviously if you echo a string, the shell automatically expands newlines “n”.



Solution

sed and tr don’t appear to like performing these substitutions. But with Perl its a steal. 🙂



Example



% echo 'hello worldnhello worldntest it' | perl -ane 's#\n# #g; print,"";'

hello world hello world test it



Reference

[tags], Unix Coding School[/tags]



Resize image command line – mogrify

Problem

You want to resize images from the command line.



Solution

Use image magik mogrify command.


mogrify -resize "percentage" image_na

Also you can view current sizing – like this:


% identify IMG_3864.JPG
IMG_3864.JPG JPEG 563x422 563x422+0+0 DirectClass 8-bit 32.8887kb



Example



mogrify -resize 20% IMG_3705.JPG



Reference

[tags]mogrify[/tags]



UNIX Pipes – event triggers action

Problem

You want to listen on a pipe and perform commands, based on the text sent to the pipe.

This could be useful for triggering action based on events;
Running code as different users – i.e. allow root or a functional user to run something – just need to allow access to group of users, via permission to write to the pipe.



Solution


1. mknod /tmp/.restart_crd

2. chmod a+w /tmp/.restart_crd

3. nohup tail -f /tmp/.restart_crd| while read line
do
case "$line" in
'restart') cmd="./stop >> /tmp/restart.log 2>&1; sleep 5; ./start >> /tmp/restart.log 2>&1";;
'stop') cmd="./stop >> /tmp/restart.log 2>&1";;
'start') cmd="./start >> /tmp/restart.log 2>&1";;
*) cmd="no match ${line}.";;
esac

echo "command is $cmd"
eval $cmd

sleep 5

done &

4. disown



Example



$ echo "restart" > /tmp/.restart_crd

$ echo "start" > /tmp/.restart_crd

$ echo "stop" > /tmp/.restart_crd

Watch output in by tail -f /tmp/restart.log





receive notification when file arrives

Problem

You want to get notified when a file arrives. Maybe it is coming in from somewhere else, or just is the result of some processing.

What ever the case, never sit there waiting for a file to arrive again! 🙂



Solution

Put the code under example tab, in a cron entry. See reference for more detail about cron.

Change the minutes, hour, filename and mail address! 🙂 You may need to use mail instead of mailx and ksh, etc instead of bash. But the basic syntax is the same. 🙂



Example



0-59 12 * * * bash -c "([ -f $HOME/test-touch ] &&
mailx -s "test-touch found at `date`" admin@acme.com



Reference

[tags]cron, UNIX Batch, UNIX Scripts, Unix Coding School[/tags]



UNIX debugging

Problem

  1. You want to debug a running process on Solaris?
  2. You want to debug running process on Linux?


Solution

See the example for debugging processes on Solaris or Linux – the syntax shows all system executes and reads.



Example


  1. truss -xall -vall -rall -t'read' -p PID #

    To run a command and have it produce debug info, run it prefix with the truss syntax:

    truss -xall -vall -rall -t'read' your command #

    Another good option when debugging, is to run truss and look at the opens. Or more specifically the failures – as this often highlights missing libraries or permissions issues.

    truss -t'open' -p PID #

  2. On Linux use strace, which takes basically the same options. You just use -e, like this:

    strace -e'open' ls
    open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/etc/ld.so.cache", O_RDONLY) = 3
    open("/lib/tls/librt.so.1", O_RDONLY) = 3
    open("/lib/libtermcap.so.2", O_RDONLY) = 3
    open("/lib/libacl.so.1", O_RDONLY) = 3
    open("/lib/tls/libc.so.6", O_RDONLY) = 3
    open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
    open("/lib/libattr.so.1", O_RDONLY) = 3
    open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
    open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
    open("/etc/mtab", O_RDONLY) = 3
    open("/proc/meminfo", O_RDONLY) = 3



Reference

[tags]UNIX Debugging, Solaris Debugging, Linux Debugging, truss, strace, Unix Coding School[/tags]



Shell Tips and Tricks

Problem

  1. You want to trace program execution on Solaris?
  2. You want to perform network tracing on Solaris?
  3. Perform a text dump of a binary file – or see ascii codes, etc?
  4. Generate some random data?
  5. List directory contents in 3 columns?
  6. Loop through a list of files and perform actions on them?
  7. Check for values in vars?
  8. Detach a process without it dying.
  9. Perform arithmetic in shell.


Solution

Some of the most useful Shell Tricks I’ve learned – check them out at Example – enjoy! 🙂



Example


  1. System process debug (solaris) all exec/reads

    truss -xall -vall -rall -t'read' -p PID

  2. Network trace connection from hostname (solaris)

    snoop -x0 src hostname

  3. Optical dump of a file, useful for seeing control chars, etc and lines with no newline.

    od -c filename

  4. Generate 512 bytes of random data, cat -ve escapes control chars.

    dd bs=1 count=512 if=/dev/urandom | cat -ve

  5. list directory contents, display in 3 columns.

    ls | paste - - -

  6. Loop through all files and perform an action on it.

    for na in * ; do echo $na ; done

  7. Check for values in vars.

    [[ $(echo $var | grep -ic "pattern") -eq 0 ]] &

    You could also go for "-eq 1" – does contain or -ne 0 for same, etc. Plus you could use || for "or" either in place of "&&" or after this block – same as if .. then "positive" else "negative" fi.

  8. Detach a process without it dying.

    disown [%1]

    nohup command > /tmp/logfile.log 2>&1&

    If you start a process with nohop (ignore the hup signal), it will not exist (usually) when your shell is exited. This can now be simulated on solaris, by backgrounding a process (with ctrl Z and bg) then typing disown. jobs command will show all backgrounded jobs. Ideally start commands like this, to avoid the shell exit problem:

  9. Perform arithmetic in shell.

    bc -l

    Binary Calculator in long format – for float, etc. Can also be automated like this, to convert between decimal and hex:

    i=0; while ((i<20)) ; do echo -n "$i: "; echo "base=10;obase=16;$i" | bc; ((i++)); done | paste - - - -
    0: 0 1: 1 2: 2 3: 3
    4: 4 5: 5 6: 6 7: 7
    8: 8 9: 9 10: A 11: B
    12: C 13: D 14: E 15: F
    16: 10 17: 11 18: 12 19: 13



Reference

[tags]Unix Shell Tips, Unix Shell Tricks, UNIX Arithmetic, Optical Dump, Unix Coding School[/tags]



undocument find secret

Problem

You want to find a pattern in a file and have the line displayed.



Solution

I “discovered” this undocument (well in all the doco I’ve ever read), pretty much by accident.

Basically find produces a list of files (type f), in the current directory and supplies them individually to the grep command. Ordinarily if grep is supplied a single file, it just returns the pattern. You can suffix grep with -n to also return the line number and -l to return file names without the pattern.

But you cannot (AFAIK) force grep to return the filename and the pattern, from a list of files supplied by find. Of course you could use xargs, to produce a similar mechanism – but that is yet another process to fire up. 🙂

Also you could call a script, instead of grep – which uses something like sed to substitute the beginning of the line with filename – sed “s/^/$1: /” for example.

See the example.



Example


Anyway, I'm quite happy just using this little secret. 🙂 Having used it many times:

find . -type f -exec grep -i "pattern" {} /dev/null ;



Reference

[tags]find undocumented secret, Unix Coding School[/tags]



Optical Dump

Problem

Ever need to look at a file, but have not been able to – due to hidden control characters, etc.



Solution

Or just needed to see spaces and end of line characters, comparing files, etc. Optical dump or more specifically the od command, can be extremely useful for display ascii characters. See the example.



Example


The basic syntax looks like this:

od -c filename
Heres a demo:
echo "hi
> world
> " | od -c
0000000 h i n w o r l d n n
0000012

Also useful if wanted to look at a binary file, without breaking your terminal. 🙂 Or looking through encryption.

Also an easy converting of ascii to hex:
dd if=/dev/urandom count=1 bs=128 | od -x
1+0 records in
1+0 records out
0000000 5715 4ac5 e457 9c7b 013b 94f8 cdb7 36b7
0000020 1ae0 a650 e236 c4e3 589f 4af2 dddd f67c
0000040 f722 58b3 7e4e 1de4 6de4 36fb 2c8a 6864
0000060 c682 f951 f966 108f b9dd eaca aefd 8a8a
0000100 da45 7e0f a3ae 350f b0b8 a1f8 1dbe aa4c
0000120 aad8 7002 f0ec 2a94 076a 70f9 0d6f cf44
0000140 1cb2 88b6 9db2 258c 6b4e 08f7 bc41 e6e2
0000160 5ec1 101a f849 b50c 322a 43cf 1520 7dba
0000200



Reference

[tags]optical dump, binary format, Unix Coding School[/tags]



Shell Utilities

Problem

  1. You want to compare 2 streams, with byte offset and line numbers?
  2. Pull off content with a byte offset?
  3. Display lines before or after a pattern?
  4. Send an email – upon job failure?


Solution

The shell is a powerful beast! Anything is possible … with a little know-how. 🙂 See the examples.



Example


  1. md5sum TARBALL | awk '{print $1;}' | cmp - TARBALL.md5
    # cmp good for comparing output with file contents, or diffing files. Outputs byte offset and line number. md5sum creates a hash (checksum) of file.
  2. dd bs=1 skip=106 if=infile
    # disk to disk copy, example outputs infile with offset of 106 bytes (block size of 1 byte, skip 106 bytes).
  3. Ever needed to display a line before or after a pattern. Yep sure you can use Perl, but this is a bit simplier to work out.

    See the code here

    See a run through screen shot here

    Usage – ppgrep #1 #2 pattern file – where #1 is the number of lines prior to the pattern (could be 0) and #2 is number of lines post pattern.

  4. Sending email upon UNIX cron failure




openssl many uses

Problem

  1. You want to generate a self-signed certificate?
  2. Encrypt a message with triple des
  3. View base 64 encoded data
  4. Generate and use random data
  5. Test ciphers
  6. Monitor certificate expiry dates


Solution

openssl to the rescue! 🙂 See examples.



Example


  1. openssl code to generate self-signed cert
  2. encrypt with tripledes
  3. openssl base64 [ -d ]
    # encrypted with base64.
  4. Useful fortransfering control/binary content – like this:

    dd bs=1 count=512 if=/dev/urandom | openssl base64

    Also useful for reading mail

    openssl base64 -d /var/spool/mqueue/qfk3GAlBMn018552

  5. Using openssl s_client to test ciphers supported, with automated script – then loop sites and email.
  6. Using openssl s_client to extract expiry dates, with automated script – then loop sites and email.


Reference

[tags]openSSL, dd, UNIX, SSL Certificates, PKI, Encryption, Generate Random Data, Unix Coding School[/tags]



uuencode send attachments

Problem

You want to send an attachment, from UNIX command line or from within a shell script



Solution

uuencode works a treat with Solaris. Not sure on RH Linux, etc.



Example



cat filename | uuencode wat_u_want_attatch_2b_called.ext | mail -s "this is the subject and here is wat u want attach 2b called" email_addr

Simple as that, it will send the filename as an attachment to a blank email, with your subject.

Here is a demo:

cat index.php | uuencode index.php | mail -s "test uuencode" junk@techie-blogs.com

To do multiple attachments, simply cat files – then uuencode them into a temp file. Like this:


cat file1.doc | uuencode attach1.doc > /tmp/send1.doc

Repeat this for say file2 into attach2 and send2. Then to send them all do this:

cat /tmp/send[0-9].doc | mail -s "your subj" youraddr





tar many uses

Problem

You want to make some backups with tar.



Solution

tar is very functional and has many uses.



Example


If you are lucky enough to have GNU tar (tar –version will return GNU if so), then you can compress and backup at the same time. 🙂

tar zcvf /tmp/filename.tgz .

This backups and zips the contents of the current directory, to filename.tgz in tmp directory.

If not – no drama. This will backup and pipe through content to gzip, which then zips it.

tar cvf - . | gzip --best > /tmp/filename.tgz

Recreate a directory:

mkdir new

tar -C old -cf - . | tar -C new -xvf

Restore over the network:

ssh remote_hostname "(tar -cf - remote_directory)" | tar -xvf -

For example:

ssh bree "(tar -cf - Perl_Bin)" | tar -xvf -



Reference

[tags]tar, backups, restores, Unix Coding School[/tags]



Port forwarding with SSH

Problem

You want to forward all traffic from a port on your box, to another host/port combination.



Solution

The command in the example will port forward, from the local box – any traffic sent to port 8081, will be sent to port 8080 on www.example.com:



Example


ssh -x -g -L 8081:www.example.com:8080 localhost

Don't forget, if you need to open LINUX (FC4) firewall, do this (as root):

vi /etc/sysconfig/iptables # add following line then save, exit
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8081 -j ACCEPT
/etc/init.d/iptables restart





Email upon cron failure

Problem

You want to receive an email upon cron job failure.



Solution

The example code can be wrapped around a normal cronjob, to send an email if the cron fails.



Example


Add to beginning:

bash -c '( msg=`

Add to the end:

2>&1 > /tmp/logfail.log`; if [ ! -z "$msg" ] ; then echo "$msg" |

/bin/mail -s "`/bin/uname -n`:$LOGNAME:`/bin/date`: YOUR MESSAGE" YOUR_MAIL_LIST; fi )'

Demo

# # # # # bash -c '( msg=`YOUR COMMAND 2>&1 > /dev/null`; if [ ! -z "$msg" ] ; then echo "$msg" |/bin/mail -s "`/bin/uname -n`:$LOGNAME:`/bin/date`: YOUR MESSAGE" YOUR_MAIL_LIST; fi )'

Example of generating an email upon cron failure

Another example





awk line length and average

Problem

You want to display line lengths – then average line length of file.



Solution

Snip using awk – see example.



Example



awk ' { thislen=length($0); printf("%-5s %d\n", NR, thislen); totlen+=thislen}
END { printf("average: %d\n", totlen/NR); } ' filename



Reference

gawk, Unix Coding School



Linux IPTables Open port range.

Problem

Had to find out how to allow a range with iptables recently, whilst setting up Samba and NFS.



Solution

Always take a backup first, then vi /etc/sysconfig/iptables. In the example I am allowing access to all ports between 600 and 699 – for udp and tcp traffic.



Example


# grep 600 /etc/sysconfig/iptables
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 600:699 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m state --state NEW -m udp --dport 600:699 -j ACCEPT

Then run /etc/init.d/iptables restart

You can test if this connection is now viable, by using my Perl port testing code under Network connections – Perl Coding School

Other gotchas with samba and nfs, was fiddling with selinux and ensuring portmap was running, etc. Happy to explain further, feel free to post a comment with your questions.



Reference

[tags]iptables, Unix Coding School[/tags]



Encrypt TripleDES

Problem

You want to encrypt some text, using the tripleDES Cipher.



Solution

Openssl is a beautiful command, that performs many functions. See example.



Example


Openssl tripleDES encrypt command:
openssl des3 -salt -in file_to_encrypt -pass pass:_your_password_

Security

Place contents to encrypt in the file reference by file_to_encrypt

Replace your_password with your secret

Openssl will output to stdout – so best to capture like this:
myvar=$(the openssl command)
Openssl tripleDES decrypt command:
openssl des3 -d -salt -in file_to_encrypt -pass pass:_your_password_
Put cipher text to decrypt in filename supplied to -in

Replace _your_password_ with the secret

Outputs to stdout
See the full demo attached here for a blow by blow:

[ How to encrypt with openssl and tripleDES ]



Reference

[tags]openssl, encryption, tripleDES, passwords[/tags]



Generate self-signed cert openssl

Problem

You want to generate a self-signed certificate, for use with a web server.

This will allow you to communicate with your web server over HTTPS, effectively encrypting your traffic.



Solution

The very first time – you need to set up your own CA (certifying authority). Do this one time only!

[ See a run through screen shot here ] [ See the code here ]

Then create a certificate request – which can be sent off to Verisign, etc or self-signed.



Example


To create a certificate request:

openssl req -new -days 730 -keyout keyna.pem -out keyna.pem

[ See a run through screen shot here ]

Self-sign the request – only do this if you are not sending off to a CA such as Verisign.

openssl ca -policy policy_anything -days 730 -out certna.pem -infiles keyna.pem

[ See a run through screen shot here ]

If you get 'unable to write random state', try this: look for command ssh-rand-helper usually under ssh install directories (or if ~/.ssh/.prng_seed exists – you can just link to it or copy it to $HOME).

ssh-rand-helper -b 1024 > $HOME/.prng_seed

To strip the password out, for restarts:
openssl rsa -in keyna.key -out keyna.key.unsecure



Reference

[tags]openssl, encryption, tripleDES, passwords, Unix Coding School[/tags]



Testing Cipher Strength

Problem

You want to be sure that your web site is only serving 128 bits!



Solution

Use openssl s_client – with specific ciphers – see the example.



Example


The following script spins around 40, 56 and 64 bit ciphers – to check the lowest a site allows.

Ideally none of these ciphers should be allowed (check out www.modssl.org for apache config – look for ciphersuite).

openssl_cipher_check will produce the cipher details from the cipher codes returned in connection output.

If you need proxy support with s_client comment on this thread.


#!/bin/bash

[ $# -lt 1 ] && {

echo "$0: site:port"
echo "Usage: $0 www.example.com:443"
exit 0
}

fortyciphers='EXP-EDH-RSA-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:EXP-RC2-CBC-MD5:EXP-RC4-MD5:EXP-RC4-MD5'

fiftysixciphers='EXP1024-DHE-DSS-DES-CBC-SHA:EXP1024-DES-CBC-SHA:EXP1024-RC2-CBC-MD5:EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:DES-CBC-SHA:DES-CBC-MD5:EXP1024-DHE-DSS-RC4-SHA:EXP1024-RC4-SHA:EXP1024-RC4-MD5'

sixtyfourciphers='RC4-64-MD5'

grabCipher() {

cipher=$1
site=$2

echo "
GET / HTTP/1.0

EOT
" | openssl s_client -connect $site -cipher $cipher

}

site=$1

echo "$site"
echo $site | sed 's/./-/g'

for na in forty fiftysix sixtyfour
do

eval ciphers="$${na}ciphers"

thiscipher=`grabCipher $ciphers $site 2>&1 | awk ' /Cipher/ { print $NF } '`

[[ $(echo $thiscipher | grep -c "^$") -ne 1 ]] && {

openssl ciphers -v $thiscipher
exit 0
}

done

echo "No 40, 56 or 64 bit ciphers supported"

exit 0





SSL Certification Expiry Date Checker

Problem

You want to automate checking expiry of SSL (HTTPS) Certificate expiry.



Solution

The link to the script is displayed in examples. It connects to the given site and shows site expiry. This can then be automated into a database, using some of my PHP or Perl Scripts, which can also be used to then report on sites about to expire.



Example



./openssl_cert_expiry_check www.example.com:443

Here is the code – but take note you may need the openssl client that supports proxying. Leave me a comment if you want this code.


#!/bin/bash

[ $# -ne 1 ] &echo -n "$1 - "

echo "
GET / HTTP/1.0

EOT
" | openssl s_client -connect $1 2>&1 |
sed -n '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/p' |
openssl x509 -enddate |
awk -F= ' /notAfter/ { printf("Expires: %sn",$NF); } '

exit 0





Compare Java Certificate Stores – cacerts

Problem

You want to compare two cacerts java certificate stores.

Say comparing development server against prod, to ensure they both have same certs stored in cacerts (sampled by java among others).



Solution

This can come in very handy, if you need to compare 2 different projects or envs. See the example.



Example


Code to generate list of certs, along with there alias, entry, owner and valid dates.

Start in the directory where your cacerts file resides.


../../../bin/keytool -list -v -keystore cacerts -storepass changeit | gawk ' {
if(/Alias name/) {
if(alias != /^$/) { printf("%s: [%s] %s [from: %s]n",alias,entry,owner,valid); }
alias=$NF;
}
if(/Entry type/) { entry=$NF; }
if(/Owner/) { owner=$2" "$3; }
if(/Valid/) { valid=$5"/"$4"/"$8; }
} END { printf("%s: [%s] %s [from: %s]n",alias,entry,owner,valid); }
' | sort

You may need to change gawk for nawk on Solaris, etc.

Dump this out into a file for each env. The run diff with something like this:


diff -s dump.cob dump.dev | egrep -v -- "^([0-9]|--)" | sort | grep -vn xxx



Reference

[tags]openssl, cacerts, java certifications, comparing java certs, Unix Coding School[/tags]



Generate Random Data with dd

Problem

You want to generate 512 bytes of random data, using UNIX dd (disk to disk) command.



Solution

Using /dev/random blocks waiting for truly random data, whereas /dev/urandom just pumps out whatever is available.

You need to escape the control characters, or else it will trash your screen. cat -ve escapes these control characters for you.

See examples



Example


dd bs=1 count=512 if=/dev/urandom | cat -ve

dd is a very powerful command, that can be used to read files, data, tapes, even disks, etc – varying the number of blocks with count and block size with bs.

if is the input device and of can be used to designate an output device.

dd bs=1 count=512 if=/dev/urandom | openssl base64

Also it can convert to upper and lower case, with conv=ucase, etc.

If you have an extremely large file, it can open at an offset – via the skip option, for example to skip the first 1k of the file and read 100 bytes:


dd if=access_log.techieblogs bs=1 skip=1024 count=1000
- - [04/Jun/2006:00:13:10 +0800] "GET /mysqldemo/run_q.php?database=mysql HTTP/1.1" 200 1364



Reference

[tags]dd, pseudo random data, cat, Unix Coding School[/tags]



While For Loops

Problem

You want to loop around a number of times and perform an action.



Solution

Being able to loop around a given number of times, fully utilizes the power of UNIX. See the example.



Example


For example, spin around 100 times and print hello:

i=0; while [[ $i -lt 100 ]] ; do echo -n "hello"; ((i++)) ; done; echo ""
hellohellohellohellohellohellohellohellohellohello
hellohellohellohellohellohellohellohellohellohello
hellohellohellohellohellohellohellohellohellohello
hellohellohellohellohellohellohellohello...

Similarly the for command is a beauty too, spin around 5 times and kick off a loop 5 times – paste the result into 5 columns:


for i in 1 2 3 4 5; do for j in 1 2 3 4 5; do echo "$i:$j"; done; done | paste - - - - -
1:1 1:2 1:3 1:4 1:5
2:1 2:2 2:3 2:4 2:5
3:1 3:2 3:3 3:4 3:5
4:1 4:2 4:3 4:4 4:5
5:1 5:2 5:3 5:4 5:5



Reference

[tags], Unix Coding School[/tags]



Split output over columns

Problem

Ever wanted to produce columns of output, rather than your output spilling off the screen.



Solution

Useful command for this is paste, see example for how it works – to split output into 3 columns.



Example


ls | paste - - -
comments.php comments-popup.php fat.js
footer.php header.php index.php
screenshot.png sidebar.php style.css

As you'd expect with UNIX, that is not the end of this commands uses! 🙂

Create a comma seperated list:

ls | paste - - - -d,
comments.php,comments-popup.php,fat.js
footer.php,header.php,index.php
screenshot.png,sidebar.php,style.css

Then courtesy of the LINUX info command:


cat num2
1
2
$ cat let3
a
b
c
$ paste num2 let3
1 a
2 b
c

And:


$ paste -s num2 let3
1 2
a b c



Reference

[tags]paste, Unix Coding School[/tags]



Looping with For and While

Problem

You want to loop a number of times in UNIX.



Solution

Being able to loop around a given number of times, fully utilizes the power of UNIX. See the examples.



Example


For example, spin around 100 times and print hello:

i=0; while [[ $i -lt 100 ]] ; do echo -n 'hello'; ((i++)) ; done; echo ''
hellohellohellohellohellohellohellohellohellohello
hellohellohellohellohellohellohellohellohellohello
hellohellohellohellohellohellohellohellohellohello
hellohellohellohellohellohellohellohello...

Similarily the for command is a beauty too, spin around 5 times and kick off a loop 5 times – paste the result into 5 columns:


for i in 1 2 3 4 5; do for j in 1 2 3 4 5; do echo "$i:$j"; done; done | paste - - - - -
1:1 1:2 1:3 1:4 1:5
2:1 2:2 2:3 2:4 2:5
3:1 3:2 3:3 3:4 3:5
4:1 4:2 4:3 4:4 4:5
5:1 5:2 5:3 5:4 5:5





Check var for pattern

Problem

You want to check a variable for a given pattern.



Solution

This is my personal favorite way to check for values in vars. See examples.



Example


[[ $(echo $var | grep -ic "pattern") -eq 0 ]] &

You could also go for "-eq 1" – does contain:

[[ $(echo $var | grep -ic "pattern") -eq 1 ]] &

Or "-ne 0" for same, etc.

[[ $(echo $var | grep -ic "pattern") -ne 0 ]] &

Plus you could use || for "or" either in place of "&&" or after this block – same as if .. then "positive" else "negative" fi.

[[ $(echo $var | grep -ic "pattern") -eq 0 ]] || { echo "var does contain pattern" }



Reference

[tags], Unix Coding School[/tags]



Detach proc from shell

Problem

When you exit your shell, the processes you’ve kicked off die.



Solution

If you start a process with nohop (ignore the hup signal), it will not exit (usually) when your shell is exited. nohup stands for no hang up, a throw back to terminal lines.



Example


nohup command > /tmp/logfile.log 2>&1&

If you forget to run the command with nohup, this can now be simulated on solaris.

Just background the process (with ctrl Z and bg) then type disown.

disown [%1]

The UNIX command jobs will show all backgrounded jobs. After running disown, it should show no jobs in the background.



Reference

[tags]nohup, disown, daemons, Unix Coding School[/tags]



Binary Calculator Hex

Problem

You want to perform some arithmetic with UNIX.



Solution

The binary calculator, installed with pretty much all version of UNIX, can be used for an array of tasks.

Not just basic calculations, but also converting between bases and it can automated, etc.



Example


It's simpliest use, it just to run bc and add, subtract divide.


bc
888+999
1887

Be aware though, that to perform precision point calculations, you must specify the "-l" long option. For example:


bc
999/222
4

Obviously wrong! 🙂 Now with the long option:


bc -l
999/222
4.50000000000000000000

Can also be automated like this, to convert between decimal and hex:


i=0; while ((i < 20)) ; do echo -n ' $i: '; echo 'base=10;obase=16;$i' | bc; ((i++)); done | paste - - - -
0: 0 1: 1 2: 2 3: 3
4: 4 5: 5 6: 6 7: 7
8: 8 9: 9 10: A 11: B
12: C 13: D 14: E 15: F
16: 10 17: 11 18: 12 19: 13



Reference

[tags]BC, HEX, Unix Coding School[/tags]