Archives for : unix

VI Tips

Problem

Started using vi probably 13 years ago and ever so often, still learn new tricks with it! 🙂

Some of my favorites are: turning on control characters; mapping function keys; performing global search and replace; remapping columns.



Solution

So to the basics, editing a file:


vi filename



Example


Editing a file from a specific line number or pattern:


vi +lineno filename
vi +/patt/ filename

To turn on control characters, once in vi – do the following (exactly)


:set list

To turn it off again:


:set nolist

Global replace:


:%s/patt/rep/g

Map function keys:


:map #1 :w![ctrl v ctrl m]

that is type colon,map,#,1,space,colon,w,!,^M then hit enter. More on this later.

Remap columns:


:%s/(col1patt) (col2patt)/2 1/

To finish here is a cool but slightly bizarre one:


:g/^/m0

It reverses the lines of file. Effectively it says – global (all lines) with this pattern (a beginning line – so all lines), move to 0 – which is shorthand for the current line. So as vi applies the command to each line, it bubbles up to the top.



Reference

[tags]vim, regular expression, Unix Coding School[/tags]



AWK Find field number

Problem

Ever needed to print out one column with awk, but do not know the field number?



Solution

Here I’m running df and looking for a field with “Used” in it:

df -k | head -1 | awk ' {      for(i=1; i< NF;i++) {         if($i ~ /Used/) {             print "# field "i         }      }} '

So stop counting columns and just use the code! 🙂



Example


Now for a real life demo. 😉


[[email protected] marcus]$ df -k | head -1
Filesystem 1K-blocks Used Available Use% Mounted on
[[email protected] marcus]$ df -k | awk ' {
for(i=1;i < NF;i++) {
if($i ~ /Used/) { print "# field "i }
}
} '
# field 3



Reference

[tags], Unix Coding School[/tags]



AWK Print line number

Problem

Ever wanted to display a line number, with each matching record from awk. Useful as a way of counting the number of matching lines too.



Solution

Here’s how with a demo which grabs network stats (netstat) off my linux box. It looks for connections to port 80 and in TIME_WAIT state – prints a line number and the matching line.



Example



[[email protected] marcus]$ netstat -an -t | awk ' /:80 / && /TIME_WAIT/ { print ++i,$0} '
0 tcp 0 0 x.x.x.x:80 x.x.x.x:44154 TIME_WAIT

If you wanted the line number, that matches your pattern – this would show that too:


[[email protected] marcus]$ netstat -an -t | awk ' /:80 / && /TIME_WAIT/ { print "line:"NR,"count:"++i,$0} '
line:17 count:1 tcp 0 0 x.x.x.x:4298 x.x.x.x:80 TIME_WAIT



Reference

[tags]gawk, netstat, print line numbers, Unix Coding School[/tags]



Send HTML email from command line

Problem

Need to send an email from UNIX command line, that contains HTML content?



Solution

Most useful when generating statistics, etc – just spin round and spit out HTML table rows, through sendmail.



Example


Here is the code:


/usr/lib/sendmail -t
To:[email protected]
From:[email protected]
Subject:This is a test
MIME-VERSION:1.0
Content-Type: multipart/mixed; boundary="gc0y0pkb9ex"

--gc0y0pkb9ex
Content-type: text/html;charset="iso-8859-1"

<html><head><title>Test HTML mail</title></head>
<body>
<table border=1>
<tr><td align=center>test mail</td></tr>
<tr><td>This is an HTML email sent from UNIX command line!</td></tr>
</table>
</body></html>
.





Debugging shell scripts

Problem

You need to debug a shell script.



Solution

Most shells (zsh, ksh, bash …) can either run with a minus x (-x) option or by specifying set -x.

If you have an existing shell, that you’d like to run in debug mode – just run it like this: bash -x scriptname. Or ksh -x for korn shell.



Example


You can also edit the follow and put a minus x after the interpreter line, for example:

#!/bin/ksh -x

or

#!/bin/bash -x

Debugging on the command line can be performed, simply by typing set -x. Like this:


$ set -x
+ set -x
$ date
+ date
Tue Jun 13 18:59:44 WST 2006
$ echo "hello world"
+ echo 'hello world'
hello world
$ set +x
+ set +x
$ date
Tue Jun 13 18:59:57 WST 2006



Reference

[tags]Debug Shell Scripts, Unix Coding School[/tags]



Procs started by cron

Problem

Sometimes you want to list all processes that have been initiated by cron.



Solution

To do this you first need to know the Process ID (PID) of the cron process, then to list all children of that process.

Luckily Solaris has ptree and pgrep, so it is simple.



Example



ptree `pgrep cron`

Looks like Linux also has pgrep and pstree:


pstree -p `pgrep cron`



Reference

[tags]Process commands, Unix Coding School[/tags]



Proc run timings

Problem

You want to ascertain how long your script takes to run, displaying thousands of seconds and including CPU utilization.



Solution

Just use the time command.

Really simple and an excellent way to see how long your scripts take. Plus showing the anticipated load on the system.



Example


[[email protected] marcus]$ time ps
PID TTY TIME CMD
27973 pts/0 00:00:00 bash
28120 pts/0 00:00:00 ps

real 0m0.025s
user 0m0.010s
sys 0m0.020s



Reference

[tags], Unix Coding School[/tags]



Ways Display Procs

Problem

You want to display processes, commands within the process tree.



Solution

There are a number of ways to search the process tree.

Here are a few:

Show all processes for user apache.


ps -fu apache

Show all process on a given terminal (try command tty to see your own)


ps -ftpts/0



Example


Show a given process


ps -fpPID



Reference

[tags], Unix Coding School[/tags]



Using FIFOs

Problem

You want to create and use an UNIX FIFO.



Solution

FIFO means first in, first out.

To create a FIFO (also called a named pipe), UNIX supplies the mknod command.



Example


This is the code to generate a FIFO:


$ mknod myFIFO p

Then the resulting named pipe:


$ ls -ld myFIFO
prw-r--r-- 1 marcus adm 0 Jun 13 21:15 myFIFO

To use the FIFO, you need to have a process that is reading from it. This could be a bit of Perl, C code, etc.

A perfect way to demonstrate using FIFOs, is tail -f:


$ tail -f myFIFO

Then a test:


$ echo "
Hello there
This is a test of FIFOs
" > myFIFO

From the tail -f when now see following output:

Hello there
This is a test of FIFOs





Find pattern in file

Problem

You need to find a pattern in a file. Using regular expression.



Solution

Normal command to display something in a file, is grep.


grep "pattern" filename



Example


You can also display the line number it occured on (-n) and ignore case (-i):


grep -n -i "pattern" filename

Also you can match more than one pattern:


egrep "patternA|patternB" filename

To match the beginning of a line (^) or the end of line ($) – this will only match lines with this specific pattern on:


grep "^patternA$" filename

We can also use wildcards and ranges – this will match any line with PATTERN in it followed by any amount of numbers:


grep "^.*PATTERN[0-9]*$" filename



Reference

[tags]grep, regular expression, Unix Coding School[/tags]



SED extract line from file

Problem

You want to extract a specific line number from a file.



Solution

Once you have your line number – see previous post on grep, we can extract lines around the pattern.

To do this sed (stream editor) can be used to print just desired lines – this says don’t print all lines (-n); start at line 456 and finish at line 466 – print:


sed -n 456,466p filename



Example


Also with sed, we can say delete specific lines – in this case remove lines 5 to 10:


sed 5,10d filename

That's not all sed can accept patterns, as start/end identifiers:


sed /start_pattern/,/end_pattern/d filename



Reference

[tags], Unix Coding School[/tags]



SED Tips

Problem

Following on from the last post on removing lines with sed, how do we use sed to substitute output on the fly?



Solution

This says substitute occurences of PATT with REPLACE global (or all occurences).


sed 's/PATT/REPLACE/g' filename



Example


So to replace all occurences of Unix with UNIX:


sed 's/Unix/UNIX/g' filename

We can also replace patterns, with variable values, like this:


sed "s/PATT/${THEVAR}/g" filename



Reference

[tags]SED, Unix Coding School[/tags]



List User Login Details

Problem

Needing to view current activity of users on the system.



Solution

Most flavours of UNIX provide a number of tools, to tell you who is logged in or when they last accessed the system.



Example


A favorite of mine, is who -Hu – really good for seeing who is idle, quite literally! 🙂

Also you can just run last – although beware of rolled logs. Generally in /var/adm/wtmp* on Solaris and /var/log/wtmp* on Linux.

If you want to run last against an older file (by default it just uses wtmpx or wtmp) – just specify minus f – like this: last -f /var/log/wtmp.1

Additionally the finger command will quite often show last login time.





Setting Up SUDO

Problem

You want to allow a user to run one command, as root user.



Solution

To allow a user access to run stuff as root, just login as root and then add the following line to /etc/sudoers.


userid hostname=command



Example


For example to allow user marcus access on bree, to restart, stop, start or check apache:


marcus bree=/etc/init.d/httpd

To allow all access on all boxes:


marcus (ALL)=(ALL)



Reference

[tags]sudo, switch user do, temporary UNIX authorisation, Unix Coding School[/tags]



Pkill User Procs

Problem

Want to kill all processes, for a specific user.



Solution

This command will send a -KILL signal, equivalent to minus 9:


pkill -KILL -U username



Example


So to kill all processes owned by username marcus:


pkill -KILL -U marcus

If you issue kill -l – that is kill minus L, it will show all the symbol to numeric codes. Although you can just use numeric with pkill:


pkill -9 -U marcus



Reference

[tags]pkill, proc kill, Unix Coding School[/tags]



Apache Logging Filter Robots

Problem

Sick of filtering through loads of logs, or just spotting real hits from the robots! 🙂

Seriously reduce your apache web logs, by filtering out images, style sheets and your own hits.



Solution

Simple with Apache’s customlog and setenvif statements.

I’ve also included capturing the user-agent in a separate file, as well as the referer, which is brill for seeing which google searches brought traffic to you.

You can even still capture robot and own hits into a separate log, here is how below.



Example


        SetEnvIf Request_URI ".(png|gif|jpg|js|css)" image-req        SetEnvIf Request_URI "favicon.ico" image-req        SetEnvIf Request_URI "/icons" image-req        SetEnvIf Request_URI "sitemap.xml.gz" image-req        SetEnvIf REMOTE_ADDR "127.0.0.1" image-req        SetEnvIf REMOTE_ADDR "127.0.0.1" home-req        SetEnvIf User-agent "(Googlebot|msnbot|Spider|crawl|slurp|Jeeves|Mediapartners|FeedBurner)" image-req        SetEnvIf User-agent "(Googlebot|msnbot|Spider|crawl|slurp|Jeeves|Mediapartners|FeedBurner)" bot-req    CustomLog logs/access_log.techieblogs"["%{Referer}i"]n %h %l %u %t "%r" %>s %b" env=!image-req    CustomLog logs/access_log.agents.techieblogs"%h ["%{Referer}i"] ["%{User-agent}i"]" env=!image-req    CustomLog logs/access_log.bots.techieblogs"["%{Referer}i"] %h %l %u %t "%r" %>s %b" env=bot-req    CustomLog logs/access_log.home.techieblogs"["%{Referer}i"] n %h %l %u %t "%r" %>s %b" env=home-req



Reference

[tags]Apache Logging, Unix Coding School[/tags]



Job Control

Problem

You want to background a process, or leave it running whilst shutting the terminal.



Solution

To start a UNIX process is the background, simply append the command with an ampersand, like this below.



Example



sleep 30&;

If you have already started a process and want to start in the background, just hit control Z. Then type bg to background (and continue running) the command.

To display jobs in the background, runs command jobs

To bring back into the foreground, run fg

It is always a good idea to redirect output, for backgrounded jobs. Also if you are going to exit the shell, need to prefix the command with nohup. Or if it is already running, background the job and run disown



Reference

[tags]UNIX Batch Mode, UNIX Background Process, Unix Coding School[/tags]



Watch Multiple Files Simultaneously

Problem

Performing debugging and you need to view the output, of more than one file at once.



Solution

A simple way to keep an eye on a file is with tail.


tail -f logfile

You can background this, by appending an ampersand:


tail -f logfile&



Example


But if you want to watch multiple files, how do you know which one received the update.

Sed to the rescue:


tail -f logfilea | sed 's/^/logfilea: /'&
tail -f logfileb | sed 's/^/logfileb: /'&



Reference

[tags], Unix Coding School[/tags]



Vi Map Keys

Problem

Are there a combinations of key strokes you constantly enter?



Solution

Maybe you are a Java programming! 🙂 Or PHP/Perl – either way if you are using vi a lot, these tips could be worth your pc’s weight in gold. 😉



Example


To create a one time map (of any key), just run this within vi:


:map #1 :!echo hey mapped f1^M

You need to hold down control and hit m to get the return, so this will run itself (no enter required).

To see current maps, just type :map

To make it permanent, just add same syntax to $HOME/.exrc

I think another good tip, is to map f1 to cat the exrc – like this:


:map #1 :!cat $HOME/.exrc^M

Then map #2, 3, 4 …





Vi Swap Columns

Problem

You need to substitute columns, whilst editing a file with vi.



Solution

Okay, bit more complex with this one.

To substitute columns, we bracket off the pattern – like this (pattern)

Then we simply use an escaped number to substitute.



Example


Here is a demo, whilst editing a file with vi:


:%s/(.*) (.*)/2 1/

This says match all lines (%), substitute (s). Match anything up to a space, store in column one. Match anything else, store in column two. Now swop column two with column one.

Here is a run through – unchanged:


a z
b y
c x
d w
e v
f u
g t
h s
i r
j q
k p
l o
m n

The substitute:


:%s/(.*) (.*)/2 1/

The result:


z a
y b
x c
w d
v e
u f
t g
s h
r i
q j
p k
o l
n m



Reference

[tags]UNIX, vi, regular expression, Unix Coding School[/tags]



Vi Exit

Problem

Exiting a file in vi.



Solution

There are quite a few ways to exit a file in vi. Here are some of my favorites.



Example


The standard: :wq

Force, in the case of read-only: :wq!

Abort updates: :q!

Reset the file without writing: :e!

Quick save and exit: 😡

Lazy one, without using a colon: ZZ

Going to another file: n filename

Go back to the first file, when you'v edited with wildcard or list of files: :rew



Reference

[tags]vi, Unix Coding School[/tags]



Environmental Variables

Problem

You want to set something once and have it remembered between UNIX settings.



Solution

Set some variables. Simplest way to set environmental variables, is via the shell rc or profile scripts.



Example


For example if your shell is zsh, use .zprofile or .zshrc. For bash it is .bashrc or .profile.

To invoke env vars from cron, use something like this:


30 * * * * zsh -c "(source $HOME/.zprofile; cmd ... ")

To source your .zprofile before command invocation. Additionally you can use a dot to source a script containing your vars:


. scriptname



Reference

[tags]UNIX, variables, shell scripting, Unix Coding School[/tags]



Counting Column Contents

Problem

You want to count the contents of a column. Accumulatively totaling each number in a given column.



Solution

To total up the contents of a given column, use following awk code:



Example



nawk -v col=# ' { tot+=$col } END { print tot } '

In this example, do a du of the current directory and then total column one (1). Notice awk receives a variable on the column to sum up.


du -ks * | awk -v col=1 ' { tot+=$col } END { print tot } '
11168



Reference

[tags]UNIX, awk, nawk, gawk, Unix Coding School[/tags]



Regular Expression Matching NOT

Problem

You want to find a line, not matching the pattern.



Solution

I find this invaluable whilst editing crons. Or in vi – like this: [^x] where x equals the character you want to ignore.



Example


For example after performing a crontab -e, wanting to skip comments.


crontab -e
/^[^#]

Also with sed – substituting tags, etc:


sed 's#]*>##g' filename

This says edit your crontab, match the next line that has a beginning, immediately followed by a character that is not a comment.

After performing this once, just type n – for the next match (i.e. next line that does not start with a comment).



Reference

[tags]UNIX, vi, regular expression, Unix Coding School[/tags]



Finding Disk Space Hogs

Problem

Common admin task, to find what is using all the disk space.



Solution

Simpliest thing to do, in a known directory – for example /var/log – is to run du -ks *.

That will show all files and directories, in the current directory along with their disk space in kilobytes.

Then cd down the top ones and rerun. You can also pump the output through sort (-n on Solaris and +n on Linux).



Example



cd /var/log
du -ks * | sort -n # Linux
cd httpd
du -ks * | sort -n

With some flavours of UNIX, du shows actual physical disk space being used – where as df shows disk space being reserved. This is only really noticeable when you remove a file, that is still being written too. du shows the current directory is only using x kb and df still says you are at 100%. 🙂 You need to find and kill the process.

Another useful way of finding largest files, is find command – something like this:


find / -xdev -type f -a size +20000 -ls

This says find files over 10MB (20 thousand 512 byte blocks), do not traverse file systems mounted on this one (-xdev). You can also say -mount to stop find traversing these file systems. That way df and find tally – just re-run it for var, etc as required.

Again this can be pumped through to sort:


find / -xdev -type f -a -size +20000 -ls | sort -k7 -n # linux
find / -xdev -type f -a -size +20000 -ls | sort +6n # Solaris - starts field count from zero



Reference

[tags]UNIX, df, du, Solaris, Unix Coding School[/tags]



Display Directory Usage As Percentage

Problem

How many times have you been given the job, to clear down space in a UNIX file system?

Ever wanted to see the percentage, on a directory by directory basis.



Solution

Here is a simple bit of code, to display the current directory’s percentage of over all space available



Example



(du -ks . | tr 'n' ' '; (df -k . | tail -1)) | gawk ' { printf("%0.0f%n",($1/$4)*100); } '

Here is a run through, checking /usr:


[[email protected] usr]# (du -ks . | tr 'n' ' '; (df -k . | tail -1)) | gawk ' { printf("%0.0f%n",($1/$4)*100); } '
37%

[[email protected] usr]# cd java

[[email protected] java]# (du -ks . | tr 'n' ' '; (df -k . | tail -1)) | gawk ' { printf("%0.0f%n",($1/$4)*100); } '
4%

[[email protected] usr]# cd ../X11R6
[[email protected] X11R6]# (du -ks . | tr 'n' ' '; (df -k . | tail -1)) | gawk ' { printf("%0.0f%n",($1/$4)*100); } '
2%

[[email protected] X11R6]# cd ../lib
[[email protected] lib]# (du -ks . | tr 'n' ' '; (df -k . | tail -1)) | gawk ' { printf("%0.0f%n",($1/$4)*100); } '
15%

[[email protected] local]# cd ../share
[[email protected] share]# (du -ks . | tr 'n' ' '; (df -k . | tail -1)) | gawk ' { printf("%0.0f%n",($1/$4)*100); } '
13%



Reference

[tags]UNIX, Admin, Unix Coding School[/tags]



Duplicate Line Finder

Problem

Have you ever duplicated a cron entry, or just wanted to find duplicate lines in a file.



Solution

Here is a simple one liner, which shows all lines which occur more than once in a file or standard output.



Example



sort cronfile.223 | uniq -c | awk ' $1 != 1 '

So to check that cron directly – just do this:


crontab -l | uniq -c | awk ' $1 != 1 '



Reference

[tags]UNIX, awk, script, uniq, Unix Coding School[/tags]



Legato Networker Restore

Problem

You want to perform a restore from a legato networker backup server.
Also called NSR backup system



Solution

To restore from an NSR (networker) backup system, just run recover.

It is best if you cd into the required directory first.



Example



cd /var/www/httpd
recover

It will list your last back ups with dir. You can type help for more info.

Or just add, then when done – type recover.

dir shows if the tape containing your backup, is online or not.



Reference

[tags]NSR, Netwoker, Net backup, Legato, Unix Coding School[/tags]



Check NSR Legato Networker Backups

Problem

You want to view the back ups performed via Legato Networker NSR.



Solution

Use mminfo – media manager for NSR. It can show which filesystems were backed up,
when they were backed up, to which tapes and the retension period.



Example



mminfo -ot -s SERVER -c CLIENT -q 'savetime>=last week' -r 'volume,client,savetime(18),ssid,name,written,pool'


mminfo -ot -s SERVER -c CLIENT -q 'savetime>=last week' -r 'volume,client,savetime(18),ssid,name,written,pool,ssretent'



Reference

For more information on available options, do a man mminfo

[tags]NSR, Backups, Legato, mminfo, Unix Coding School[/tags]



RISC System 6000 Boot LEDs

Problem

Your RS6000 ain’t booting. 🙂 Or you are just interested what the LEDs mean.



Solution

See Example



Example


  1. Power On [LED 100 – 199]
  2. BIST [LED 100 – 199]
  3. POST [LED 201 – 298]
  4. Load Kernel [LED 299]
  5. Configuration [LED 500 – 999]
  6. INIT [LED 553]

init than loads /etc/inittab services – then /sbin/rc.boot (phase 3 of system boot),
/etc/rc (startup shell script), subsystems (cron, qdaemon, tcpip), getty (enable terminals)



Reference

[tags]RISC System bootup LEDs, AIX startup sequence, Unix Coding School[/tags]