Category : unix

Clearing an AIX Print Queue

Problem

You have jobs stuck in the print queue. You want to clear the queues and remove all jobs.



Solution

You have to stop all processes and remove the files from spooling directories.

Probably worth taking a backup of these files first. πŸ™‚ All at own risk.



Example


1.


stopsrc -cs qdaemon # bring down qdaemon and clear qdforks.
stopsrc -s lpd

2.


rm /etc/qconfig.bin # compiled /etc/qconfig

3.


rm /var/spool/lpd/qdir/* # queue requests; JDF

4.


rm /var/spool/lpd/stat/* # info on status of devices

5.


rm /var/spool/lpd/pio/msg* # error message files.

6.


rm /var/spool/qdaemon # temp copies of queue jobs

7.


rm /var/spool/lpd/* # temp copies of remote jobs

8.


ps -ef | grep pio # kill any of these processes; look for piobe, pioformat, pioout

9.


ps -ef | grep qd # kill any of these processes; look for qdforks.

10.


startsrc -s qdaemon # restarting qdaemon
startsrc -s lpd # restarting lpd

11.


lssrc -s qdaemon #look for active
ps -ef | grep qdaemon
lssrc -s lpd # look for active
ps -ef | grep lpd

12.


ps -ef | grep srcmstr # if there are 2 running, kill -9 one of them with higher PID.

13.

As a last resort – reboot



Reference

[tags], Unix Coding School[/tags]



DNS, Sendmail and Building Linux

Problem

Have been aware of DNS and sendmail for over 10 years, but never really dabbled.



Solution

Hit a lot of issues, mainly with dodgy clients – like outlook (ended up moving over completely to mozilla’s thunderbird email client!). Also with port forwarding and firewall, default routes, etc.

To set it all up these were my steps:

1) Build RH Linux server with fedora core 4 (5 is out now too I think).
2) Ensure BIND was installed. yum is your friend! πŸ™‚ Do yum –help as root.
3) Edit /etc/named.conf – add my zone file. Always make a bu: cp -i /etc/named.conf /etc/named.conf.$(date +%j)
4) Copied localdomain.zone to puterpet.zone # this domain is retired now. πŸ™‚
5) Modified puterpet.zone to include my info: SOA (start of authority) – in my case I said zion.puterpet.com as this then relates to the ip of broadband router. Bit backward in logic, specify your DNS server with a name registered in your DNS server! πŸ™‚ Then I created an A record and NS for zion; CNAME for www to zion and CNAME for mail; lastly an MX record for mail. I’ll happily supply my zone file if someone wants a peek. πŸ™‚
6) Register domain name with www.domaincentral.com.au – $12.50 a year! πŸ™‚ Give it the ip of my DNS server – which is actually the broadband router, with port 53 forwarded to my RH Linux box. domaincentral have been great (I’ve now got 3 domains with them). www.puterpet.com – this one www.techie-blogs.com and darlingranges.com (still setting this up).
7) Setup port forwarding on broadband router, then fire up named and it was all good. Setup up virtual hosts (www.puterpet.com and mail.puterpet.com) in that domain. Now can setup any mail account or host in that domain, it is just beautiful! πŸ™‚
8) Installed/updated sendmail. Especially like watching sendmail’s incoming and outgoing messages. πŸ™‚
9) Edit /etc/mail/sendmail.mc – which is a macro config file that gets fed into m4 (macro language). Had to update m4, etc as kept breaking during conversion.
10) Edited various other files under /etc/mail – such as relay_domains (allow email from puterpet.com, etc); aliases …
12) Installed dovecot – which services POP and SMTP requests.
13) Created some accounts to handle mail – sales; enquiries, etc.
14) Modified my apache confs to handle and redirectmatch to new domain.



Example




Reference

[tags]sendmail, DNS, POP, SMTP, Red Hat Linux, Unix Coding School[/tags]



AIX ODM removing device

Problem

Someone removes a tty, printer, etc by doing rm /dev/lp3.

Then you cannot recreate the device, as system returns “already defined at that location”.



Solution

You can try to recreate the device with mknod, but need the major/minor numbers.

Then do rmdev -l lp1 -d.

To get major, minor numbers you need to query the ODM.



Example



odmget -q "value3=lp1" CuDvDr

Should return something like:

CuDvDr:

resoure="devno"
value1="15"
value2="2"
value3="lp1"

The major number is 15 and minor 2.

Therefore mknod /dev/lp1 c 15 2 creates the device again.

Then rmdev -l /dev/lp1 -d should remove it cleanly.

In extreme cases, where this still does not remove it, you may have to manipulate
the ODM directly. * Proceed with caution *

Backup

cd /etc/objrepos
cp -i CuDv CuDv.$(date +%j)
cp -i CuAt CuAt.$(date +%j)
cp -i CuDvDr CuDvDr.$(date +%j)

Remove ODM entries

odmdelete -q "name=lp1" CuAt
odmdelete -q "name=lp1" CuDv
odmdelete -q "value=lp1" CuDvDr
synclvodm rootvg # resync odm

Reboot if necessary.



Reference

[tags]AIX ODM, odmdelete, synclvodm, Unix Coding School[/tags]



who command 5 ways

Problem

You want to know who is on the system. Or which userid you are using.
What runlevel you are at – or when the system was booted.

All doable with who.



Solution

See the example and reference tabs, to see 5 ways to run who, showing different things about the system and users.



Example


1. Basic


[[email protected] puterpet]$ who
marcus pts/1 Sep 19 16:07 (10.0.0.8)
marcus pts/3 Sep 22 17:46 (10.0.0.8)

2. With Heading and Idle time dot means currently active.


[[email protected] puterpet]$ who -Hu
NAME LINE TIME IDLE PID COMMENT
marcus pts/1 Sep 19 16:07 ? 29984 (10.0.0.8)
marcus pts/3 Sep 22 17:46 . 29566 (10.0.0.8)

3. Show who I am in the current shell and where I came from.


[[email protected] puterpet]$ who am i
marcus pts/3 Sep 22 17:46 (10.0.0.8)

4. Show the current run level Shows boot time and last runlevel too.


[[email protected] puterpet]$ who -r
run-level 5 Sep 9 10:47 last=S

5. Show last boot time


[[email protected] puterpet]$ who -b
system boot Sep 9 10:47



Reference

[tags]who command, Unix Coding School[/tags]



List files 12 ways

Problem

You want to list files under UNIX! πŸ™‚ Here I show 12 different ways to do it.



Solution

See the example and reference tabs.



Example


1. Basic


[[email protected] Perl_Bin]$ ls
d-dump-tst.pl demo-ldap-entry2.pl demo-ldap-entry5.pl ldap-to-sql.pm
logcam.pm showstat.pl
demo-ldap-entry1.pl demo-ldap-entry3.pl demo-ldap-entry6.pl listenport.pl perl_eval_pattern.pl showtime.pl
demo-ldap-entry1.pm demo-ldap-entry4.pl ldap-to-sql.pl logcam.pl
scanport.pl

2. Long


[[email protected] Perl_Bin]$ ls -l
total 68
-rwxr--r-- 1 marcus adm 335 Sep 12 2005 d-dump-tst.pl
...
-rwx------ 1 marcus adm 151 Feb 5 2006 showtime.pl

2. Hidden files


[[email protected] Perl_Bin]$ ls -la
total 76
drwxr-xr-x 2 marcus adm 4096 Aug 15 04:53 .
drwx------ 29 marcus adm 4096 Sep 23 11:47 ..
-rwxr--r-- 1 marcus adm 335 Sep 12 2005 d-dump-tst.pl
...
-rwx------ 1 marcus adm 151 Feb 5 2006 showtime.pl

3. Sort by last modified


[[email protected] Perl_Bin]$ ls -lt
total 76
-rwxr--r-- 1 marcus adm 286 Aug 15 04:53 perl_eval_pattern.pl
-rw-r--r-- 1 marcus adm 210 Apr 30 08:07 logcam.pm
-rwxr--r-- 1 marcus adm 3483 Feb 5 2006 logcam.pl
...
-rwxr--r-- 1 marcus adm 335 Sep 12 2005 d-dump-tst.pl

4. Sort by last modified – reverse


[[email protected] Perl_Bin]$ ls -ltr
total 76
-rwxr--r-- 1 marcus adm 335 Sep 12 2005 d-dump-tst.pl
...
-rwxr--r-- 1 marcus adm 3483 Feb 5 2006 logcam.pl
-rw-r--r-- 1 marcus adm 210 Apr 30 08:07 logcam.pm
-rwxr--r-- 1 marcus adm 286 Aug 15 04:53 perl_eval_pattern.pl

5. Show hidden characters in file name


[[email protected] Perl_Bin]$ ls -ld *test*
-rw-r--r-- 1 marcus adm 5 Sep 23 11:59 ???testfile1
[[email protected] Perl_Bin]$ ls -lb *test*
-rw-r--r-- 1 marcus adm 5 Sep 23 11:59 bbbtestfile1

6. Show a digest of directories With asterix for executable and slash for directory, etc.


[[email protected] Perl_Bin]$ ls -aF
./ demo-ldap-entry1.pl* demo-ldap-entry3.pl* demo-ldap-entry6.pl*
listenport.pl* perl_eval_pattern.pl* showtime.pl*
../ demo-ldap-entry1.pm demo-ldap-entry4.pl* ldap-to-sql.pl*
logcam.pl* scanport.pl*
d-dump-tst.pl* demo-ldap-entry2.pl* demo-ldap-entry5.pl* ldap-to-sql.pm
logcam.pm showstat.pl*

7. Do a recursive listing

[[email protected] Perl_Bin]$ ls -lR
.:
total 76
-rwxr--r-- 1 marcus adm 335 Sep 12 2005 d-dump-tst.pl
...
-rw-r--r-- 1 marcus adm 5 Sep 23 11:59 ???testfile1

./test1:
total 4
drwxr-xr-x 2 marcus adm 4096 Sep 23 12:01 test2

./test1/test2:
total 0
-rw-r--r-- 1 marcus adm 0 Sep 23 12:01 showit

8. List the directory not its contents.


[[email protected] marcus]$ ls -ld Perl_Bin
drwxr-xr-x 3 marcus adm 4096 Sep 23 12:01 Perl_Bin

9. Digest the listing


[[email protected] Perl_Bin]$ ls -m
d-dump-tst.pl, demo-ldap-entry1.pl, demo-ldap-entry1.pm, demo-ldap-entry2.pl, demo-ldap-entry3.pl, demo-ldap-entry4.pl, demo-ldap-entry5.pl,
demo-ldap-entry6.pl, ldap-to-sql.pl, ldap-to-sql.pm, listenport.pl, logcam.pl, logcam.pm, perl_eval_pattern.pl, scanport.pl, showstat.pl,
showtime.pl, test1, ???testfile1

10. Include the inode – useful for then doing find . -inum #### – to remove, etc


[[email protected] Perl_Bin]$ ls -li
total 76
647846 -rwxr--r-- 1 marcus adm 335 Sep 12 2005 d-dump-tst.pl
...
648063 -rw-r--r-- 1 marcus adm 5 Sep 23 11:59 ???testfile1

11. Force ls to produce one column


[[email protected] Perl_Bin]$ ls -1
d-dump-tst.pl
...

12. Show file modified time in seconds Okay I cheated here – it is Perl! πŸ™‚ But hey it's my blog!! πŸ˜‰


[[email protected] Perl_Bin]$ perl -e
'print(localtime((stat($ARGV[0]))[9])."n");' mytestfile
Sat Sep 23 12:50:03 2006



Reference

[tags]Listing files, Unix Coding School[/tags]



Permissions extension setfacl

Problem

You want to allow access to files or directories, without giving exclusive use of a given group, user, etc.



Solution

Use setfacl. πŸ™‚ See the example and reference tabs.



Example



setfacl -m user:USERID:rw- filename

getfacl filename

Can also do this: getfacl filename1 | setfacl -f – filename

Performing an ls -ld filename, will show a plus in the permission column.



Reference

[tags]setfacl, Unix Coding School[/tags]



Linux dircmp – directory compare

Problem

I wanted to run a dircmp with the -d option, to show all files that are different.

Plus display those differences.

Linux replied – command not found! πŸ™‚



Solution

Just diff the directories, with the -d flag. See the example and reference tabs for more info.



Example



diff -d dir1 dir2



Reference

[tags]UNIX, Linux, Dircmp, Unix Coding School[/tags]



Redundant key strokes

Problem

Carrying on with my lazy steps, there are bits of code used with UNIX commands, which are completely redundant.



Solution

Save yourself some key strokes, remember these examples. Or come back to unix.coding-school.com and check them out again. πŸ™‚



Example



chmod a+r filename

The "a" is redundant – if you just do a "+r" – all is implied.


ls -l | awk ' $4 > 1000 { print $0 } '

In awk printing $0 is default action.


find . -type f -print

With find default action is to print.


echo a b c | tr '[abcdef]' '[ABCDEF]'

Use a range: echo a b c | tr '[a-z]' '[A-Z]'

In vi use ZZ instead of :wq πŸ™‚

Only 2 chars less (including the shift) – but also just your little finger, instead of a hand span. πŸ™‚



Reference

[tags]Redundant key strokes, UNIX shortcuts, Unix Coding School[/tags]



Restarting Linux services boot time

Problem

Every time you reboot your Linux Box – web server, samba server, etc is down.



Solution

Use chkconfig. See the example tab and then reference tab for man page.



Example


Show all services


chkconfig --list

List just httpd


chkconfig --list httpd

Change restart settings so web server starts at run level 5.
(do a runlevel to see normal level – most likely 5).


chkconfig --levels 5 httpd on



Reference

[tags]chkconfig, Linux reboot, Linux services, Linux restarts, Unix Coding School[/tags]



Lazy tip – wrong password

Problem

You start typing your password in UNIX, then realize – hold on that ain’t right.

Sometimes your control c it – other times just hit enter and do it again.

There is a quicker way. πŸ™‚



Solution

When your sat at that password: prompt and lord only knows what you’ve typed. No problem – just do one control u and all is right with the world again. πŸ™‚

I only know from the billion odd times I’ve done it too! πŸ˜‰ That equates to many wasted hours, this lazy tip will save you – over time.



Example


Nothing to see here, check out the solution tab.



Reference

[tags]UNIX Lazy tip, UNIX passwords, Unix Coding School[/tags]



YUM fails with 404

Problem

If you (like me) was well keen, to get your shiny new fedora core 5 system up and running. Then you run yum and think, hang on that ain’t right – 404. Linux is better than this!! πŸ™



Solution

Don’t bother searching the news groups, for posts that go nowhere (like I did for close on 2 hours – that I’ll never get back!! πŸ™‚ ), here is the answer –

www.fedorafaq.org Do a search for yum – follow the prompts and you’re in.



Example


Nothing to see here, check out the solution tab.



Reference

[tags]Linux YUM, YUM fails, YUM problems, Unix Coding School[/tags]



Linux iptables blocking IP address

Problem

You do a netstat -an and see a raft of connections to your apache host, coming in at 10 – 20 a second.

It shows ::ffff:10.0.0.10:80 ::ffff:212.248.1.1:1403 in netstat output. These are IPV6 connections AFAIK.

Also you see ::1 – – [DD/MMM… in apache log. Restarting apache does nothing, the connections just reappear.



Solution

You need to block all connections from this IP address using iptables. See example tab.



Example



cp -i /etc/sysconfig/iptabes /etc/sysconfig/iptables.$(date +%j).bu
vi /etc/sysconfig/iptables

Add a line like the one below:

#block
-A INPUT -s 212.248.225.12 -j DROP

Then restart iptables like this: /etc/init.d/iptables restart as root.

Finally check your rules like this: /etc/init.d/iptables status as root.



Reference

[tags]Linux Security, Linux iptables, Fedora Core iptables, Linux Firewall, Unix Coding School[/tags]



UNIX Lesson Six

Problem

You want to write elegant code, which integrates well (pipes) with other UNIX commands.



Solution

Learn these UNIX Laws, from the inventor of UNIX pipes.



Example


No examples – see the reference tab.



Reference

1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new features.

2. Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.

3. Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.

4. Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them.

Doug McIlroy, the inventor of Unix pipes and one of the founders of the Unix tradition.

[tags]UNIX Law, UNIX pipes, Unix Coding School[/tags]



UNIX Lesson Five

Problem

Coding in UNIX is an art form, where certain philosophies prevail.



Solution

To really excel at UNIX coding and produce elegant, efficient, low maintenance code – you need to learn UNIX philosophy.



Example


No examples – just see the reference tab.



Reference

1. Rule of Modularity: Write simple parts connected by clean interfaces.

2. Rule of Clarity: Clarity is better than cleverness.

3. Rule of Composition: Design programs to be connected to other programs.

4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines.

5. Rule of Simplicity: Design for simplicity; add complexity only where you must.

6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.

7. Rule of Transparency: Design for visibility to make inspection and debugging.easier.

8. Rule of Robustness: Robustness is the child of transparency and simplicity.

9. Rule of Representation: Fold knowledge into data so program logic can be stupid and robust.

10. Rule of Least Surprise: In interface design, always do the least surprising thing.

11. Rule of Silence: When a program has nothing surprising to say, it should say nothing.

12. Rule of Repair: When you must fail, fail noisily and as soon as possible.

13. Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.

14. Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.

15. Rule of Optimization: Prototype before polishing. Get it working before you optimize it.

16. Rule of Diversity: Distrust all claims for “one true way”.

17. Rule of Extensibility: Design for the future, because it will be here sooner than you think.

[tags]UNIX Laws, UNIX Philosophy, Unix Coding School[/tags]



UNIX Lesson One

Problem

You are a complete beginner and keen to learn UNIX!



Solution

Okay in the examples, I am going to start very basically.

From the outset, you need to understand some basic terms.

  • Firstly – what is UNIX. Unix is an operating system. Other operating systems including Windows, Apple Mac, Novell, etc. It is what turns the computer, from an over-sized paper weight – into a living, breathing machine. πŸ™‚

  • The operating system, talks to all the hardware. Definition of hardware is that if you drop it on your foot, it hurts. πŸ™‚ It talks to the CPU (brain chip) of the computer, the memory (RAM Chips) which is sort of like your desk space, a working area. The bigger your desk space the more you can work on at once. Why size does matter – how big is big. Well currently 256mb (256 megabyte) should be the minimum, ideally 1GB (1 gigabyte) should be plenty.

  • So how do we interface with UNIX? Through a terminal and shell generally. Yes you can use GUIs (Window) – but the real power of UNIX is in the shell.

  • What is a terminal and a shell for that matter? A terminal is just a program that runs, that allows you to connect to UNIX machines – including your own. If you have just installed your own Linux machine, or having to use one at work – you’ll generally start from a windows environment – not necessarily Microsofts either! πŸ™‚ If your are running Linux, generally you can right click your mouse on the background of your desktop and select terminal. From MS Windows there is a terminal program you can run, or ideally look up putty SSH client at the reference tab.

  • more to follow shortly …


Example




Reference

[tags]Free Linux Lessons, Lesson One, Unix Coding School[/tags]



Compare command output with file

Problem

You want to run a command, save the output to a file – then compare the output at a later date with the file.

For example – you want to update your crontab: save a backup crontab -l > cronfile.$(date +%j), make your updates, then compare it with the backup file.



Solution

Use diff with the hypen to represent read from stdin.

This saves keeping multiple copies.



Example



crontab -l > cronfile.$(date +%j)
crontab -e # make your diffs
crontab -l | diff - cronfile.$(date +%j)

Also you can use cmp, which just reports the line number, etc


crontab -l > cronfile.$(date +%j)
crontab -e # make your diffs
crontab -l | cmp - cronfile.$(date +%j)



Reference

[tags]crontab, diff, cmp, comparing command output with file, UNIX Coding School[/tags]



Simple Basic UNIX Permission Denied

Problem

You are quite happily editing away, then you come to save … oh dear, permission denied.

Either the file is set to readable, or it is owned by another user.

But you have write permission to the directory, either by user or group!



Solution

If it is just a simple case of the file being unwritable and you own it, if in vi following the first example.

If you are not the owner, write the file to /tmp. Then follow the second example.



Example


Allow write permission to a file you are currently vi editing – but the file is not writable.

:!chmod u+w %

If you are not the owner, no need to worry, even without root we can change the owner if you have write access to the directory! πŸ™‚


:!zsh
mv -i your_filename /tmp/your_filename.mv
cp -i /tmp/your_filename.mv your_filename

Obviously if not in vi, you'll need to break out to a shell using the mechanism in your editor. Also you may not have zsh, so ksh – bash, etc will do.

I tend to do all this on the command line, with a semi-colon between the mv and cp – so I can use command line expansion (the move has not happened yet!).

mv -i thefile[tab] /tmp/thefileXXXX;cp -i /tmp/thefileXXXX . [return]





Network sniffing on Linux

Problem

Just what the hell is your apache web server doing!! πŸ™‚

Seeing how clients are connecting to servers and what data is passing between, can prove invaluable in finding the solutions to issues.



Solution

Network trace it – showing the headers and content being sent and received.



Example



tcpdump -vvv -A -XX host 10.0.0.8 and port 80

Just replace 10.0.0.8 with your client ip address.

tcpdump -c 50 port 80
tcpdump: listening on eth0
00:43:09.262942 10.0.0.8.1288 > bree.http: S 2445348839:2445348839(0) win 16384
00:43:09.263069 bree.http > 10.0.0.8.1288: S 3557400364:3557400364(0) ack 2445348840 win 5840 (DF)
00:43:09.265882 10.0.0.8.1288 > bree.http: . ack 1 win 17520
00:43:09.268591 10.0.0.8.1288 > bree.http: P 1:493(492) ack 1 win 17520

On AIX you can ordinarly use iptrace or tcpdump.

On Solaris, we simply use snoop.
snoop -x0 src hostname

Where source is the client and hostname is the server. Other options are port portnum, etc. snoop –help or man snoop for more options.



Reference

[tags]Network, Tracing, tcpdump, Linux[/tags]



Removing links to missing files

Problem

You have many symbolic links that point to missing files.

I guess it is all just about good housekeeping! πŸ™‚ Well and the fact your application might break, depending on something being there, that is not.



Solution

This bit of shell under example tab, deals with them swiftly and sharply. πŸ™‚

Effectively it just does a find from the current directory, then loops around looking for missing files.



Example



for na in `find . -type l`
do
ls -lLd $na 2>&1 | grep "No such file" |
while read junk delme junk
do
rm ${delme%:}
done
done



Reference

[tags]symbolic links, find, rm, while, UNIX Coding School[/tags]



compressed using zip gzip bzip2

Problem

You have been sent (or want to download) a zip file. But also want to verify the contents, before decompressing.



Solution

Under UNIX there are a number of ways to compress files, or deal with those that have already been compressed.

See the examples tab.



Example


  • Firstly dealing with files ending with .gz.

    To view the contents of a gzipped file, just use gzip -l filename.gz

    To unzip the file, use gzip -d filename.gz or gunzip filename.gz

    To view the contents of the gzipped file, without decompressing it: gzip -c -d filename.gz.

    To create a gzip – gzip –best filename – or gzip reads from stdin, in the best of UNIX traditions. πŸ™‚ You can also pipe gzip output or redirect it.

  • Next unzip.

    To view the contents of a zipped file, just use zipinfo filename.zip

    To unzip the file, use unzip filename.zip.

    To view the contents of the gzipped file, without decompressing it: unzip -c filename.gz.

    To create a zip – zip filename – or zip reads from stdin.

  • Lastly bzip.

    To decompress the file, use bunzip2 filename.bz2.

    To view the contents of the bzipped file, without decompressing it: bzcat filename.bz2.

    To create a zip – bzip2 filename – or zip reads from stdin.



Reference

[tags]gzip, zip, bzip2, UNIX Coding School[/tags]



Strip XML comments

Problem

You want to strip XML comments out of your XML file.

This should also work with HTML comments.

Very handy for spotting errors, especially in configuration files! πŸ™‚



Solution

In this example, I use pure shell commands to identify and remove comments.

Obviously it assumes comments are not embedded, within valid XML, but on their own line.

I’ve done some brief testing, appears to work fine so far.

Have n’t tried nested comments either. πŸ™‚



Example



#!/bin/bash

[ $# -ne 1 ] && { echo "Usage: $0 file.xml"; exit 1; }

file=$1

[ ! -f $file ] && {
echo "$0: $file is not a file, exiting ..."; exit 1
}

flag=0

cat $file | while read line
do
[[ $(echo "$line" | grep -c -- '<!--') -eq 1 ]]
&& { flag=1 }

[[ $(echo "$line" | grep -c -- '-->') -eq 1 ]]
&& { flag=2 }

[ $flag -eq 0 ] && { echo "$line" } ||
[ $flag -eq 2 ] && { flag=0 }

done

exit 0

Drop me a comment, with your experiences of it.



Reference

[tags]Strip XML Comments, UNIX Coding School[/tags]



catch unix process – ps

Problem

You want to catch a process, but they just shoot by too quickly.

Every time you run a ps, you cannot spot that specific command.

You need a capture script.



Solution

Run the code at the example and it will catch changing and new processes. To be more aggressive, just change “sleep 5” to “sleep 1”.



Example


user=”marcus”

ps –fu $user > catchdiff

while :

do

ps -fu $user | diff - catchdiff | egrep -v 'catchdiff|-bash$|^---|^[0-9]'

sleep 5

done



Reference

[tags]ps, diff, egrep, while, process[/tags]



Remove content from gzips

Problem

You need to remove content from zip files.

Effectively unzip, remove the pattern and re-zip.



Solution

A collection of shell commands, to perform the unzip, swap and rezip.



Example


* First to find zips with those patterns in (substitute pattA and B with your patterns, on each of the commands below):


for na in *gz
do
[[ $(gzip -c -d $na | egrep -ic 'pattA|pattB') -ne 0 ]] && {
echo $na
}
done > /tmp/filelist.out

* Then to test removing the offending lines:


for na in $(</tmp/filelist.out)
do
gzip -c -d $na | egrep -iv 'pattA|pattB' |
gzip --best > /tmp/replace.gz
echo going to mv /tmp/replace.gz $na
ls -ld /tmp/replace.gz $na

done

* Now to the actual move:


for na in $(</tmp/filelist.out)
do
gzip -c -d $na | egrep -iv 'pattA|pattB' |
gzip --best > /tmp/replace.gz
mv /tmp/replace.gz $na

done

* Then a retest:


for na in $(</tmp/filelist.out)
do
gzip -c -d $na | egrep -i 'pattA|pattB' |
sed "s/^/$na: /"

done



Reference

[tags]gzip, sed, Unix Coding School[/tags]



Cut and paste a zip file? You bet!

Problem

You have a shell open on two UNIX boxes, but scp, ftp, etc, etc is locked down?

Well they left Perl!!! πŸ™‚



Solution

I recognise this could be messy for massive files – but it works beautifully on binaries, images, etc.

I am demo’ing it here on tar gzipped output. It produces some errors, but still recreates the files.



Example


  • First to prove the files with checksums (11171, 28913)

  • $ find test -type f -exec sum {} ; -ls
    11171 18
    294632 20 -rwxr--r-- 1 marcus marcus 17461 Feb 15 21:49 test/testA/mun-weir.jpg
    28913 19
    294633 20 -rw-r--r-- 1 marcus marcus 18505 Feb 15 21:49 test/testB/perth-swan-river.jpg

  • Now to generate the text for copying – using base64 encoding to convert binary to text

  • tar zcvf - test | perl -MMIME::Base64 -e 'while() {
    $str.=(encode_base64($_)); } print "$strn";'

    This produces a listing of the files/dirs and then starting from H4, the actual text version of the binary output. I have truncated this – as it gets quite big. You can redirect this into a file, if using cygwin – then just open it with wordpad.

    test/
    test/testA/
    test/testA/mun-weir.jpg
    test/testB/
    test/testB/perth-swan-river.jpg
    H4sIAGVX1EUAA+z8BWxlwfMuBl6PmZnZvmYaMzMzMzMzMzMz22P7mpnZ4zHD2B4zM+OYGXZ+/5dk
    k0hRFCl5u6t9daVzjlTd1V931anuOlf63Mxd3ZgA/8/K93/Cyc7+nzsrJwvz//r+PwuA+Ts7B/t3
    dlZWFk7Ad2ZmNmZmAAn7/8O4/kvcXd2MXUhIAPbGLqburv/H7f7P9P8/Km7/8f9/LsL/z0XB/yX/
    s7P+8z8LG+f3/+H//x7yv/K/vbsDg6e5tQujjZPl/61j/Jf/2dj+D/zPxvKdg/1/5382TjZ2AMn3
    /1tR/B/I/5/7/2vlaweAIiMhLQEAAwMDSP37Ab52AVjiXtYWAIC8PID2XyOY/6ntOkAUAAkOAQEB
    DvnvAgkJCQWNAAP9T5Dh4WERUJHR0FCRUVHRsQiw0THwMFFRcUhx8AgJiYmJ0bHJgGREQAIiYqL/
    GAH71xUaCg==
    GgkGBokIAxWD6P+yfPUDUGEAYWD34GBkgG+oYOCoYF9DAMJ/GKH/M4v/t3wDh/gHEgwG9p9aHgXw

  • Now to extract it
  • First we paste it into tar.uu – which could be anything or you can just do cat | perl and paste to stdin.


    [[email protected] ~]$ cat tar.uu | perl -MMIME::Base64 -e 'while() { $str.=(decode_base64($_)); } print "$strn";' | tar zxvf -

    gzip: stdin: decompression OK, trailing garbage ignored
    test/
    test/testA/
    test/testA/mun-weir.jpg
    test/testB/
    test/testB/perth-swan-river.jpg
    tar: Child returned status 2
    tar: Error exit delayed from previous errors

    As you can see produces a bit of garbage.

  • Prove the files are reproduced – with sums (11171, 28913)

  • [[email protected] ~]$ find test -type f -exec sum {} ; -ls
    28913 19
    1555 24 -rw-r--r-- 1 marcus adm 18505 Feb 15 21:49 test/testB/perth-swan-river.jpg
    11171 18
    1553 24 -rwxr--r-- 1 marcus adm 17461 Feb 15 21:49 test/testA/mun-weir.jpg

    And you are done. πŸ™‚



    Reference

    [tags]cut and paste binary files, base64, Unix Coding School[/tags]



    Optical dump file – od

    Problem

    Whenever you cat a file, it shows nothing!

    Or it destroys your terminal. πŸ™‚

    … or when you diff to files – the look identical – but it reports them as not.



    Solution

    You probably have control characters in the file, or no newline at the end of the file.

    The example shows how to run an optical dump on a file.



    Example


    # optical dump of a file, useful for seeing control chars, etc and lines with no newline.

    od -c filename



    Reference

    [tags]Optical dump, od, Unix Coding School[/tags]



    Find secrets – undocumented feature

    Problem

    You want to find a pattern in a files, but have the file name and pattern printed.

    If you run find . -type f -exec grep -i “pattern” {} ; it will only show the pattern turning up – i.e not the file name as well.

    If you run find . -type f -exec grep -il “pattern” {} ; it will only show the file name turning up – i.e not the line containing the pattern as well! πŸ™‚



    Solution

    Maybe you have seen this solution before, but AFAIK it is completely undocumented.

    I discovered it whilst trying to do this very thing and I just thought, hmmm … I wonder. πŸ™‚



    Example


    To force grep to print the line containing the pattern and the file name, we pass /dev/null as another argument. Find sees this as multiple arguments, so prints the file name and line containing pattern.

    • The normal basic find

    • $ find . -type f -exec grep -i author {} ;

      <ADDRESS CLASS="doc-author">The OpenLDAP Project <<A target=_blank HREF="http://www.openldap.org/">http://www.openldap.org/</A>></ADDRESS>
      <META name="Author" content="TechRock">
      <META HTTP-EQUIV="AUTHOR" content="TechRock">

    • find looking for just the file name

    • $ find . -type f -exec grep -il author {} ;

      ./cs/ldap-head.inc
      ./meta.inc

    • Undocumented feature, print file name and line content

    • $ find . -type f -exec grep -i author {} /dev/null ;

      ./cs/ldap-head.inc:<ADDRESS CLASS="doc-author">The OpenLDAP Project <<A target=_blank HREF="http://www.openldap.org/">http://www.openldap.org/</A>></ADDRESS>
      ./meta.inc:<META name="Author" content="TechRock">
      ./meta.inc:<META HTTP-EQUIV="AUTHOR" content="TechRock">



    Reference

    [tags]unix find, find undocumented features, find secrets, Unix Coding School[/tags]



    Synchronize directories between firewalled hosts

    Problem

    You have an admin host (ADMIN), which can see two servers (hostA and hostB) behind firewalls.

    You are not allowed to scp/rdist, etc from hostA or B to ADMIN. Nor are you allowed to scp/rdist, etc between hostA and B.

    ADMIN can scp, etc though to both hosts. Now in this real life example, updates are made directly to hostA – but not hostB. So they need to be synchronized somehow, looking at the real files and performing a pull to ADMIN from hostA and then a push to hostB.

    But then we only want new files …



    Solution

    In this script at the example, I pull from hostA and push to hostB. But only “look” for files that have updated, since last time I looked. πŸ™‚



    Example


    It starts off just setting variables, creating temporary directories on ADMIN host.

    Then at line 26 I obtain a list of files from hostA, that have been updated since last time I ran this script.

    At line 29 I spin around this list and copy the given file to the temp directory (line 34). Notice I use ${file##*/} this strips off the path – so I just get the file (i.e. bob.gif not /dirABCD/…./bob.gif).

    I then copy this file (line 36) back out to the full path on hostB. Build a test and check the file exists on hostB – lines 38 – 42. Then remove the temporary copy of the file (line 44).

    Spin back round for the next file at line 45. Lastly cleaning up.

    Whilst this is not as efficient as rdist (opening one ssh connection), it will only copy updated files. So not a tar of the whole directory, etc. Plus by pepper potting between hostA and hostB – there is a few seconds delay between each file update – so should n't overload either host.


    1:#!/bin/zsh
    2:
    3:tempdir="/tmp/sync.$$"
    4:filelist="files.lst"
    5:basedir="/doc_root"
    6:touchfile="/tmp/.sync"
    7:dtg=$(perl -M'English' -e 'print $BASETIME."n";')
    8:report="/tmp/sync.$dtg"
    9:
    10:
    11:[ -d $tempdir ] && {
    12: echo "$0: $tempdir already exists - exiting ..."
    13: exit 1
    14:}
    15:
    16:mkdir $tempdir || {
    17: echo "$0: mkdir $tempdir failed - exiting ..."
    18: exit 1
    19:}
    20:
    21:cd $tempdir || {
    22: echo "$0: $tempdir not accessible - exiting ..."
    23: exit 1
    24:}
    25:
    26:/bin/ssh hostA /bin/find $basedir -type f -a
    -newer $touchfile > $filelist 2> /dev/null
    27:#/bin/ssh hostA /bin/find $basedir -type f -a
    -mtime -1 > $filelist 2> /dev/null
    28:
    29:for file in $(<$filelist)
    30:do
    31: echo "copying $file"
    32: echo "copying $file" >> $report
    33:
    34: /bin/scp hostA:$file . 2> /dev/null
    35:
    36: /bin/scp ${file##*/} hostB:$file 2> /dev/null
    37:
    38: check=$(/bin/ssh hostB /bin/ls -ld "${file}" 2>&1)
    39:
    40: [[ $(echo $check | grep -c "No such") -gt 0 ]] && {
    41: echo "$0:$dtg:$file failed" >> $report
    42: }
    43:
    44: /bin/rm ${file##*/}
    45:done
    46:
    47:/bin/ssh hostA /bin/touch $touchfile 2> /dev/null
    48:
    49:/bin/rm $filelist
    50:
    51:cd /tmp
    52:
    53:rmdir $tempdir
    54:
    55:exit 0

    See reference to download this code.





    UNIX chat shell code

    Problem

    As most UNIX bods know, with fairly recent audit lock downs – talkd has been disabled.

    But you want to chat remotely with someone, i.e. another techie bod! πŸ™‚



    Solution

    Now where is the fun in that! πŸ™‚

    Don’t know about you, but quite often I’ve needed to talk to other people remotely and don’t want to hold a phone or use hands free.

    See the example.



    Example


    Here is a quick and dirty solution, that works with bash – should work with zsh too.

    I welcome any enhancements or suggestions, via the comments.


    #!/bin/bash

    trap 'kill $tailpid;echo "*$nick leaves chat*" >> ${chatdir}/discussion;
    rm ${chatdir}/.${nick};exit 0' 2

    chatdir="/var/tmp/unixchat"

    [ -d $chatdir ] || {
    mkdir $chatdir || (echo "failed to mkdir $chatdir";exit 1)
    }

    chmod 777 /var/tmp/unixchat

    while [ $(echo X${nick} | grep -c "^X$") -eq 1 ]
    do
    echo -n "Nick: "
    read nick

    [ -f "${chatdir}/.${nick}" ] && {
    echo "$nick already exists, try another";nick=""
    }

    done

    touch ${chatdir}/.${nick}

    [ ! -f ${chatdir}/discussion ] && {
    touch ${chatdir}/discussion
    chmod 666 ${chatdir}/discussion
    }

    echo "*$nick joins chat*" >> ${chatdir}/discussion

    tail -1000f ${chatdir}/discussion&
    tailpid=$!

    echo ">> type quit to exit <<"

    while [ $(echo X${entry} | grep -c "^Xquit$") -ne 1 ]
    do
    read entry
    echo "[${nick}@`date '+%d-%b %H:%M:%S'`] says: $entry" >> ${chatdir}/discussion&

    done

    kill $tailpid; rm ${chatdir}/.${nick}

    echo "*$nick leaves chat*" >> ${chatdir}/discussion

    exit 0

    Here is a screen shot:

    UNIX chat code - simple solution written in shell





    extract specific line number from file using sed

    Problem

    You want to extract a specific line number from a file.

    Or a range of numbers from a file.



    Solution

    Once you have your line number – see grep, we can extract lines around the pattern.

    To do this sed (stream editor) can be used to print just desired lines – see the example which says don’t print all lines (-n); start at line 456 and finish at line 466 – print:



    Example


    This says don't print all lines (-n); start at line 456 and finish at line 466 – print:


    sed -n 456,466p filename

    Also with sed, we can say delete specific lines – in this case remove lines 5 to 10:


    sed 5,10d filename

    That's not all sed can accept patterns, as start/end identifiers:


    sed /start_pattern/,/end_pattern/d filename



    Reference

    [tags]UNIX sed, UNIX, sed, UNIX Coding School[/tags]



    Debugging Sendmail

    Problem

    Issues with sendmail, tacking on another domain name, etc.

    You want to manually test you mail host, from the command line.



    Solution

    Whilst debugging sendmail issues, came across this great test.

    You can run this from the command line. See example.



    Example



    telnet mailrelay.domain 25

    .... answer from mail relay .....

    helo unix hostname
    mail from: [email protected] hostname
    rcpt to: email [email protected] out
    data

    mail test from unix
    .



    Reference

    [tags]sendmail debugging, debugging sendmail, sendmail, debugging, UNIX Coding School[/tags]