sqlite 10 sec guide

[problem]

You want to get up and running with sqlite

[/problem]

[solution]

Install sqlite3 on linux, via yum: yum install sqlite3

Then just run sqlite dbname

[/solution]

[example]


% sqlite3 testdata.db
SQLite version 3.5.9
Enter ".help" for instructions
sqlite> create table memos(text,priority INTEGER);
sqlite> insert into memos values('deliver project description',10);
sqlite> insert into memos values('lunch with Christine',100);
sqlite> select * from memos;
deliver project description|10
lunch with Christine|100
sqlite>

[/example]

[reference]

  1. sqlite3 linux man page
  2. sqlite official site

[/reference]

starting binary – library file not found

[problem]

Getting an error about not being able to find libraries (file not found), when starting up a binary – such as httpd.

[/problem]

[solution]

You need to identify the shared libraries required by your binary. Ordinarly we use the ldd command for this and amended LD_LIBRARY_PATH to find the libs – see example below.

You might also want to get the shared libs of a running process – with pldd or need to dig into a lib/binary.

[/solution]

[example]

# find dependencies for a binary


% ldd /usr/bin/httpd
libpng.so.2 => /usr/local/lib/libpng.so.2
libcrypt_i.so.1 => /usr/lib/libcrypt_i.so.1
libresolv.so.2 => /usr/lib/libresolv.so.2

# demo of how ldd walks the values in LD_LIBRARY_PATH


% unset LD_LIBRARY_PATH

% ldd /usr/bin/httpd
libpng.so.2 => (file not found)
libcrypt_i.so.1 => /usr/lib/libcrypt_i.so.1
libresolv.so.2 => /usr/lib/libresolv.so.2

# find files installed via a package on Solaris


% grep libpng.so.2 /var/sadm/install/contents | head -1
/usr/sf/lib/libpng.so=libpng.so.2 s none SUNWpng

Another way to see which libraries are being picked up, i.e. if a binary works on one system and not another:


% truss -t'open' -f date
21667: open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT
21667: open("/usr/lib/libc.so.1", O_RDONLY) = 3
21667: open("/usr/lib/libdl.so.1", O_RDONLY) = 3
...

Getting the dependent shared libraries of a running process (especially where vars might have changed since startup):


% pldd 24832
24832: /usr/bin/httpd
/usr/local/lib/libpng.so.2.1.0.5
/usr/lib/libcrypt_i.so.1
/usr/lib/libresolv.so.2
...

Seeing more detail via the headers within an object file (use objdump on linux):


% elfdump /usr/bin/httpd | grep NEEDED
[0] NEEDED 0x15755 libpng.so.2
[1] NEEDED 0x15761 libcrypt_i.so.1
[2] NEEDED 0x15771 libresolv.so.2
...

See configure parameters passed to a binary during compilation (can also be useful for getting a usage out a binary, that does not appear by default or with –h, etc):

[/example]

[reference]

  1. Linux ldd man pages
  2. Linux objdump man pages
  3. Reverse Engineering Linux Binaries

[/reference]

[tags][/tags]

Stripping out newlines from string

[problem]

Different versions of UNIX handle the stripping of newlines in a string differently. Obviously if you echo a string, the shell automatically expands newlines “n”.

[/problem]

[solution]

sed and tr don’t appear to like performing these substitutions. But with Perl its a steal. 🙂

[/solution]

[example]


% echo 'hello worldnhello worldntest it' | perl -ane 's#\n# #g; print,"";'

hello world hello world test it

[/example]

[reference]

[tags], Unix Coding School[/tags]

[/reference]

Resize image command line – mogrify

[problem]

You want to resize images from the command line.

[/problem]

[solution]

Use image magik mogrify command.


mogrify -resize "percentage" image_na

Also you can view current sizing – like this:


% identify IMG_3864.JPG
IMG_3864.JPG JPEG 563x422 563x422+0+0 DirectClass 8-bit 32.8887kb

[/solution]

[example]


mogrify -resize 20% IMG_3705.JPG

[/example]

[reference]

[tags]mogrify[/tags]

[/reference]

UNIX Pipes – event triggers action

[problem]

You want to listen on a pipe and perform commands, based on the text sent to the pipe.

This could be useful for triggering action based on events;
Running code as different users – i.e. allow root or a functional user to run something – just need to allow access to group of users, via permission to write to the pipe.

[/problem]

[solution]


1. mknod /tmp/.restart_crd

2. chmod a+w /tmp/.restart_crd

3. nohup tail -f /tmp/.restart_crd| while read line
do
case "$line" in
'restart') cmd="./stop >> /tmp/restart.log 2>&1; sleep 5; ./start >> /tmp/restart.log 2>&1";;
'stop') cmd="./stop >> /tmp/restart.log 2>&1";;
'start') cmd="./start >> /tmp/restart.log 2>&1";;
*) cmd="no match ${line}.";;
esac

echo "command is $cmd"
eval $cmd

sleep 5

done &

4. disown

[/solution]

[example]


$ echo "restart" > /tmp/.restart_crd

$ echo "start" > /tmp/.restart_crd

$ echo "stop" > /tmp/.restart_crd

Watch output in by tail -f /tmp/restart.log

[/example]

[reference]

[tags], Unix Coding School[/tags]

[/reference]

receive notification when file arrives

[problem]

You want to get notified when a file arrives. Maybe it is coming in from somewhere else, or just is the result of some processing.

What ever the case, never sit there waiting for a file to arrive again! 🙂

[/problem]

[solution]

Put the code under example tab, in a cron entry. See reference for more detail about cron.

Change the minutes, hour, filename and mail address! 🙂 You may need to use mail instead of mailx and ksh, etc instead of bash. But the basic syntax is the same. 🙂

[/solution]

[example]


0-59 12 * * * bash -c "([ -f $HOME/test-touch ] &&
mailx -s "test-touch found at `date`" [email protected]

[/example]

[reference]

[tags]cron, UNIX Batch, UNIX Scripts, Unix Coding School[/tags]

[/reference]

UNIX debugging

[problem]

  1. You want to debug a running process on Solaris?
  2. You want to debug running process on Linux?

[/problem]

[solution]

See the example for debugging processes on Solaris or Linux – the syntax shows all system executes and reads.

[/solution]

[example]

  1. truss -xall -vall -rall -t'read' -p PID #

    To run a command and have it produce debug info, run it prefix with the truss syntax:

    truss -xall -vall -rall -t'read' your command #

    Another good option when debugging, is to run truss and look at the opens. Or more specifically the failures – as this often highlights missing libraries or permissions issues.

    truss -t'open' -p PID #

  2. On Linux use strace, which takes basically the same options. You just use -e, like this:

    strace -e'open' ls
    open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
    open("/etc/ld.so.cache", O_RDONLY) = 3
    open("/lib/tls/librt.so.1", O_RDONLY) = 3
    open("/lib/libtermcap.so.2", O_RDONLY) = 3
    open("/lib/libacl.so.1", O_RDONLY) = 3
    open("/lib/tls/libc.so.6", O_RDONLY) = 3
    open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
    open("/lib/libattr.so.1", O_RDONLY) = 3
    open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
    open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
    open("/etc/mtab", O_RDONLY) = 3
    open("/proc/meminfo", O_RDONLY) = 3

[/example]

[reference]

[tags]UNIX Debugging, Solaris Debugging, Linux Debugging, truss, strace, Unix Coding School[/tags]

[/reference]

Shell Tips and Tricks

[problem]

  1. You want to trace program execution on Solaris?
  2. You want to perform network tracing on Solaris?
  3. Perform a text dump of a binary file – or see ascii codes, etc?
  4. Generate some random data?
  5. List directory contents in 3 columns?
  6. Loop through a list of files and perform actions on them?
  7. Check for values in vars?
  8. Detach a process without it dying.
  9. Perform arithmetic in shell.

[/problem]

[solution]

Some of the most useful Shell Tricks I’ve learned – check them out at Example – enjoy! 🙂

[/solution]

[example]

  1. System process debug (solaris) all exec/reads

    truss -xall -vall -rall -t'read' -p PID

  2. Network trace connection from hostname (solaris)

    snoop -x0 src hostname

  3. Optical dump of a file, useful for seeing control chars, etc and lines with no newline.

    od -c filename

  4. Generate 512 bytes of random data, cat -ve escapes control chars.

    dd bs=1 count=512 if=/dev/urandom | cat -ve

  5. list directory contents, display in 3 columns.

    ls | paste - - -

  6. Loop through all files and perform an action on it.

    for na in * ; do echo $na ; done

  7. Check for values in vars.

    [[ $(echo $var | grep -ic "pattern") -eq 0 ]] &

    You could also go for “-eq 1” – does contain or -ne 0 for same, etc. Plus you could use || for “or” either in place of “&&” or after this block – same as if .. then “positive” else “negative” fi.

  8. Detach a process without it dying.

    disown [%1]

    nohup command > /tmp/logfile.log 2>&1&

    If you start a process with nohop (ignore the hup signal), it will not exist (usually) when your shell is exited. This can now be simulated on solaris, by backgrounding a process (with ctrl Z and bg) then typing disown. jobs command will show all backgrounded jobs. Ideally start commands like this, to avoid the shell exit problem:

  9. Perform arithmetic in shell.

    bc -l

    Binary Calculator in long format – for float, etc. Can also be automated like this, to convert between decimal and hex:

    i=0; while ((i<20)) ; do echo -n "$i: "; echo "base=10;obase=16;$i" | bc; ((i++)); done | paste - - - -
    0: 0 1: 1 2: 2 3: 3
    4: 4 5: 5 6: 6 7: 7
    8: 8 9: 9 10: A 11: B
    12: C 13: D 14: E 15: F
    16: 10 17: 11 18: 12 19: 13

[/example]

[reference]

[tags]Unix Shell Tips, Unix Shell Tricks, UNIX Arithmetic, Optical Dump, Unix Coding School[/tags]

[/reference]

undocument find secret

[problem]

You want to find a pattern in a file and have the line displayed.

[/problem]

[solution]

I “discovered” this undocument (well in all the doco I’ve ever read), pretty much by accident.

Basically find produces a list of files (type f), in the current directory and supplies them individually to the grep command. Ordinarily if grep is supplied a single file, it just returns the pattern. You can suffix grep with -n to also return the line number and -l to return file names without the pattern.

But you cannot (AFAIK) force grep to return the filename and the pattern, from a list of files supplied by find. Of course you could use xargs, to produce a similar mechanism – but that is yet another process to fire up. 🙂

Also you could call a script, instead of grep – which uses something like sed to substitute the beginning of the line with filename – sed “s/^/$1: /” for example.

See the example.

[/solution]

[example]

Anyway, I’m quite happy just using this little secret. 🙂 Having used it many times:

find . -type f -exec grep -i "pattern" {} /dev/null ;

[/example]

[reference]

[tags]find undocumented secret, Unix Coding School[/tags]

[/reference]

Optical Dump

[problem]

Ever need to look at a file, but have not been able to – due to hidden control characters, etc.

[/problem]

[solution]

Or just needed to see spaces and end of line characters, comparing files, etc. Optical dump or more specifically the od command, can be extremely useful for display ascii characters. See the example.

[/solution]

[example]

The basic syntax looks like this:

od -c filename
Heres a demo:
echo "hi
> world
> " | od -c
0000000 h i n w o r l d n n
0000012

Also useful if wanted to look at a binary file, without breaking your terminal. 🙂 Or looking through encryption.

Also an easy converting of ascii to hex:
dd if=/dev/urandom count=1 bs=128 | od -x
1+0 records in
1+0 records out
0000000 5715 4ac5 e457 9c7b 013b 94f8 cdb7 36b7
0000020 1ae0 a650 e236 c4e3 589f 4af2 dddd f67c
0000040 f722 58b3 7e4e 1de4 6de4 36fb 2c8a 6864
0000060 c682 f951 f966 108f b9dd eaca aefd 8a8a
0000100 da45 7e0f a3ae 350f b0b8 a1f8 1dbe aa4c
0000120 aad8 7002 f0ec 2a94 076a 70f9 0d6f cf44
0000140 1cb2 88b6 9db2 258c 6b4e 08f7 bc41 e6e2
0000160 5ec1 101a f849 b50c 322a 43cf 1520 7dba
0000200

[/example]

[reference]

[tags]optical dump, binary format, Unix Coding School[/tags]

[/reference]