Archives for : perl

Date handling in Perl

Problem

You want to convert epoch into readable date and time.

Maybe you have a log file, which is pumping out lines prefixed with the epoch?



Solution

We use the perl function localtime to convert epoch into a date and time.

At the example tab is the code to convert that epoch, into a recognizable date and time.

localtime, can also return an array – take a look at the reference tag for more info.



Example


We quite simply supply the epoch to localtime and print it.


$ perl -e 'print localtime(1145980815)."n";'
Wed Apr 26 00:00:15 2006



Reference

[tags]Perl Date handling, Perl Coding School[/tags]



Column handling in Perl

Problem

During my first encounters with Perl many years ago, I asked how can I get a specific column.

It was so easy in AWK! 😉



Solution

Well here it is in Perl – see example tab.

BTW you can always write an awkscript and run it through a2p – very good for learning Perl! 🙂



Example


Show column 1:

perl -ane 'print $F[0]."n";'

Show column 2:

perl -ane 'print $F[1]."n";'

Show last column:

perl -ane 'print $F[$#F]."n";'

Show last but one column:

perl -ane 'print $F[($#F-1)]."n";'

So you just run your program, or cat your file, etc and pipe it through this code to get specific column.



Reference

[tags]Perl column handling, Awk to Perl, a2p, Perl Coding School[/tags]



Listen on port – client server demo

Problem

You want to listen on a port, maybe to test firewalls! 🙂

Excellent introduction to client server software.



Solution

Another use bit of socket programming, courtesy of Perl! 🙂

Extremely useful bit of Perl, which can be used with the other post on this site, to confirm network connectivity (port scanner).

This code (see example tab) will listen on a port (although be careful it is over 1024, unless you are running as root/admin).



Example



perl -MIO::Socket -e '$srv=IO::Socket::INET->
new(LocalPort=>$ARGV[0],Type=>SOCK_STREAM,Reuse=>1,Listen=>5)

or die "Failed trying to listen on $ARGV[0]n";while($cl=$srv->accept()) { while() { print }

} close($socket);' port

Can also use it as a cheap chat service! 🙂 Everything gets echo'd through – so just telnet host port and type away! 🙂

Also see my port scan code at the reference tab.



Reference

[tags]Perl Socket Programming, Network Programming, Perl Port Listen, Perl Client Server demo, Perl Coding School[/tags]



scan network port with perl

Problem

You want to test a network port, on a remote system – over TCP/IP.

Maybe this is a new setup, or you want confirmation it is working.

Perhaps the firewall rules have just been changed! 🙂



Solution

A nice small bit of Perl code that I’ve used thousands of times!

In fact I’m running it in most of my production environments,
as a check that the a process is not only running – but also responding.

Yep, you could just use telnet – but some systems have that taken off for security.

Additionally it is n’t as easy to program telnet – I know, I know – you can with expect. 🙂



Example


Here is just a one liner, but you can easily incorporate this into a script.

perl -MIO::Socket -e '$socket=IO::Socket::INET->
new(Proto=>tcp,PeerAddr=>$ARGV[0],PeerPort=>$ARGV[1]);
if([email protected]) { print "Failed: [email protected]" } else { print "Succeedn"; }' host port



Reference

[tags]Perl, Network, TCPIP, Port Scanner, Perl Coding School[/tags]



Obtain epoch time and calculate date yesterday

Problem

You want to capture the current epoch. Maybe to use in a log file, or as a filename.

Or maybe you want to calculate the date yesterday.



Solution

This piece of code is very useful for performing date calculations. You can obtain the current epoch (time in seconds since Jan 1 1970), then add 3600 for 1 hour – or 86400 for 24 hours hence.



Example


So use in a UNIX variable like this:

epoch=perl -M'English' -e 'print $BASETIME."n";'

To work out 24 hours ago, just subtract 86400.

perl -M'English' -e 'print(($BASETIME-86400)."n");'

Then to see the date yesterday:


$ perl -M'English' -e 'print(($BASETIME-86400)."n");'
1180746252
$ perl -M'English' -e 'print(localtime(1180746252)."n");'
Sat Jun 2 09:04:12 2007



Reference

[tags]Perl, epoch, date manipulation, Perl Coding School[/tags]



Calculate largest field big data file

Problem

I wanted to upload a delimited field to mysql db, but hit the problem that the file contained nearly 3000 rows and no schema on the required size of each field.

Therefore I needed to traverse the file and calculate the length of each field. Then at the end, print the largest field found for each column.



Solution

Perl to the rescue! Pretty easy in Perl, after scratching my head attempting with awk. 🙂

Anyway the code is under the example tab. I’m using the pipe symbol as a delimiter “|” – so just substitute this with your delimiter, cat your file and pipe it through this script.

In the example I show how you can manipulate the file, to produce pipe delimited fields too.



Example


This is how to run the script. Basically you just need to pipe your output through find largest.

cat yourfile | ./find_largest.pl

Here is the code.

#!/usr/bin/perl

@highest=();

while( < STDIN > ) {

@thisline=split(/|/);

for($i=0;$i<=$#thisline;$i++) {

$thislength=length($thisline[$i]);

if($thislength > $highest[$i]) { $highest[$i]=$thislength; }

}

}

print(join("|",@highest)."n");

exit(0);

__END__

So for example, if I want to find the largest fields in one of my web logs - for crunching into a db:

  • First off I only want lines starting with a space, then a number.
  • Next I need to replace all multiple spaces between fields, with 1 space.
  • Then I replace the spaces between each field, with a pipe.
  • Lastly I pump it through find_largest.pl - which gives me the largest sized field


[[email protected]]/var/log/httpd% grep "^ [0-9]" access_log.tools
| sed -e 's/ / /g' -e 's/ /|/g' |
~/Perl_Bin/find_largest.pl
|14|1|1|21|6|5|103|9|3|5



Reference

[tags]Perl, Data Analysis, Perl Coding School[/tags]



Perl TimeOut

Problem

You have a script which runs too long and you want to time it out, after a given number
of seconds.



Solution

Useful bit of code to time-out a section of your Perl script, via the alarm function.

See the example tab.



Example



#!/usr/bin/perl

eval {

   local %SIG;
   $SIG{ALRM}=
     sub{ die "timeout reached, after 20 seconds!n"; };
   alarm 20;
   print "sleeping for 60 secondsn";
   sleep 60; # This is where to put your code, between the alarms
   alarm 0;
};

alarm 0;

if([email protected]) { print "Error: [email protected]"; }

exit(0);

__END__

View screen shot demo of perl timeout

view screen shot of perl timeout demo



Reference

[tags]Perl timeout, Perl Coding School[/tags]



Perl WIN32 OLE – Outlook save text

Problem

How to use Perl to connect to MS Outlook. Then descend through given folders and save items to disk, as text files.



Solution

I wrote this some time ago, to traverse predefined outlook mail folders, saving items with given subject to text.

Requires WIN32:OLE perl module (which comes with activeperl by default).

Hardcoded is the upload and uploaded mail folders. Also a subject that contains – pattern: upload.

I used one outlook rule to move items from a specific source, with this subject into upload.



Example


Here is the code – if you have any dramas with it, leave a comment.


#!perl

use Win32::OLE qw(in with);
use Win32::OLE::Const 'Microsoft Outlook';

# Connect to a running version of Outlook
eval { $Outlook =
   Win32::OLE->GetActiveObject('Outlook.Application')
};

die "Outlook not installed" if [email protected];

# If that fails start up Outlook
unless(defined $Outlook) {
   $Outlook =
     Win32::OLE->new('Outlook.Application', 'Quit')
   or die "Opps, cannot start Outlook";
}

# This appears to return a ref to the object
$namespace = $Outlook->GetNamespace('MAPI');
$thisFolder=$namespace->Folders("Mailbox - mailboxna")->
   Folders('upload');
$toFolder=$namespace->Folders("Mailbox - mailboxna")->
   Folders('uploaded');

# Workaround to be able to extract key/value pairs
%thisHash=%{$thisFolder};
$name=$thisHash{'Name'};

# This is the number of items in designated folder
$count=$thisHash{'Items'}{'Count'};
open(LOGFH,">> ol_save_to_text.log")
   or die("cannot open log filen");

# Drop out if there are no mail items in this folder
if($count > 0) {

   print LOGFH "Count: $count for $namen";

   $filename='yourname';
   open(FH,"> $filename")
   or die ("cannot open $filenamen");

   for($i=1;$i<=$count;$i++) {
     print LOGFH "Count: $countn";

     $oItems=$thisFolder->Items(1);

     %thisItem=%{$oItems};
     $subject=$thisItem{'Subject'};

     if($subject =~ /pattern: upload/) {
       print LOGFH "$i: $subjectn";
       $body=$thisItem{'Body'};
       print FH "$body";
       $oItems->Move($toFolder);
     } else {
       $nonitem+=1;
     }

     $oItems->Move($toFolder);
   }

} else { print LOGFH "No Files to Processn"; }

close(LOGFH);
1;



Reference

Microsoft Outlook Keyboard Shortcuts – Courtesy of RNIB

[tags]MS Outlook OLE Perl, MS Outlook, OLE, Perl win32, cygwin, Perl, Perl Coding School[/tags]



Perl libcurl demo

Problem

You want to use libcurl, driven through Perl.



Solution

Beautiful for parse HTML and either extracted (screen scraping) content or performing actions based on results.

See the examples tab for this simple script, demonstrating the libcurl API for Perl.



Example



#!/usr/bin/perl

$url="http://perl.coding-school.com/"; # set your url here
$|++;

use Curl::easy;
# Init the curl session

my $curl= Curl::easy::init() or die "curl init failed!n err: $!n";

sub body_callback {
   my ($chunk,$context)[email protected]_;
   push @{$context}, $chunk;
   return length($chunk);
}

Curl::easy::setopt
   ($curl, CURLOPT_PROXY, $proxy) if($proxy);
Curl::easy::setopt
   ($curl, CURLOPT_PROXYPORT, $proxyport) if($proxyport);
Curl::easy::setopt
   ($curl, CURLOPT_SSL_VERIFYHOST, 0);
Curl::easy::setopt
   ($curl, CURLOPT_SSL_VERIFYPEER, 0);
Curl::easy::setopt
   ($curl, CURLOPT_URL, $url);
Curl::easy::setopt
   ($curl, CURLOPT_WRITEFUNCTION, &body_callback);

my @body;

Curl::easy::setopt
   ($curl, CURLOPT_FILE, @body);
Curl::easy::setopt
   ($curl, CURLOPT_ERRORBUFFER, "errbuf");
if (Curl::easy::perform
   ($curl) != 0) { print "Failed : $errbufn"; };
Curl::easy::cleanup($curl);

# Separate each line into one element in array
@lines=();

foreach (@body) { push(@lines,split('n', $_, 9999)); }

foreach (@lines) {
   # just to demonstrate it works!
   if(/icons/) { print("$_n"); }
}

exit(0);

Here is a demo screen shot of this code using perl and libcurl.



Reference

[tags]Perl libcurl demo, Perl libcurl, Perl, libcurl, curl, Perl Coding School[/tags]



Perl SHA digest

Problem

You want to generate a SHA digest for a given string.



Solution

Perl has great encryption and digesting algorithms.

In the example tab we use the Digest Perl module.



Example



perl -M'Digest::SHA1 qw(sha1_hex)'
-e '$dig=sha1_hex("my big test"); print "$dign";'

419e6139a21f51a3f2ea1a783cfe536a0dada873



Reference

[tags]Perl SHA Digest, Perl, SHA, Encryption, Perl Coding School[/tags]



epoch generation and converting

Problem

You want to generate the current epoch offset for UNIX. The count forward since 8am on 1st Jan 1970.

Or conversely you want to see the date and time, for a given epoch offset.



Solution

You can use perl’s mktime or localtime – or just use my tools below.



Example


Either enter epoch:

Or enter the date (plus time)

daymonthyear
hrminssecs

epoch offset or date will appear here



Reference

[tags]epoch, unix, 2038[/tags]

  1. techno park epoch year 2038 problem


libcurl lookup taking long time

Problem

You notice for some hosts curl and/or libcurl are taking 7 or so secs to lookup the host. But nslookup or dig respond almost immediately.



Solution

Try doing curl -4 -w “time_namelookup: %{time_namelookup}n” …. and try without the -4. If you see drastic differences, you could be falling foul of the IPV6 bug. To work around this, you can either just use -4 with curl or set CURLOPT_IPRESOLVE with libcurl.



Example


I spents hours and hours and hours, trying to get this working with libcurl and perl. It turned out I need to upgrade to WWW::Curl::Easy version 3.0 – as opposed to version 2. Do a search on search.cpan.org for Curl and it will tell you how to check your version.

Then if you like me upgrade, you need to also modify all your code from: Curl::easy::setopt($curl, … to $curl->setopt( …



Reference



HTTP Basic Auth with HTTP headers and libcurl

Problem

Came across an interesting problem with curl.

For curl to perform HTTP Basic Authentication, it is easy to pass –user to the curl command, but harder with libcurl.

Suspect there is an attribute that can be set, but I monitor a multitude of web sites through some perl scripts and libcurl. I did n’t want to have to modify my wrapper scripts, which mesh perl hashes with the code that drives curl (via libcurl).

I do allow for headers though, having needed to pass different things through, like HTTP_REFERER, LAST_MODIFIED, etc.

Therefore I just needed to pass the HTTP BASIC Authentication through as a header.



Solution

First off you need to base64 encode the user and password.

Use Google’s tool



Example


We then strip out the equals and pass following through to curl or libcurl:

@myheaders=('Authorization: Basic YWRtaW46YWRtaW4');

Curl::easy::setopt($curl, Curl::easy::CURLOPT_HTTPHEADER, @myheaders);

Or from the command line:

curl ... -H'Authorization: Basic YWRtaW46YWRtaW4'

Obviously you need to put your user, followed by a colon and your password – to obtain the correct base64 encoding back – strip the equals out and away you go.



Reference



Useful Perl script – convert text to ascii

Problem

You want to convert text to ascii values.



Solution

Perl script to convert standard input to ascii values, useful with web code.

You can take the output and insert directly into HTML pages.

#!/usr/bin/perl

while(sysread(STDIN,$a,1)) {

$c=ord($a);if($c==10) { print "n"; } else { print "&#$c;"; }

}

exit(0);

__END__


Example


Here is a screen shot:

convert characters to ascii values in perl



Reference