Perl
Contents
Perl
Perl is an interpreted language with syntax similar to bash and PHP. It's a high level, general purpose language that is used primarily to automate repetitive tasks, and to parse and process text. Indeed, text processing was the original intended use for the language, although it has grown to be an extremely flexible language for many different types of processing.
Running Perl
Like many scripting languages, Perl programs can be run in several ways. The programs, or scripts, themselves are plain text files, usually ending in '.pl' to indicate that it is a Perl script. Given a Perl script file, the most common ways to execute the script are shown below.
- Pass the script as input to the 'perl' executable:
perl -w SCRIPT
- Execute the script directly by adding the following line to the top of the script:
#!/usr/bin/perl -w
Then, change the script to be executable, and finally execute the script:chmod a+x SCRIPT ./SCRIPT
While flexibility is one of the major strengths of Perl, the flexibility in the language itself can often lead to unintended consequences. To overcome this, it is good practice to explicitly enable warnings and strict behavior for you scripts. The -w above when running perl gets you part of the way there, and although it's not necessary to supply the -w it is nearly always the right thing to do. You should also add the line:
use strict;
to all of your scripts, near the top, to enable strict error checking.
Variables and Arrays
Standard variables start with a dollar sign, $, at the beginning of their names. Unlike shell scripts, the $ is always used, regardless of whether the variable is being set or accessed. As with other scripting languages, you don't specify the type of the variable; it will be used treated as a integer, floating point number, string, etc, depending on how it is used in your script.
#this is an integer $i=5; #this is a string $msg="my string";
Comments start with the number/sharp/hash sign, #, and extend until the end of the line. All non-comment statements end with a semicolon, ;. You can print variables with the print function:
print $msg;
Variables can be used inside strings, and the value of that variable will be included in the string. (Note that you can also use the printf function when you need specific formatting of the output).
$i=5; print "The value of i is $i";
This will generate the string:
The value of i is 5
All variables are global by default, so you need to specify locality if you want local variables. This can be done with the local and my keywords. Generally, my is safer and faster, and is therefore used most often. If a variable is defined by my in a block, it will be local to the block. If a variable is defined by local, any subroutine call from that block will also have that variable defined. For example:
$i=0;
while($i<5)
{
my $squared = $i*$i;
$cubed = $i*$i*$i;
$i++;
}
print $squared;
print $cubed;
In the above program segment, $squared is defined with my inside the while loop, so it is only defined inside that loop. $cubed is not given any special scope, so it is a global variable. Hence, after the loop, $squared will be undefined, but $cubed with have the value 64 (the last value it was assigned in the loop).
Arrays
Arrays are represented with two different symbols, depending on whether you are referring to elements of the array, or the entire array. Array variable names start with @ when declaring them or when you are referring to the array itself. The normal $ is used when you want to access individual elements of the array. Declaring arrays is fairly easy. You can enter all elements one by one, or you can enter them together:
$student[0]="Alice";
$student[1]="Bruce";
$student[2]="Charlie";
$student[3]="Dan";
or alternatively
@student=("Alice","Bruce","Charlie","Dan");
In general, lists strings or variables enclosed in parentheses, (value1,value2,...,valuen ), are interpreted as arrays. You can access the contents with an array index:
print $student[2];
You can also print the whole array:
print @student;
The above will concatenate all the elements together without a delimiter. Alternatively, you can print the array inside of a quotes
print "@student';
which will print each elements separated by a space.
The current size of an array can be accessed with the scalar() function:
$sz=scalar(@student);
You can also use special syntax to access the index of the last element in an array like so:
$last_index = $#student;
Note that you start the array variable with $# instead of @ or just $. In the example above, scalar(@student) is 4, and $#student is 3.
You can declare a sequence with
(START..END);
START and END could be numbers or a characters:
@newarray = (1 .. 20);
@newchars = ('l' .. 't');
There is also support for adding and removing elements.
- push() - adds an element to the end of an array.
- unshift() - adds an element to the beginning of an array.
- pop() - removes the last element of an array.
- shift() - removes the first element of an array.
push() and unshift() take an array and a variable as arguments:
push @ARRAY, VARIABLE
VARIABLE can actually be a single variable or another array:
push @student, "Eve";
push @student, @newstudents;
push @student, ("Frank","Gabriel");
pop() and shift() take only the array as an argument:
$laststudent = pop(@student);
You can use the slice operation to retrieve portions of an array.
@newarray = @student[1..3]; # elements 1-3 @newarray = @student[1..3,5,7,9..15]; # elements 1-3,5,7,9-15
The splice() function lets you replace arbitrary elements within an array. Here is the general syntax:
splice(@ARRAY,STARTING_INDEX,LENGTH,@REPLACING-ARRAY);
Here is an example that replaces element 3 with Jerry and elements 4 with Lena:
splice(@student,3,2,("Jerry","Lena"));
The split() operation creates an array from a string. It takes a delimiter and a string as arguments. For example:
@array=split(':',"test:array:1:2:3");
The above will produce an array with 5 elements (test,array,1,2,3).
The join() function does the reverse, i.e., it creates a string from the elements of an array. It takes a delimiter and an array as arguments. For example:
$studentNames=join(',',@student);
The above will form one string with every element from @student separated by a comma.
Perl also has hash arrays, which are similar to associate arrays. Whereas arrays are represented with the @, hash arrays are represented by the %symbol.
%ages = ("Leela", 25,
"Fry", 28,
"Bender", 4,
"Lord Nibbler", "Unknown");
Elements within a hash array accessed by prefacing the hash array name with $ and then using curly braces for the index. For example:
$ages{"Leela"}; # Returns 25
$ages{"Fry"}; # Returns 28
$ages{"Bender"}; # Returns 4
$ages{"Lord Nibbler"}; # Returns "Unknown"
Conditional Statements
if statements work as they do in most other languages. The general format is
if ( CONDITION )
{
#some statements
}
elsif ( ANOTHER_CONDITION )
{
#whatever
}
else
{
#other stuff
}
The standard comparison operators work as usual, including ==, <, >, etc. The only thing to be aware of is that you should use eq and neq to compare strings instead of == and !=.
if($string eq "some string")
{
print "strings are equal\n";
}
elsif($int == 5)
{
print "integer is five\n";
}
Loops
There are three main types of loops. for and while loops work similarly to other languages. Here are the general forms:
for(initialization;condition;loop_increment)
{
}
while(condition)
{
}
Here are simple examples:
$sum=0;
$i=1;
while($i<=10)
{
$sum=$sum+$i;
$i++;
}
print "sum is $sum\n";
for($j=0; $j<5; ++$j)
{
print "j is $j\n";
}
next and last are special commands that will jump to the next loop iteration, or to the end of the loop, respectively. For example:
for($j=0; $j<5; ++$j)
{
if($j == 2) { next; }
if($j == 4) { last; }
print "j is $j\n";
}
The above code will output:
j is 0 j is 1 j is 3
Finally, there is the foreach loop. foreach loops are most useful when iterating through arrays or hashs. Here is the general form:
foreach $iterator (@array)
{
}
For example, to print out all the values in an array, you could do this:
@names = ('Bugs', 'Daffy', 'Marvin');
foreach $name (@names)
{
print "$name\n";
}
Functions
Perl functions are identified with the keyword sub followed by the function name. Arguments passed to functions are stored in a special array @_. You can either access all of the arguments by assigning them to local variables via @_ or you can use the shift function. Here is the general form for a function:
sub functionname
{
my ($arg1,$arg2,...,$argn)=@_;
}
Or:
sub functionname
{
my $arg1 = shift;
my $arg2 = shift;
...
my $argn = shift;
}
Here is a simple function to add two numbers and a call to that function (note that functions can be defined above or below where they are called):
sub add
{
my ($x,$y) = @_;
return ($x+$y);
}
print "sum of 2+5=" . add(2,5) . "\n";
Regular Expressions
Perl Modules
Perl modules are simply Perl libraries that provide new or enhanced functionality. Although some modules are distributed as part of the core Perl distribution, most of them are written and maintained by other Perl users or developers who needed some specific functionality that was missing. The CPAN repository is the portal for finding and getting modules. CPAN also contains manuals for the modules, most of which contain examples. Generally speaking, the modules are all perl programs, although sometimes they require libraries from other languages.
The typical location for perl modules is /usr/lib/perl/XXX where XXX is the version of perl. For current Ubuntu installs, the perl version is 5.10, so the modules are in /usr/lib/perl/5.10/. You can define other locations by setting the PERL5LIB environmental variable.
export PERL5LIB=directory1:directory2
A module is contained in .pm files located in one of the module directories (either the default or through PERL5LIB). They are accessed in a perl script by including them via the use command. Most modules are grouped in to common packages. For example, there is a DBI package that contains many database interface modules. One of those modules is the mysql module. To include that module (assuming you have it installed on your system) in your script, you would add this line:
use DBI::mysql;
If a module you want is not installed on your system, you need to install it either manually or through the automated cpan command. The later approach takes a bit of set up, but it makes installing new modules much simpler.
Manual Module Installation
Search the CPAN website for the module you want, then download the module. Untar it (or unzip it) and change to the directory containing the source code. You will notice that there is a file named Makefile.PL. It's a perl script that generates a makefile for your perl installation. Run this:
perl Makefile.PL
This generates the actual makefile you need to build the module. If you want to install the module to a non-default module location, set the PREFIX environment variable:
perl Makefile.PL PREFIX=my_module_directory
Once you have the makefile, you can build the module, test it, and install it. Most Perl modules require testing to ensure they are functioning correctly on your system, so doing the tests is part of the build process. If a critical component of the test fails, you may have to resolve any problems before moving on.
make make test make install
Some modules require prerequisites which you need to install yourself. In some cases, the dependencies can easily spiral out of control so that it becomes very difficult to install everything manually, which is why the CPAN approach is preferred.
Automated Module Installation
cpan is a perl script that comes with the perl package that uses the CPAN perl module. It is a tool that installs new modules with minimum user interaction and that auto installs prerequisites. To install new modules system-wide, you'll need to run cpan as root. So, run:
sudo cpan
If it is the first time you are running cpan, you will need to answer some configuration questions. When prompted, you should select the automated configuration to get things going without any extra hassle. You will then get to the cpan prompt, where you can run commands to search for and install new modules.
cpan>
The first thing should probably install is Bundle::CPAN which installs a bunch of packages that make cpan easier to use. To do that, at the cpan> prompt, run:
cpan> install Bundle::CPAN
Note that as it installs, it will probably ask if it's ok to install additional packages and such along the way. It's generally ok to accept whatever it asks when you are prompted. The help command shows you all the available commands and options. You can search for a module with the i command:
cpan>' i /KEYWORDS/
You can install a module with the install command:
cpan>' install MODULE
install downloads the module, compiles and tests it, and finally installs it if everything went well. The downloaded modules are stored under your ~/.cpan directory (unless you specified another one during configuration), in case you need to look at source code or anything related to the module. Although cpan will download any perl modules that are needed by the module you actually want to install, it might fail if any of the modules rely on non-perl resources elsewhere on the system (e.g., graphics libraries). If that happens, you can go to the build directory, ~/.cpan/build/MODULE, and try to fix the problem.
Connecting MySQL with Perl
In order to connect a MySQL you need to have DBI module and DBI::mysql installed in your system. Following guide has a good description of DBI interface in general [1]
Then you can connect a mysql database with the command
DBI->connect(DATABASE_DESCRIPTION,USERNAME,PASS);
The databse description contains the type of the database (so that DBI knows how to connect), the name of the database and the host name. It returns a handle that you will use for this connection.
use DBI;
$database="company;host=www.bayazit.net";
$datasource="dbi:mysql";
$user="boss";
$pass="employer";
$dbh=DBI->connect("$datasource:$database",$user,$pass) || die "error connecting"; #note $dbh is your database handle
Once you have the connection established you can send SQL queries to your database server. First you need to prepare your SQL with prepare command on the database handle.
$dbh->prepare("TYPE YOUR SQL HERE");
prepare command actually returns a statement handle ($sth in our example below) that you will use for the database operations.
$sth=$dbh->prepare("insert into employee values ('Alaaddin,'Arabian Nights','Adventurer')");
You have to be careful about the your quotation marks. An easier way is to use perl qq function (which is generalize quotation)
$sql=qq(insert into employee values ('Alaaddin,'Arabian Nights','Adventurer'));
$sth=$dbh->prepare($sql);
Once you have a statement prepared, you can execute it with execute command:
$sth->execute() ||die "Can't execute sql";
prepare command actually helps you bind variables so that you don't have to repeat them. For example, if you want to have a generic way to insert new values, you can prepare your statement
$inserth=$dbh->prepare("insert into employee values (?,?,?)");
and during execution specify the arguments represented with ?
$inserth->execute('Alaaddin,'Arabian Nights','Adventurer')|| die "Can't execute SQL";
If you are querying your database, the query results are also accessed through your statement handle. After execution, $sth->fetchrow_array returns a row from your results. This function returns an array, so you can access any attribute from this array. As fetchrow_array returns only one row, you can call it continuously until it fails to return a value.
sql="select name,jobtitle from employee";
$sth=$dbh->prepare($sql);
$sth->execute()|| die "Can't execute my sql";
while(@attributes=$sth->fetchrow_array) {
print "$attributes[0] $attributes[1] \n";
}
#!/usr/bin/perl
use DBI;
$database="company;host=www.bayazit.net";
$datasource="dbi:mysql";
$user="boss";
$pass="employer";
$dbh=DBI->connect("$datasource:$database",$user,$pass) || die "error connecting";
$sql="insert into employee values ('aladdin','Arabian Nights','Adventurer')"; # sql for inserting
$sql="select name,jobtitle from employee";
$sth=$dbh->prepare($sql);
$sth->execute()|| die "Can't execute my sql";
while(@attributes=$sth->fetchrow_array) {
print "$attributes[0] $attributes[1] \n";
}
$sth->finish();
Parsing HTML with Perl
Perl module HTML::Element provides a structured way to represent HTML elements (starting with <tag>, have some attributes and finish with </tag>). Any HTML document can be represented as a tree made up of HTML::Elements.
For example, consider the following HTML page
<html>
<head>
<title>List of Employees</title>
</head>
<body>
<center>
<h1>Employees</h1> <br>
<table>
<tr>
<td>Name</td><td>Department</td><td>Title</td>
</tr>
<tr>
<td>Alice</td><td>Wonderland</td><td>Lost traveler</td>
</tr>
<tr>
<td>Peter</td><td>Neverland</td><td>Team leader</td>
</tr>
</table>
</center>
</body>
</html>
This code can be represented in a tree format
<html>
<head>
<title>
"List of Employees"
<body>
<center>
<h1>
"Employees"
<br>
<table>
<tr>
<td>
"Name"
<td>
"Department"
<td>
"Title"
<tr>
<td>
"Alice"
<td>
"Wonderland"
<td>
"Lost traveler"
<tr>
<td>
"Peter"
<td>
"Neverland"
<td>
"Team leader"
Notice the hiararchical relation. For example, <html> node has two direct children, <head> and <body> and these siblings have their own children. Perl module HTML::Tree builds this tree. You can find more about HTML::Trees at [2]]. In order to use HTML::Trees, you have to use HTML::TreeBuilder module.
use HTML::TreeBuilder; $tree=TML::TreeBuilder->new;
will create a new tree for you. You can then use this tree parse an html file. There are two ways to parse HTML, either directly parse a file with parse_file or parse a string with parse.
$tree->parse_file('downloaded.html');
You can dump the contents of an HTML::Tree by calling dump function.
$tree->dump;
You can call the parse function several times for different content, but before doing that you need to empty the existing parse data stored in the tree with delete function, otherwise, you may run out of memory.
$tree->delete;
HTML Scanning
Lets say we are interested in capturing headlines for science news at BBC Science News. Assuming you have downloaded this web page to bbc.html, you can build a tree on it.
#!/usr/bin/perl
#
use HTML::TreeBuilder;
$tree=HTML::TreeBuilder->new;
$tree->parse_file("bbc.html");
Now you can use this tree to retrieve information. Function look_down on an HTML::Tree or HTML::Element will return any child satisfying some criteria
$tree->look_down(SOMEATTRIBUTE,ITSVALUE);
For example, following code will return all HTML::Elements that are actually hyperlinks in the page and put it to an array.
@links=$tree->look_down("_tag","a");
left hand side of the call could be a variable instead of array,
$first_link=$tree->look_down("_tag","a");
In this case, instead of returning all links, it returns the first hyperlink element found.
The power of look_down function comes from ability to specify further criteria in the following format:
$tree->look_down(SOMEATTRIBUTE,ITSVALUE,
sub { SOME FUNCTION CODE HERE });
This will return any element that has its SOMEATTRIBUTE has ITVALUE and the function specified in look_down returns 1. Usually that function evaluates other attributes of the element. The element that is being evaluated is referred as $_[0]. For example, the following code returns all hyperlinks that contains URLs to cse330 website.
@links=$tree->look_down("_tag","a",
sub {
$_[0]->attr("href")=~m/www.cse330.org/
});
Note that we are accessing href attribute and using a regular expression to match it to www.cse330.org.
Now, as we said we are interested in capturing the BBC news headlines, we need to look at bbc.html to see what are the characteristics of the HTML::Elements containing the headlines.
For example, in the current website, the main headline has the following lines (headline followed by a description)
<div class="mvb">
<a class="tsh" href="/2/hi/health/7026443.stm">
Chilli compound fires painkiller
</a>
</div>
<div class="mvb">
A chemical from chilli peppers may be able to kill pain without affecting touch or movement.
</div>
.......
<div class="mvb">
<a href="/2/hi/science/nature/7023731.stm"><img src="http://newsimg.bbc.co.uk/media/images/44151000/jpg/_44151791_art_66pic.jpg" align="left" width="66" height="49" alt="Artists impression of Gryposaurus monumentensis (copyright: Larry Felder)" border="0" vspace="0" hspace="0"></a>
<img src="http://newsimg.bbc.co.uk/shared/img/o.gif" align="left" width="5" height="49" alt="" border="0" vspace="0" hspace="0">
<a class="shl" href="/2/hi/science/nature/7023731.stm">
Duck-billed dinosaur had big bite
</a>
<br clear="all" />
</div>
<div class="o">
A new species of duck-billed dinosaur that had up to 800 teeth is described by scientists.
</div>
....
<div class="mvb">
<a href="/2/hi/technology/7024672.stm"><img src="http://newsimg.bbc.co.uk/media/images/44152000/jpg/_44152097_tombstone_66pic.jpg" align="left" width="66" height="49" alt="Before and after tombstone" border="0" vspace="0" hspace="0"></a>
<img src="http://newsimg.bbc.co.uk/shared/img/o.gif" align="left" width="5" height="49" alt="" border="0" vspace="0" hspace="0">
<a class="shl" href="/2/hi/technology/7024672.stm">
Scans reveal lost gravestone text
</a>
<br clear="all" />
</div>
<div class="o">
Illegible words on church headstones could be read once more thanks to a new scan technology.
</div>
Hence if we look down on the hyperlinks that have class types tsh and shl, we will get the hyperlinks that contain the headlines.
@links=$tree->look_down('_tag','a',
sub {
$_[0]->attr('class') eq "tsh" ||
$_[0]->attr('class') =~m/hl$/
;
}
);
Then by printing out these hyperlink elements as text, we can get the headlines
for($i=0;$i<scalar(@links);$i++) {
print $links[$i]->as_text;
}
However, these elements just give us the headlines but not the summaries. But no worries, if you look at the BBC html code, the headlines are inside a
#!/usr/bin/perl
#
use HTML::TreeBuilder;
$tree=HTML::TreeBuilder->new;
$tree->parse_file("bbc.html");
@links=$tree->look_down('_tag','a',
sub {
$_[0]->attr('class') eq "tsh" ||
$_[0]->attr('class') =~m/hl$/
;
}
);
for($i=0;$i<scalar(@links);$i++) {
print $links[$i]->as_text,":";
print $links[$i]->parent->right->as_text,"\n";
}
you should be careful about these kind of operations, if, for example parent was undefined, the above code was going to be interrupted during run time. A better way is first to check if $links[$i]->parent was defined and then check if parent's right sibling was defined.
Finally, you can use LWP::UserAgent to actually make a request to the website, get the contents and pass those contents to the tree, instead of downloading the a website and calling the parser on that file.
#!/usr/bin/perl
#
use LWP::UserAgent;
$ua=LWP::UserAgent->new;
$req=$ua->get("http://news.bbc.co.uk/2/hi/science/nature/default.stm");
use HTML::TreeBuilder;
$ua->agent('Mozilla/5.0'); #you can modify several internal parameters, such as browser identification
$tree=HTML::TreeBuilder->new;
$tree->parse($req->as_string);
@links=$tree->look_down('_tag','a', #get all links
sub {
$_[0]->attr('class') eq "tsh" || #that are tsh class
($_[0]->parent->attr('class') eq "arr" &&
$_[0]->attr('href')=~m/science/
) ||# or their url contain the keyword science and their parents belong to class arr
$_[0]->attr('class') =~m/hl$/ #or their class name ends with hl
;
}
);
for($i=0;$i<scalar(@links);$i++) {
print $links[$i]->as_text,":";
if( !($links[$i]->parent->attr('class') eq "arr")) {print $links[$i]->parent->right->as_text,"\n";}
else {
print "Sideline \n";
}
}
Some other examples can be found at Perl documentation for Tree Scanning at [3]
