Getting Data from the Web with Perl and Faroo API

As stated on Wikipedia “The number of available web APIs has grown consistently over the past years, as businesses realize the growth opportunities associated with running an open platform, that any developer can interact with.” [1]
For web developers web API (application programming interface) allows to create own application using existing functionality from another web application instead of creating everything from scratch.

For example if you are building application that is delivering information from the web to the users you can take Faroo API and you will need only to add user interface and connection to Faroo API service. Faroo API seems like a perfect solution for providing news api in all kind of format, either you are building a web application, website or mobile application.

This is because Faroo API is doing the work of getting data from the web. This API provides data from the web and has such functionalities as web search (more than 2 billion pages indexed as at the time of writing), news search (newspapers, magazines and blogs), trending news (grouped by topic, topics sorted by buzz), trending topics, trending terms, suggestions. The output of this API can be in different formats (json, xml, rss). You need only to make a call to Faroo API service and send the returned data to user interface. [2]

In this post the perl script that doing is showing data returned from web search for the given by user keywords will be implemented.

Connecting to Faroo API
The first step is to connect to Faroo API. The code snippet for this is shown below. The required parameters are specified via query string for server endpoint URL. For more details see Faroo API website [2]

  • q – search terms or keywords for web search
  • start – starting number of results
  • l – language, in our case english
  • src – source of data, in our case web search
  • f – format of returned data, in this example we use json format
  • key – registration key, should be obtained from Faroo API website for free

use LWP::UserAgent;
use HTTP::Request::Common qw{ POST };

my $data = CGI->new();
my $q = $data->param('q');

my $ua = LWP::UserAgent->new;
my $server_endpoint = "http://www.faroo.com/api";
$server_endpoint=$server_endpoint."?q=$q&start=1&l=en&src=web&f=json&key=xxxxxxxxx&jsoncallback=?";
my $req = HTTP::Request->new(GET => $server_endpoint);
$resp = $ua->request($req);

Processing JSON Data and Displaying Data to Web User
If our call to Faroo API was successful we would get data and can start to display to web user as in the below code snippet:


use JSON qw( decode_json );

$resp = $ua->request($req);
if ($resp->is_success) {
   
my $message = decode_json($resp->content);

$items_N= $message->{count};
if($items_N >10) {$items_N=10;}
for ($i=0; $i<$items_N; $i++)
{

print  $message->{results}->[$i]->{title};

print  $message->{results}->[$i]->{url};

print  $message->{results}->[$i]->{kwic};

}

$next_number = 10 + $message->{start};    
}
else {
    print "HTTP GET error code: ", $resp->code, "\n";
    print "HTTP GET error message: ", $resp->message, "\n";
}

Full Source Code and Online Demo
The web search based on this perl code can be viewed and tested online at Demo for web search based on Faroo API

And here is the perl script, please note that some HTML formatting is not shown.


#!/usr/bin/perl
print "Content-type: text/html\n\n";
use LWP::UserAgent;
use HTTP::Request::Common qw{ POST };
use JSON qw( decode_json );
use CGI;


my $data = CGI->new();
my $q = $data->param('q');

my $ua = LWP::UserAgent->new;
my $server_endpoint = "http://www.faroo.com/api";
$server_endpoint=$server_endpoint."?q=$q&start=1&l=en&src=web&f=json&key=xxxxxxxx&jsoncallback=?";
my $req = HTTP::Request->new(GET => $server_endpoint);

$resp = $ua->request($req);
if ($resp->is_success) {
     print " SUCCESS...";
    
my $message = decode_json($resp->content);

$items_N= $message->{count};
if($items_N >10) {$items_N=10;}
for ($i=0; $i<$items_N; $i++)
{
print  $message->{results}->[$i]->{title};
print  $message->{results}->[$i]->{url};
print  $message->{results}->[$i]->{kwic};
}

$next_number = 10 + $message->{start};    
}
else {
    print "HTTP GET error code: ", $resp->code, "\n";
    print "HTTP GET error message: ", $resp->message, "\n";
}

Thus we looked at how to connect to Faroo API , how to get data returned by this API service, how to process json data and how to display data to user.
If your website is showing some content then it can be complemented by content returned from Faroo API.
Feel free to ask questions, suggestions, modifications.

References

1. Wikipedia

2. Faroo – Free Search API

3. Demo for web search based on Faroo API



Leave a Comment