{"id":220,"date":"2016-05-24T01:44:57","date_gmt":"2016-05-24T01:44:57","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=220"},"modified":"2016-06-19T17:02:39","modified_gmt":"2016-06-19T17:02:39","slug":"getting-data-from-the-web","status":"publish","type":"post","link":"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/","title":{"rendered":"Getting Data from the Web with Perl and Faroo API"},"content":{"rendered":"<p>As stated on Wikipedia &#8220;The number of available web APIs has grown consistently over the past years, as businesses realize the growth opportunities associated with running an open platform, that any developer can interact with.&#8221; [1]<br \/>\nFor web developers web API (application programming interface) allows to create own application using existing functionality from another web application instead of creating everything from scratch.<\/p>\n<p>For example if you are building application that is delivering information from the web to the users you can take Faroo API and you will need only to add user interface and connection to Faroo API service. Faroo API seems like a perfect solution for providing news api in all kind of format, either you are building a web application, website or mobile application. <\/p>\n<p>This is because Faroo API is doing the work of getting data from the web. This API provides data from the web and has such functionalities as web search (more than 2 billion pages indexed as at the time of writing), news search (newspapers, magazines and blogs), trending news (grouped by topic, topics sorted by buzz), trending topics, trending terms, suggestions. The output of this API can be in different formats (json, xml, rss). You need only to make a call to Faroo API service and send the returned data to user interface. [2]<\/p>\n<p>In this post the perl script that doing is showing data returned from web search for the given by user keywords will be implemented. <\/p>\n<p><strong>Connecting to Faroo API<\/strong><br \/>\nThe first step is to connect to Faroo API. The code snippet for this is shown below. The required parameters are specified via query string for server endpoint URL. For more details see Faroo API website [2]<\/p>\n<ul>\n<li>q &#8211; search terms or keywords for web search<\/li>\n<li>start &#8211; starting number of results<\/li>\n<li>l &#8211; language, in our case english<\/li>\n<li>src &#8211; source of data, in our case web search<\/li>\n<li>f &#8211; format of returned data, in this example we use json format<\/li>\n<li>key &#8211; registration key, should be obtained from Faroo API website for free<\/li>\n<\/ul>\n<pre><code>\r\nuse LWP::UserAgent;\r\nuse HTTP::Request::Common qw{ POST };\r\n\r\nmy $data = CGI->new();\r\nmy $q = $data->param('q');\r\n\r\nmy $ua = LWP::UserAgent->new;\r\nmy $server_endpoint = \"http:\/\/www.faroo.com\/api\";\r\n$server_endpoint=$server_endpoint.\"?q=$q&start=1&l=en&src=web&f=json&key=xxxxxxxxx&jsoncallback=?\";\r\nmy $req = HTTP::Request->new(GET => $server_endpoint);\r\n$resp = $ua->request($req);\r\n<\/code><\/pre>\n<p><strong>Processing JSON Data and Displaying Data to Web User<\/strong><br \/>\nIf our call to Faroo API was successful we would get data and can start to display to web user as in the below code snippet:<\/p>\n<pre><code>\r\nuse JSON qw( decode_json );\r\n\r\n$resp = $ua->request($req);\r\nif ($resp->is_success) {\r\n   \r\nmy $message = decode_json($resp->content);\r\n\r\n$items_N= $message->{count};\r\nif($items_N >10) {$items_N=10;}\r\nfor ($i=0; $i<$items_N; $i++)\r\n{\r\n\r\nprint  $message->{results}->[$i]->{title};\r\n\r\nprint  $message->{results}->[$i]->{url};\r\n\r\nprint  $message->{results}->[$i]->{kwic};\r\n\r\n}\r\n\r\n$next_number = 10 + $message->{start};    \r\n}\r\nelse {\r\n    print \"HTTP GET error code: \", $resp->code, \"\\n\";\r\n    print \"HTTP GET error message: \", $resp->message, \"\\n\";\r\n}\r\n<\/code><\/pre>\n<p><strong>Full Source Code and Online Demo<\/strong><br \/>\nThe web search based on this perl code can be viewed and tested online at  <a href=\"http:\/\/intelligentonlinetools.com\/cgi-bin\/g\/faroo.cgi\" target=\"_blank\">Demo for web search based on Faroo API<\/a><\/p>\n<p>And here is the perl script, please note that some HTML formatting is not shown.<\/p>\n<pre><code>\r\n#!\/usr\/bin\/perl\r\nprint \"Content-type: text\/html\\n\\n\";\r\nuse LWP::UserAgent;\r\nuse HTTP::Request::Common qw{ POST };\r\nuse JSON qw( decode_json );\r\nuse CGI;\r\n\r\n\r\nmy $data = CGI->new();\r\nmy $q = $data->param('q');\r\n\r\nmy $ua = LWP::UserAgent->new;\r\nmy $server_endpoint = \"http:\/\/www.faroo.com\/api\";\r\n$server_endpoint=$server_endpoint.\"?q=$q&start=1&l=en&src=web&f=json&key=xxxxxxxx&jsoncallback=?\";\r\nmy $req = HTTP::Request->new(GET => $server_endpoint);\r\n\r\n$resp = $ua->request($req);\r\nif ($resp->is_success) {\r\n     print \" SUCCESS...\";\r\n    \r\nmy $message = decode_json($resp->content);\r\n\r\n$items_N= $message->{count};\r\nif($items_N >10) {$items_N=10;}\r\nfor ($i=0; $i<$items_N; $i++)\r\n{\r\nprint  $message->{results}->[$i]->{title};\r\nprint  $message->{results}->[$i]->{url};\r\nprint  $message->{results}->[$i]->{kwic};\r\n}\r\n\r\n$next_number = 10 + $message->{start};    \r\n}\r\nelse {\r\n    print \"HTTP GET error code: \", $resp->code, \"\\n\";\r\n    print \"HTTP GET error message: \", $resp->message, \"\\n\";\r\n}\r\n<\/code><\/pre>\n<p>Thus we looked at how to connect to Faroo API , how to get data returned by this API service, how to process json data and how to display data to user.<br \/>\nIf your website is showing some content then it can be complemented by content returned from Faroo API.<br \/>\nFeel free to ask questions, suggestions, modifications.<\/p>\n<p><strong>References<\/strong><\/p>\n<p>1. <a href=\"https:\/\/en.wikipedia.org\/wiki\/Web_API\" target=\"_blank\">Wikipedia<\/a><\/p>\n<p>2. <a href=\"http:\/\/www.faroo.com\/hp\/api\/api.html\" target=\"_blank\">Faroo &#8211; Free Search API<\/a><\/p>\n<p>3. <a href=\"http:\/\/intelligentonlinetools.com\/cgi-bin\/g\/faroo.cgi\" target=\"_blank\">Demo for web search based on Faroo API<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>As stated on Wikipedia &#8220;The number of available web APIs has grown consistently over the past years, as businesses realize the growth opportunities associated with running an open platform, that any developer can interact with.&#8221; [1] For web developers web API (application programming interface) allows to create own application using existing functionality from another web &#8230; <a title=\"Getting Data from the Web with Perl and Faroo API\" class=\"read-more\" href=\"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[11,7],"tags":[],"jetpack_publicize_connections":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v20.4 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Getting Data from the Web with Perl and Faroo API - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Getting Data from the Web with Perl and Faroo API - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"As stated on Wikipedia &#8220;The number of available web APIs has grown consistently over the past years, as businesses realize the growth opportunities associated with running an open platform, that any developer can interact with.&#8221; [1] For web developers web API (application programming interface) allows to create own application using existing functionality from another web ... Read more\" \/>\n<meta property=\"og:url\" content=\"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2016-05-24T01:44:57+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-06-19T17:02:39+00:00\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/\",\"url\":\"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/\",\"name\":\"Getting Data from the Web with Perl and Faroo API - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2016-05-24T01:44:57+00:00\",\"dateModified\":\"2016-06-19T17:02:39+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Getting Data from the Web with Perl and Faroo API\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Getting Data from the Web with Perl and Faroo API - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/","og_locale":"en_US","og_type":"article","og_title":"Getting Data from the Web with Perl and Faroo API - Machine Learning Applications","og_description":"As stated on Wikipedia &#8220;The number of available web APIs has grown consistently over the past years, as businesses realize the growth opportunities associated with running an open platform, that any developer can interact with.&#8221; [1] For web developers web API (application programming interface) allows to create own application using existing functionality from another web ... Read more","og_url":"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/","og_site_name":"Machine Learning Applications","article_published_time":"2016-05-24T01:44:57+00:00","article_modified_time":"2016-06-19T17:02:39+00:00","author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/","url":"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/","name":"Getting Data from the Web with Perl and Faroo API - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2016-05-24T01:44:57+00:00","dateModified":"2016-06-19T17:02:39+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Getting Data from the Web with Perl and Faroo API"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-3y","jetpack-related-posts":[{"id":827,"url":"https:\/\/intelligentonlinetools.com\/blog\/2017\/01\/11\/apis\/","url_meta":{"origin":220,"position":0},"title":"Useful APIs for Your Web Site","date":"January 11, 2017","format":false,"excerpt":"Here\u2019s a useful list of resources on how to create an API, compiled from posts that were published recently on this blog. The included APIs can provide a fantastic ways to enhance websites. 1. The WordPress(WP) API exposes a simple yet powerful interface to WP Query, the posts API, post\u2026","rel":"","context":"In &quot;API Programming&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":313,"url":"https:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/","url_meta":{"origin":220,"position":1},"title":"Getting Data from the Web with Perl and The  Guardian API","date":"June 15, 2016","format":false,"excerpt":"In one of previous post the Faroo API was used in order to get data content from the web. In this post we will look at different API that can be also used for downloading content from web. Here we will use the Guardian API \/ open platform. At the\u2026","rel":"","context":"In &quot;API Programming&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":383,"url":"https:\/\/intelligentonlinetools.com\/blog\/2016\/07\/03\/getting-the-data-from-the-web-using-php-for-api-using-the-api-with-php\/","url_meta":{"origin":220,"position":2},"title":"Getting the Data from the Web using PHP or Python for API","date":"July 3, 2016","format":false,"excerpt":"In the previous posts [1],[2] perl was used to get content from the web through Faroo API and Guardian APIs. In this post PHP and Pyhton will be used to get web data using same APIs. PHP has a powerful JSON parsing mechanism, which, because PHP is a dynamic language,\u2026","rel":"","context":"In &quot;API Programming&quot;","img":{"alt_text":"Trend for Python, Perl, PHP","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2016\/07\/trend_for_python_perl_php-300x144.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":748,"url":"https:\/\/intelligentonlinetools.com\/blog\/2016\/12\/25\/quotes-api-for-web-designers-and-developers\/","url_meta":{"origin":220,"position":3},"title":"Quotes API for Web Designers and Developers","date":"December 25, 2016","format":false,"excerpt":"No one can deny the power of a good quote. They motivate and inspire us to be our best. [1] Here are 3 quotes API that can be integrated in your website with source code example in perl. 1. Random Famous Quotes provides a random quote from famous movies in\u2026","rel":"","context":"In &quot;API Programming&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":65,"url":"https:\/\/intelligentonlinetools.com\/blog\/2016\/02\/09\/cloud-computing\/","url_meta":{"origin":220,"position":4},"title":"Cloud Computing","date":"February 9, 2016","format":false,"excerpt":"Cloud computing is the use of computing resources (hardware and software) that are delivered as a service over a network (typically the Internet). The name comes from the use of a cloud-shaped symbol as an abstraction for the complex infrastructure it contains in system diagrams. Cloud computing entrusts remote services\u2026","rel":"","context":"Similar post","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2427,"url":"https:\/\/intelligentonlinetools.com\/blog\/2018\/12\/16\/integrating-sentiment-analysis-api-python-django-web-application\/","url_meta":{"origin":220,"position":5},"title":"Integrating Sentiment Analysis API Python Django into Web Application","date":"December 16, 2018","format":false,"excerpt":"In this post we will learn how to use sentiment analysis with API python from paralleldots.com. We will look at running this API from python environment on laptop and also in web application environment with python Django on pythonanywhere hosting site. In the one of previous post we set python\u2026","rel":"","context":"In &quot;API Programming&quot;","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/12\/neural-network-3637503_640-e1544973390491.png?resize=350%2C200","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/220"}],"collection":[{"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=220"}],"version-history":[{"count":31,"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/220\/revisions"}],"predecessor-version":[{"id":346,"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/220\/revisions\/346"}],"wp:attachment":[{"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=220"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=220"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=220"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}