{"id":313,"date":"2016-06-15T01:10:48","date_gmt":"2016-06-15T01:10:48","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=313"},"modified":"2016-07-04T16:16:02","modified_gmt":"2016-07-04T16:16:02","slug":"faroo","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/","title":{"rendered":"Getting Data from the Web with Perl and The Guardian API"},"content":{"rendered":"

In one of previous post the Faroo API was used in order to get data content from the web. In this post we will look at different API that can be also used for downloading content from web. Here we will use the Guardian API \/ open platform.
\nAt the time of writing at stated on website it has over 1.7 million pieces of content that can be used to build apps. This is the great opportunity to supplement your articles with related Guardian content. And we will look how to do this.<\/p>\n

Specifically the perl script will be used for getting web search results with Guardian API. The following are the main steps in this perl script:
\nConnecting to Gurdian API<\/strong>
\nIn this step we provide our API key and parameters to search function with the search terms string. <\/p>\n

\r\nuse LWP::UserAgent;\r\nuse HTTP::Request::Common qw{ POST };\r\nmy $ua = LWP::UserAgent->new;\r\nmy $server_endpoint = \"http:\/\/content.guardianapis.com\/search\";\r\n$server_endpoint=$server_endpoint.\"?q=$q&format=json&api-key=xxxxxxxx&page-size=10&page=$page\";\r\nmy $req = HTTP::Request->new(GET => $server_endpoint);\r\n<\/code><\/pre>\n
Getting the Search Results and Decoding json data<\/strong>
\nIn this step we decode json text that we got returned from our call to web search function.<\/p>\n
\r\nuse JSON qw( decode_json );\r\n$resp = $ua->request($req);\r\nif ($resp->is_success) {\r\n        my $message = decode_json($resp->content);\r\n###if we want to print to look at raw data:\r\n###print $resp->content;\r\n}\r\n<\/code><\/pre>\n
Displaying data<\/strong>
\nNow we are displaying data<\/p>\n
\r\n\r\nuse JSON qw( decode_json );\r\n\r\n$items_N=10;\r\nfor ($i=0; $i<$items_N; $i++)\r\n{\r\n\r\nprint  $message->{response}->{results}->[$i]->{webTitle};\r\nprint  $message->{response}->{results}->[$i]->{webUrl};\r\n}\r\n\r\n<\/code><\/pre>\n
Conclusion<\/strong>
\nThus we looked at how to connect to The Guardian API , how to get data returned by this API service, how to process json data and how to display data to user.
\nIf your website is showing some content then it can be complemented by content returned from The Guardian API.
\nFeel free to ask questions, suggestions, modifications.<\/p>\n
References<\/strong>
\n1. The Guardian<\/a>
\n2. Online example of web search with the perl script<\/a><\/p>\n
Below is the full perl source code<\/p>\n
\r\n#!\/usr\/bin\/perl\r\n\r\nuse LWP::UserAgent;\r\nuse HTTP::Request::Common qw{ POST };\r\nuse JSON qw( decode_json );\r\nuse CGI;\r\n\r\nmy $data = CGI->new();\r\nmy $q = $data->param('q');\r\nmy $start = $data->param('start');\r\n\r\nif (($start eq \"\") || ($start == 0)) \r\n{$page=1;}\r\nelse\r\n{  \r\n    $page= int($start \/ 10) ;\r\n}\r\n\r\n\r\nif ($start eq \"\") {$start = 1;}\r\n\r\nmy $ua = LWP::UserAgent->new;\r\n\r\n\r\n# http:\/\/open-platform.theguardian.com\/documentation\/search\r\nmy $server_endpoint = \"http:\/\/content.guardianapis.com\/search\";\r\n$server_endpoint=$server_endpoint.\"?q=$q&format=json&api-key=xxxx&page-size=10&page=$page\";\r\n\r\n\r\nmy $req = HTTP::Request->new(GET => $server_endpoint);\r\n\r\n$resp = $ua->request($req);\r\nif ($resp->is_success) {\r\n     \r\nmy $message = decode_json($resp->content);\r\n\r\n###if we want to print to look at raw data:\r\n###print $resp->content;\r\n\r\n$items_N=10;\r\nfor ($i=0; $i<$items_N; $i++)\r\n{\r\nprint  $message->{response}->{results}->[$i]->{webTitle};\r\nprint $message->{response}->{results}->[$i]->{webUrl};\r\n}\r\n  \r\n}\r\nelse {\r\n    print \"HTTP GET error code: \", $resp->code, \"\\n\";\r\n    print \"HTTP GET error message: \", $resp->message, \"\\n\";\r\n}\r\n\r\n<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
In one of previous post the Faroo API was used in order to get data content from the web. In this post we will look at different API that can be also used for downloading content from web. Here we will use the Guardian API \/ open platform. At the time of writing at stated … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[11,7],"tags":[],"jetpack_publicize_connections":[],"yoast_head":"\nGetting Data from the Web with Perl and The Guardian API - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Getting Data from the Web with Perl and The Guardian API - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"In one of previous post the Faroo API was used in order to get data content from the web. In this post we will look at different API that can be also used for downloading content from web. Here we will use the Guardian API \/ open platform. At the time of writing at stated ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2016-06-15T01:10:48+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-07-04T16:16:02+00:00\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/\",\"name\":\"Getting Data from the Web with Perl and The Guardian API - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2016-06-15T01:10:48+00:00\",\"dateModified\":\"2016-07-04T16:16:02+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Getting Data from the Web with Perl and The Guardian API\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Getting Data from the Web with Perl and The Guardian API - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/","og_locale":"en_US","og_type":"article","og_title":"Getting Data from the Web with Perl and The Guardian API - Machine Learning Applications","og_description":"In one of previous post the Faroo API was used in order to get data content from the web. In this post we will look at different API that can be also used for downloading content from web. Here we will use the Guardian API \/ open platform. At the time of writing at stated ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/","og_site_name":"Machine Learning Applications","article_published_time":"2016-06-15T01:10:48+00:00","article_modified_time":"2016-07-04T16:16:02+00:00","author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/","name":"Getting Data from the Web with Perl and The Guardian API - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2016-06-15T01:10:48+00:00","dateModified":"2016-07-04T16:16:02+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Getting Data from the Web with Perl and The Guardian API"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/s7h1IJ-faroo","jetpack-related-posts":[{"id":827,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/11\/apis\/","url_meta":{"origin":313,"position":0},"title":"Useful APIs for Your Web Site","date":"January 11, 2017","format":false,"excerpt":"Here\u2019s a useful list of resources on how to create an API, compiled from posts that were published recently on this blog. The included APIs can provide a fantastic ways to enhance websites. 1. The WordPress(WP) API exposes a simple yet powerful interface to WP Query, the posts API, post\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":383,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/07\/03\/getting-the-data-from-the-web-using-php-for-api-using-the-api-with-php\/","url_meta":{"origin":313,"position":1},"title":"Getting the Data from the Web using PHP or Python for API","date":"July 3, 2016","format":false,"excerpt":"In the previous posts [1],[2] perl was used to get content from the web through Faroo API and Guardian APIs. In this post PHP and Pyhton will be used to get web data using same APIs. PHP has a powerful JSON parsing mechanism, which, because PHP is a dynamic language,\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"Trend for Python, Perl, PHP","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2016\/07\/trend_for_python_perl_php-300x144.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":220,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/","url_meta":{"origin":313,"position":2},"title":"Getting Data from the Web with Perl and Faroo API","date":"May 24, 2016","format":false,"excerpt":"As stated on Wikipedia \"The number of available web APIs has grown consistently over the past years, as businesses realize the growth opportunities associated with running an open platform, that any developer can interact with.\" [1] For web developers web API (application programming interface) allows to create own application using\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":748,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/25\/quotes-api-for-web-designers-and-developers\/","url_meta":{"origin":313,"position":3},"title":"Quotes API for Web Designers and Developers","date":"December 25, 2016","format":false,"excerpt":"No one can deny the power of a good quote. They motivate and inspire us to be our best. [1] Here are 3 quotes API that can be integrated in your website with source code example in perl. 1. Random Famous Quotes provides a random quote from famous movies in\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":766,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/","url_meta":{"origin":313,"position":4},"title":"Retrieving Post Data Using the WordPress API with Python Script","date":"December 31, 2016","format":false,"excerpt":"In this post we will create python script that is able to get data from WordPress (WP) blog using WP API. This script will save downloaded data into csv file for further analysis or other purposes. WP API is returning data in json format and is accessible through link http:\/\/hostname.com\/wp-json\/wp\/v2\/posts.\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":65,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/02\/09\/cloud-computing\/","url_meta":{"origin":313,"position":5},"title":"Cloud Computing","date":"February 9, 2016","format":false,"excerpt":"Cloud computing is the use of computing resources (hardware and software) that are delivered as a service over a network (typically the Internet). The name comes from the use of a cloud-shaped symbol as an abstraction for the complex infrastructure it contains in system diagrams. Cloud computing entrusts remote services\u2026","rel":"","context":"Similar post","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/313"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=313"}],"version-history":[{"count":31,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/313\/revisions"}],"predecessor-version":[{"id":399,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/313\/revisions\/399"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=313"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=313"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=313"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}