{"id":766,"date":"2016-12-31T21:31:50","date_gmt":"2016-12-31T21:31:50","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=766"},"modified":"2017-01-02T13:44:49","modified_gmt":"2017-01-02T13:44:49","slug":"retrieving-post-data-using-the-wordpress-api-with-python-script","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/","title":{"rendered":"Retrieving Post Data Using the WordPress API with Python Script"},"content":{"rendered":"

In this post we will create python script that is able to get data from WordPress (WP) blog using WP API. This script will save downloaded data into csv file for further analysis or other purposes.
\nWP API is returning data in json format and is accessible through link http:\/\/hostname.com\/wp-json\/wp\/v2\/posts. The WP API is packaged as a plugin so it should be added to WP blog from plugin page[6]<\/p>\n

Once it is added and activated in WordPress blog, here\u2019s how you\u2019d typically interact with WP-API resources:
\nGET \/wp-json\/wp\/v2\/posts to get a collection of Posts. <\/p>\n

Other operations such as getting random post, getting posts for specific category, adding post to blog (with POST method, it will be shown later in this post) and retrieving specific post are possible too. <\/p>\n

During the testing it was detected that the default number of post per page is 10, however you can specify the different number up to 100. If you need to fetch more than 100 then you need to iterate through pages [3]. The default minimum per page is 1.<\/p>\n

Here are some examples to get 100 posts per page from category sport:
\nhttp:\/\/hostname.com\/wp-json\/wp\/v2\/posts\/?filter[category_name]=sport&per_page=100<\/p>\n

To return posts from all categories:
\nhttp:\/\/hostname.com\/wp-json\/wp\/v2\/posts\/?per_page=100<\/p>\n

To return post from specific page 2 (if there are several pages of posts) use the following:
\nhttp:\/\/hostname.com\/wp-json\/wp\/v2\/posts\/?page=2<\/p>\n

Here is how can you make POST request to add new post to blog. You would need use method post with requests instead of get. The code below however will not work and will return response code 401 which means “Unauthorized”. To do successful post adding you need also add actual credentials but this will not be shown in this post as this is not required for getting posts data. <\/p>\n

\r\nimport requests\r\nurl_link=\"http:\/\/hostname.com\/wp-json\/wp\/v2\/posts\/\"\r\n\r\ndata = {'title':'new title', \"content\": \"new content\"}\r\nr = requests.post(url_link, data=data)\r\nprint (r.status_code)\r\nprint (r.headers['content-type'])\r\nprint (r.json)\r\n<\/code><\/pre>\n
The script provided in this post will be doing the following steps:
\n 1. Get data from blog using WP API and request from urlib as below:<\/p>\n
\r\nfrom urllib.request import urlopen\r\nwith urlopen(url_link) as url:\r\n    data = url.read()\r\nprint (data)\r\n<\/code><\/pre>\n
 2. Save json data as the text file. This also will be helpful when we need to see what fields are available, how to navigate to needed fields or if we need extract more information later.<\/p>\n
 3. Open the saved text file, read json data, extract needed fields(such as title, content) and save extracted information into csv file.<\/p>\n
In the future posts we will look how to do text mining from extracted data.<\/p>\n
Here is the full source code for retrieving post data script.<\/p>\n
\r\n# -*- coding: utf-8 -*-\r\n\r\nimport os\r\nimport csv\r\nimport json\r\n\r\nurl_link=\"http:\/\/hostname.com\/wp-json\/wp\/v2\/posts\/?per_page=100\"\r\n\r\nfrom urllib.request import urlopen\r\n\r\nwith urlopen(url_link) as url:\r\n    data = url.read()\r\n\r\nprint (data)\r\n\r\n# Write data to file\r\nfilename = \"posts json1.txt\"\r\nfile_ = open(filename, 'wb')\r\nfile_.write(data)\r\nfile_.close()\r\n\r\n\r\ndef save_to_file (fn, row, fieldnames):\r\n    \r\n         if (os.path.isfile(fn)):\r\n             m=\"a\"\r\n         else:\r\n             m=\"w\"\r\n  \r\n         \r\n         with open(fn, m, encoding=\"utf8\", newline='' ) as csvfile: \r\n           \r\n             writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\r\n             if (m==\"w\"):\r\n                 writer.writeheader()\r\n             writer.writerow(row)\r\n\r\nwith open(filename) as json_file:\r\n    json_data = json.load(json_file)\r\n    \r\n \r\n \r\nfor n in json_data:  \r\n \r\n  r={}\r\n  r[\"Title\"] = n['title']['rendered']\r\n  r[\"Content\"] = n['content']['rendered']\r\n  save_to_file (\"posts.csv\", r, ['Title', 'Content'])  \r\n<\/code><\/pre>\n
Below are the links that are related to the topic or were used in this post. Some of them are describing the same but in javascript\/jQuery environment.<\/p>\n
References<\/strong><\/p>\n
1. WP REST API<\/a>
\n2. Glossary<\/a>
\n3. Get all posts for a specific category<\/a>
\n4. How to Retrieve Data Using the WordPress API<\/a>  (javascript, jQuery)
\n5. What is an API and Why is it Useful?<\/a>
\n6. Using the WP API to Fetch Posts<\/a>
\n7. urllib Tutorial Python 3<\/a>
\n8. Using Requests in Python<\/a>
\n9.Basic urllib get and post with and without data<\/a>
\n10. Submit WordPress Posts from Front-End with the WP API<\/a> (javascript)<\/p>\n","protected":false},"excerpt":{"rendered":"
In this post we will create python script that is able to get data from WordPress (WP) blog using WP API. This script will save downloaded data into csv file for further analysis or other purposes. WP API is returning data in json format and is accessible through link http:\/\/hostname.com\/wp-json\/wp\/v2\/posts. The WP API is packaged … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[11,10],"tags":[],"jetpack_publicize_connections":[],"yoast_head":"\nRetrieving Post Data Using the WordPress API with Python Script - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Retrieving Post Data Using the WordPress API with Python Script - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"In this post we will create python script that is able to get data from WordPress (WP) blog using WP API. This script will save downloaded data into csv file for further analysis or other purposes. WP API is returning data in json format and is accessible through link http:\/\/hostname.com\/wp-json\/wp\/v2\/posts. The WP API is packaged ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2016-12-31T21:31:50+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-01-02T13:44:49+00:00\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/\",\"name\":\"Retrieving Post Data Using the WordPress API with Python Script - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2016-12-31T21:31:50+00:00\",\"dateModified\":\"2017-01-02T13:44:49+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Retrieving Post Data Using the WordPress API with Python Script\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Retrieving Post Data Using the WordPress API with Python Script - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/","og_locale":"en_US","og_type":"article","og_title":"Retrieving Post Data Using the WordPress API with Python Script - Machine Learning Applications","og_description":"In this post we will create python script that is able to get data from WordPress (WP) blog using WP API. This script will save downloaded data into csv file for further analysis or other purposes. WP API is returning data in json format and is accessible through link http:\/\/hostname.com\/wp-json\/wp\/v2\/posts. The WP API is packaged ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/","og_site_name":"Machine Learning Applications","article_published_time":"2016-12-31T21:31:50+00:00","article_modified_time":"2017-01-02T13:44:49+00:00","author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/","name":"Retrieving Post Data Using the WordPress API with Python Script - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2016-12-31T21:31:50+00:00","dateModified":"2017-01-02T13:44:49+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Retrieving Post Data Using the WordPress API with Python Script"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-cm","jetpack-related-posts":[{"id":827,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/11\/apis\/","url_meta":{"origin":766,"position":0},"title":"Useful APIs for Your Web Site","date":"January 11, 2017","format":false,"excerpt":"Here\u2019s a useful list of resources on how to create an API, compiled from posts that were published recently on this blog. The included APIs can provide a fantastic ways to enhance websites. 1. The WordPress(WP) API exposes a simple yet powerful interface to WP Query, the posts API, post\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":383,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/07\/03\/getting-the-data-from-the-web-using-php-for-api-using-the-api-with-php\/","url_meta":{"origin":766,"position":1},"title":"Getting the Data from the Web using PHP or Python for API","date":"July 3, 2016","format":false,"excerpt":"In the previous posts [1],[2] perl was used to get content from the web through Faroo API and Guardian APIs. In this post PHP and Pyhton will be used to get web data using same APIs. PHP has a powerful JSON parsing mechanism, which, because PHP is a dynamic language,\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"Trend for Python, Perl, PHP","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2016\/07\/trend_for_python_perl_php-300x144.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":313,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/06\/15\/faroo\/","url_meta":{"origin":766,"position":2},"title":"Getting Data from the Web with Perl and The  Guardian API","date":"June 15, 2016","format":false,"excerpt":"In one of previous post the Faroo API was used in order to get data content from the web. In this post we will look at different API that can be also used for downloading content from web. Here we will use the Guardian API \/ open platform. At the\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":220,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/05\/24\/getting-data-from-the-web\/","url_meta":{"origin":766,"position":3},"title":"Getting Data from the Web with Perl and Faroo API","date":"May 24, 2016","format":false,"excerpt":"As stated on Wikipedia \"The number of available web APIs has grown consistently over the past years, as businesses realize the growth opportunities associated with running an open platform, that any developer can interact with.\" [1] For web developers web API (application programming interface) allows to create own application using\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":510,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/08\/19\/getting-data-from-wikipedia-using-python\/","url_meta":{"origin":766,"position":4},"title":"Getting Data From Wikipedia Using Python","date":"August 19, 2016","format":false,"excerpt":"Recently I come across python package Wikipedia which is a Python library that makes it easy to access and parse data from Wikipedia. Using this library you can search Wikipedia, get article summaries, get data like links and images from a page, and more. Wikipedia wraps the MediaWiki API so\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":797,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/","url_meta":{"origin":766,"position":5},"title":"Topic Extraction from Blog Posts with LSI , LDA and Python","date":"January 8, 2017","format":false,"excerpt":"In the previous post we created python script to get posts from Wordpress (WP) blog through WP API. This script was saving retrieved posts into csv file. In this post we will create script for topic extraction from the posts saved in this csv file. We will use the following\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/766"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=766"}],"version-history":[{"count":16,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/766\/revisions"}],"predecessor-version":[{"id":795,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/766\/revisions\/795"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=766"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=766"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=766"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}