{"id":1356,"date":"2017-07-21T23:41:18","date_gmt":"2017-07-21T23:41:18","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=1356"},"modified":"2017-07-23T21:29:29","modified_gmt":"2017-07-23T21:29:29","slug":"application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/","title":{"rendered":"Application for Machine Learning for Analyzing Blog Text and Google Analytics Data"},"content":{"rendered":"

In the previous post we looked how to download data from WordPress blog. [1] So now we can have blog data. We can get also web metrics data from Google Analytics such us the number of views, time on the page. How do we connect post text data with metrics data to see how different topics\/keywords correlate with different metrics data? Or may be we want to know what terms contribute to higher time on page or number of views?<\/p>\n

Here is the experiment that we can do to check how we can combine blog post text data with web metrics. I downloaded data from blog and saved in the csv file. This is actually same file that was obtained in [1].<\/p>\n

\"\"<\/p>\n

In this file time on page from Google Analytics was added manually as additional column. The python program was created. In the program the numeric value in sec is converted in two labels 0 and 1 where 0 is assigned if time less than 120 sec, otherwise 1 is assigned.<\/p>\n

\r\nThen machine learning was applied as below:\r\n   for each label\r\n            load the post data that have this label from file\r\n            apply TfidfVectorizer\r\n            cluster data\r\n            save data in dataframe\r\n    print dataframe\r\n<\/code><\/pre>\n
So the dataframe will show distribution of keywords for groups of posts with different time on page.
\nThis is useful if we are interesting why some posts doing well and some not.<\/p>\n
Below is sample output and source code:<\/p>\n
\"\"<\/p>\n
\r\n# -*- coding: utf-8 -*-\r\n\r\nfrom sklearn.cluster import KMeans\r\nfrom sklearn.feature_extraction.text import TfidfVectorizer\r\nimport numpy as np\r\nimport pandas as pd\r\n\r\npd.set_option('max_columns', 50)\r\n\r\n#only considers the top n words ordered by term frequency\r\nn_features=250\r\nuse_idf=True\r\nnumber_of_runs = 3\r\n\r\nimport csv\r\nimport re\r\n\r\ndef remove_html_tags(text):\r\n        \"\"\"Remove html tags from a string\"\"\"\r\n        clean = re.compile('<.*?>')\r\n        return re.sub(clean, '', text)\r\n\r\n\r\n\r\n\r\nfn=\"posts.csv\" \r\nlabelsY=[0,1]\r\nk=3\r\n\r\nexclude_words=['row', 'rows', 'print', 'new', 'value', 'column', 'count', 'page', 'short', 'means', 'newline', 'file', 'results']\r\ncolumns = ['Low Average Time on Page', 'High Average Time on Page']\r\nindex = np.arange(50) # array of numbers for the number of samples\r\ndf = pd.DataFrame(columns=columns , index = index)\r\n\r\nfor z in range(len(labelsY)):\r\n\r\n    doc_set = []\r\n  \r\n    with open(fn, encoding=\"utf8\" ) as f:\r\n                csv_f = csv.reader(f)\r\n                for i, row in enumerate(csv_f):\r\n                   if i > 1 and len(row) > 1 :\r\n                       include_this = False\r\n                       if  labelsY[z] ==0:\r\n                           if (int(row[3])) < 120 :\r\n                               include_this=True\r\n                       if  labelsY[z] ==1:    \r\n                            if (int(row[3])) >= 120 :\r\n                               include_this=True\r\n                               \r\n                       if  include_this:       \r\n                             temp=remove_html_tags(row[1])\r\n                             temp=row[0] + \" \" + temp \r\n                             temp = re.sub(\"[^a-zA-Z ]\",\"\", temp)\r\n                             \r\n                             for word in exclude_words:\r\n                               if word in temp:        \r\n                                        temp=temp.replace(word,\"\")\r\n                             doc_set.append(temp)\r\n                             \r\n    \r\n    vectorizer = TfidfVectorizer(max_df=0.5, max_features=n_features,\r\n                                         min_df=2, stop_words='english',\r\n                                         use_idf=use_idf)\r\n            \r\n   \r\n    X = vectorizer.fit_transform(doc_set)\r\n    print(\"n_samples: %d, n_features: %d\" % X.shape)\r\n    \r\n    km = KMeans(n_clusters=k, init='k-means++', max_iter=100, n_init=1)\r\n    km.fit(X)\r\n    order_centroids = km.cluster_centers_.argsort()[:, ::-1]\r\n    terms = vectorizer.get_feature_names()\r\n    count=0\r\n    for i in range(k):\r\n          print(\"Cluster %d:\" % i, end='')\r\n          for ind in order_centroids[i, :10]:\r\n                   print(' %s' % terms[ind], end='')\r\n                   df.set_value(count, columns[z], terms[ind])\r\n                   count=count+1\r\n\r\nprint (\"\\n\")\r\nprint (df)\r\n<\/code><\/pre>\n
References<\/strong><\/p>\n
1. Retrieving Post Data Using the WordPress API with Python Script<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
In the previous post we looked how to download data from WordPress blog. [1] So now we can have blog data. We can get also web metrics data from Google Analytics such us the number of views, time on the page. How do we connect post text data with metrics data to see how different … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[5,2,9,10],"tags":[19,23,18,22],"jetpack_publicize_connections":[],"yoast_head":"\nApplication for Machine Learning for Analyzing Blog Text and Google Analytics Data - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Application for Machine Learning for Analyzing Blog Text and Google Analytics Data - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"In the previous post we looked how to download data from WordPress blog. [1] So now we can have blog data. We can get also web metrics data from Google Analytics such us the number of views, time on the page. How do we connect post text data with metrics data to see how different ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2017-07-21T23:41:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-07-23T21:29:29+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/07\/Blog-Analytics-300x91.png\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/\",\"name\":\"Application for Machine Learning for Analyzing Blog Text and Google Analytics Data - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2017-07-21T23:41:18+00:00\",\"dateModified\":\"2017-07-23T21:29:29+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Application for Machine Learning for Analyzing Blog Text and Google Analytics Data\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Application for Machine Learning for Analyzing Blog Text and Google Analytics Data - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/","og_locale":"en_US","og_type":"article","og_title":"Application for Machine Learning for Analyzing Blog Text and Google Analytics Data - Machine Learning Applications","og_description":"In the previous post we looked how to download data from WordPress blog. [1] So now we can have blog data. We can get also web metrics data from Google Analytics such us the number of views, time on the page. How do we connect post text data with metrics data to see how different ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/","og_site_name":"Machine Learning Applications","article_published_time":"2017-07-21T23:41:18+00:00","article_modified_time":"2017-07-23T21:29:29+00:00","og_image":[{"url":"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/07\/Blog-Analytics-300x91.png"}],"author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/","name":"Application for Machine Learning for Analyzing Blog Text and Google Analytics Data - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2017-07-21T23:41:18+00:00","dateModified":"2017-07-23T21:29:29+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/07\/21\/application-for-machine-learning-for-analyzing-blog-text-and-google-analytics-data\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Application for Machine Learning for Analyzing Blog Text and Google Analytics Data"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-lS","jetpack-related-posts":[{"id":1446,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/11\/06\/10-new-top-resources-on-machine-learning-from-around-the-web\/","url_meta":{"origin":1356,"position":0},"title":"10 New Top Resources on Machine Learning from Around the Web","date":"November 6, 2017","format":false,"excerpt":"For this post I put new and most interesting machine learning resources that I recently found on the web. This is the list of useful resources in such areas like stock market forecasting, text mining, deep learning, neural networks and getting data from Twitter. Hope you enjoy the reading. 1.\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1131,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/04\/18\/extracting-google-adsense-and-google-analytics-data-for-website-analytics\/","url_meta":{"origin":1356,"position":1},"title":"Extracting Google AdSense and Google Analytics Data for Website Analytics","date":"April 18, 2017","format":false,"excerpt":"Recently I decided to get information that is showing for each page of my website Google Analytics account number and all Google AdSense links on this page. Connecting this information with Google Publisher Pages data would be very useful for better analysis and understanding of ads performance. So I created\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/04\/extractingGAGS_info-130x300.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":133,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/03\/11\/133\/","url_meta":{"origin":1356,"position":2},"title":"7 Ideas for Building Text Mining Application","date":"March 11, 2016","format":false,"excerpt":"It is no doubt that the web is growing at an incredible pace. And as the most documents of the web consist of the text, the applications of text analytics or text mining are getting more use. In such applications the textual data are used for extracting intelligence from a\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1470,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/11\/13\/getting-data-driven-insights-from-blog-data-analysis-with-feature-selection\/","url_meta":{"origin":1356,"position":3},"title":"Getting Data-Driven Insights from Blog Data Analysis with Feature Selection","date":"November 13, 2017","format":false,"excerpt":"Machine learning algorithms are widely used in every business - object recognition, marketing analytics, analyzing data in numerous applications to get useful insights. In this post one of machine learning techniques is applied to analysis of blog post data to predict significant features for key metrics such as page views.\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/11\/feature_selection-300x253.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":65,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/02\/09\/cloud-computing\/","url_meta":{"origin":1356,"position":4},"title":"Cloud Computing","date":"February 9, 2016","format":false,"excerpt":"Cloud computing is the use of computing resources (hardware and software) that are delivered as a service over a network (typically the Internet). The name comes from the use of a cloud-shaped symbol as an abstraction for the complex infrastructure it contains in system diagrams. Cloud computing entrusts remote services\u2026","rel":"","context":"Similar post","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1385,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/10\/15\/scraping\/","url_meta":{"origin":1356,"position":5},"title":"Combining Machine Learning and Data Scraping","date":"October 15, 2017","format":false,"excerpt":"I often come across web posts about extracting data (data scraping) from websites. For example recently in [1] Scrapy tool was used for web scraping with Python. Once we get scraping data we can use extracted information in many different ways. As computer algorithms evolve and can do more, the\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1356"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=1356"}],"version-history":[{"count":16,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1356\/revisions"}],"predecessor-version":[{"id":1378,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1356\/revisions\/1378"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=1356"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=1356"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=1356"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}