{"id":797,"date":"2017-01-08T13:34:01","date_gmt":"2017-01-08T13:34:01","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=797"},"modified":"2018-09-09T00:11:55","modified_gmt":"2018-09-09T00:11:55","slug":"topic-extraction-from-blog-posts-with-lsi-and-lda-and-python","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/","title":{"rendered":"Topic Extraction from Blog Posts with LSI , LDA and Python"},"content":{"rendered":"

In the previous post we created python script to get posts from WordPress (WP) blog through WP API. This script was saving retrieved posts into csv file. In this post we will create script for topic extraction from the posts saved in this csv file. We will use the following 2 techniques (LSI and LDA) for topic modeling:<\/p>\n

1. Latent semantic indexing (LSI) is an indexing and retrieval method that uses a mathematical technique called singular value decomposition (SVD) to identify patterns in the relationships between the terms and concepts contained in an unstructured collection of text. LSI is based on the principle that words that are used in the same contexts tend to have similar meanings. A key feature of LSI is its ability to extract the conceptual content of a body of text by establishing associations between those terms that occur in similar contexts.[1]<\/p>\n

2. Latent Dirichlet allocation (LDA) is a generative statistical model that allows sets of observations to be explained by unobserved groups that explain why some parts of the data are similar. For example, if observations are words collected into documents, it posits that each document is a mixture of a small number of topics and that each word’s creation is attributable to one of the document’s topics. LDA is an example of a topic model. [2] <\/p>\n

In one of the previous posts we looked how to use LDA with python. [8] So now we just applying this script to data in csv file with blog posts. Additionally we will use LSI method as alternative for topic modeling.<\/p>\n

The script for LDA\/LSA consists of the following parts:
\n1. As the first step the script is opens csv data file and load data to the memory. During this the script also performs some text preprocessing. As result of this we have the set of posts (documents).
\n2. The script iterates through set of posts and converts documents into tokens and saves all documents into texts. After iteration is completed the script builds dictionary and corpus.
\n3. In this step the script uses LSI model and LDA model.
\n4. Finally in the end for LDA method the script prints some information about topics, including document – topic information.<\/p>\n

Comparing results of LSI and LDA methods it seems that LDA gives more understable topics.
\nAlso LDA coefficients are all in range 0 – 1 as they indicate probabilities. This makes easy to explain results.<\/p>\n

In our script we used LDA and LSI from gensim library, but there are another packages that allows to do LDA:
\nMALLET for example allows also to model a corpus of texts [4]
\nLDA – another python package for Latent Dirichlet Allocation [5]<\/p>\n

There are also other techniques to do approximate topic modeling in Python. For example there is a technique called non-negative matrix factorization (NMF) that strongly resembles Latent Dirichlet Allocation [3] Also there is a probabilistic latent semantic analysis (PLSA), also known as probabilistic latent semantic indexing (PLSI) technique that evolved from LSA. [9]<\/p>\n

In the future posts some of the above methods will be considered.<\/p>\n

There is an interesting discussion on quora site how to run LDA and here you can find also some insights on how to prepare data and how to evaluate results of LDA. [6]<\/p>\n

Here is the source code of script. <\/p>\n

\r\n# -*- coding: utf-8 -*-\r\n\r\nimport csv\r\nfrom stop_words import get_stop_words\r\nfrom nltk.stem.porter import PorterStemmer\r\nfrom gensim import corpora\r\nimport gensim\r\n\r\nimport re\r\nfrom nltk.tokenize import RegexpTokenizer\r\n\r\nM=\"LDA\"\r\n\r\ndef remove_html_tags(text):\r\n        \"\"\"Remove html tags from a string\"\"\"\r\n     \r\n        clean = re.compile('<.*?>')\r\n        return re.sub(clean, '', text)\r\n        \r\n\r\n\r\ntokenizer = RegexpTokenizer(r'\\w+')\r\n\r\n# use English stop words list\r\nen_stop = get_stop_words('en')\r\n\r\n# use p_stemmer of class PorterStemmer\r\np_stemmer = PorterStemmer()\r\n\r\nfn=\"posts.csv\" \r\ndoc_set = []\r\n\r\nwith open(fn, encoding=\"utf8\" ) as f:\r\n            csv_f = csv.reader(f)\r\n            for i, row in enumerate(csv_f):\r\n               if i > 1 and len(row) > 1 :\r\n                \r\n                \r\n                 temp=remove_html_tags(row[1]) \r\n                 temp = re.sub(\"[^a-zA-Z ]\",\"\", temp)\r\n                 doc_set.append(temp)\r\n                 \r\ntexts = []\r\n\r\nfor i in doc_set:\r\n    print (i)\r\n    # clean and tokenize document string\r\n    raw = i.lower()\r\n    raw=' '.join(word for word in raw.split() if len(word)>2)    \r\n       \r\n    raw=raw.replace(\"nbsp\", \"\")\r\n    tokens = tokenizer.tokenize(raw)\r\n       \r\n    stopped_tokens = [i for i in tokens if not i in en_stop]\r\n    stemmed_tokens = [p_stemmer.stem(i) for i in stopped_tokens]\r\n \r\n    texts.append(stemmed_tokens)\r\n\r\n# turn our tokenized documents into a id <-> term dictionary\r\ndictionary = corpora.Dictionary(texts)\r\n# convert tokenized documents into a document-term matrix\r\ncorpus = [dictionary.doc2bow(text) for text in texts]\r\n\r\nlsi = gensim.models.lsimodel.LsiModel(corpus, id2word=dictionary, num_topics=5  )\r\nprint (lsi.print_topics(num_topics=3, num_words=3))\r\n\r\nfor i in  lsi.show_topics(num_words=4):\r\n    print (i[0], i[1])\r\n\r\nif M==\"LDA\":\r\n ldamodel = gensim.models.ldamodel.LdaModel(corpus, num_topics=5, id2word = dictionary, passes=20)\r\n print (ldamodel)\r\n print(ldamodel.print_topics(num_topics=3, num_words=3))\r\n for i in  ldamodel.show_topics(num_words=4):\r\n    print (i[0], i[1])\r\n\r\n # Get Per-topic word probability matrix:\r\n K = ldamodel.num_topics\r\n topicWordProbMat = ldamodel.print_topics(K)\r\n print (topicWordProbMat) \r\n \r\n for t in texts:\r\n     vec = dictionary.doc2bow(t)\r\n     print (ldamodel[vec])\r\n<\/code><\/pre>\n
References<\/strong>
\n1. Latent_semantic_analysis<\/a>
\n2. Latent_Dirichlet_allocation<\/a>
\n3. Topic modeling in Python<\/a>
\n4. Topic modeling with MALLET<\/a>
\n5. Getting started with Latent Dirichlet Allocation in Python<\/a>
\n6. What are good ways of evaluating the topics generated by running LDA on a corpus?<\/a>
\n7. Working with text<\/a>
\n8. Latent Dirichlet Allocation (LDA) with Python Script<\/a>
\n9. Probabilistic latent semantic analysis<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
In the previous post we created python script to get posts from WordPress (WP) blog through WP API. This script was saving retrieved posts into csv file. In this post we will create script for topic extraction from the posts saved in this csv file. We will use the following 2 techniques (LSI and LDA) … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[9,10],"tags":[29,93,94],"jetpack_publicize_connections":[],"yoast_head":"\nTopic Extraction from Blog Posts with LSI , LDA and Python - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Topic Extraction from Blog Posts with LSI , LDA and Python - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"In the previous post we created python script to get posts from WordPress (WP) blog through WP API. This script was saving retrieved posts into csv file. In this post we will create script for topic extraction from the posts saved in this csv file. We will use the following 2 techniques (LSI and LDA) ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2017-01-08T13:34:01+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-09-09T00:11:55+00:00\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/\",\"name\":\"Topic Extraction from Blog Posts with LSI , LDA and Python - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2017-01-08T13:34:01+00:00\",\"dateModified\":\"2018-09-09T00:11:55+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Topic Extraction from Blog Posts with LSI , LDA and Python\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Topic Extraction from Blog Posts with LSI , LDA and Python - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/","og_locale":"en_US","og_type":"article","og_title":"Topic Extraction from Blog Posts with LSI , LDA and Python - Machine Learning Applications","og_description":"In the previous post we created python script to get posts from WordPress (WP) blog through WP API. This script was saving retrieved posts into csv file. In this post we will create script for topic extraction from the posts saved in this csv file. We will use the following 2 techniques (LSI and LDA) ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/","og_site_name":"Machine Learning Applications","article_published_time":"2017-01-08T13:34:01+00:00","article_modified_time":"2018-09-09T00:11:55+00:00","author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/","name":"Topic Extraction from Blog Posts with LSI , LDA and Python - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2017-01-08T13:34:01+00:00","dateModified":"2018-09-09T00:11:55+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/08\/topic-extraction-from-blog-posts-with-lsi-and-lda-and-python\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Topic Extraction from Blog Posts with LSI , LDA and Python"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-cR","jetpack-related-posts":[{"id":852,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/22\/data-visualization-visualizing-an-lda-model-using-python\/","url_meta":{"origin":797,"position":0},"title":"Data Visualization – Visualizing an LDA Model using Python","date":"January 22, 2017","format":false,"excerpt":"In the previous post Topic Extraction from Blog Posts with LSI , LDA and Python python code was created for text documents topic modeling using Latent Dirichlet allocation (LDA) method. The output was just an overview of the words with corresponding probability distribution for each topic and it was hard\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/01\/word_topic_dataframe-300x112.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":721,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/15\/latent-dirichlet-allocation-lda-with-python\/","url_meta":{"origin":797,"position":1},"title":"Latent Dirichlet Allocation (LDA) with Python Script","date":"December 15, 2016","format":false,"excerpt":"In the previous posts [1],[2] few scripts for extracting web data were created. Combining these scripts, we will create now web crawling script with text mining functionality such as Latent Dirichlet Allocation (LDA). In LDA, each document may be viewed as a mixture of various topics. Where each document is\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"Program Flow Chart for Extracting Data from Web and Doing LDA","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2016\/12\/program-flow-300x247.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":766,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/12\/31\/retrieving-post-data-using-the-wordpress-api-with-python-script\/","url_meta":{"origin":797,"position":2},"title":"Retrieving Post Data Using the WordPress API with Python Script","date":"December 31, 2016","format":false,"excerpt":"In this post we will create python script that is able to get data from WordPress (WP) blog using WP API. This script will save downloaded data into csv file for further analysis or other purposes. WP API is returning data in json format and is accessible through link http:\/\/hostname.com\/wp-json\/wp\/v2\/posts.\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":678,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/11\/19\/web-content-extraction-is-now-easier-than-ever-using-python-scripting\/","url_meta":{"origin":797,"position":3},"title":"Web Content Extraction is Now Easier  than Ever Using Python Scripting","date":"November 19, 2016","format":false,"excerpt":"As more and more Web content is created, there is a need for simple and efficient Web data extraction tools or scripts. With some recently released python libraries Web content extraction is now easier than ever. One example of such python library package is newspaper [1]. This module can do\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":431,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/07\/30\/automating-csv-file-reading-and-writing-with-python\/","url_meta":{"origin":797,"position":4},"title":"Automating CSV File Reading and Writing with Python","date":"July 30, 2016","format":false,"excerpt":"Python is widely used programming language in many fields, such as data science, machine learning, web development. It is also a great choice for automation of different computer tasks such as data downloading from websites or data manipulation. Python can open, read or save data in different file formats. In\u2026","rel":"","context":"In "Python Scripts"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":827,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/01\/11\/apis\/","url_meta":{"origin":797,"position":5},"title":"Useful APIs for Your Web Site","date":"January 11, 2017","format":false,"excerpt":"Here\u2019s a useful list of resources on how to create an API, compiled from posts that were published recently on this blog. The included APIs can provide a fantastic ways to enhance websites. 1. The WordPress(WP) API exposes a simple yet powerful interface to WP Query, the posts API, post\u2026","rel":"","context":"In "API Programming"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/797"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=797"}],"version-history":[{"count":22,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/797\/revisions"}],"predecessor-version":[{"id":823,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/797\/revisions\/823"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=797"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=797"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=797"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}