Data Mining Twitter Data with Python

Twitter is an online social networking service that enables users to send and read short 140-character messages called “tweets”. [1]
Twitter users are tweeting about different topics based on their interests and goals.
A word, phrase or topic that is mentioned at a greater rate than others is said to be a “trending topic”. Trending topics become popular either through a concerted effort by users, or because of an event that prompts people to talk about a specific topic. [1]
There is wide interest in analyzing of trending data from Twitter.
And in this post we will look at searching and downloading the tweets related to specific hashtag. We will use Python and Twitter API. Our example will be search tweets related to “deep learning”. After downloading Twitter data we will also look at some data manipulations with the data.

The example of downloading of Twitter data is based on the work [2]
Below is source code:

import twitter
import json
CONSUMER_KEY =”xxxxxx”
OAUTH_TOKEN = “xxxxxx”
twitter_api= twitter.Twitter(auth=auth)
q=’#deep learning’
search_results = (q=q, count=count)
for _ in range(5):
    print “Length of statuses”, len(statuses)
    except KeyError, e: #result does not exist
    kwargs=dict( [kv.split(‘=’) for kv in next_results[1:].split(“&”)])
    search_results =**kwargs)
    statuses += search_results[‘statuses’]
# Show one sample search result by slicing the list
print json.dumps(statuses[0], indent=10)
hashtags = [ hashtag[‘text’]
    for status in statuses
        for hashtag in status[‘entities’][‘hashtags’] ]
urls = [ urls[‘url’]
    for status in statuses
        for urls in status[‘entities’][‘urls’] ]
texts = [ status[‘text’]
    for status in statuses
#Created_at is date time when created
created_ats = [ status[‘created_at’]
    for status in statuses
print json.dumps(hashtags[0:50], indent=1)
print json.dumps(urls[0:50], indent=1)
print json.dumps(texts[0:50], indent=1)
print json.dumps(created_ats[0:50], indent=1)
# Now we append some data into the file
with open(“data.txt”, “a”) as myfile:
    for w in hashtags:         myfile.write(w)
# count of word frequencies
wordcounts = {}
for term in hashtags:
    wordcounts[term] = wordcounts.get(term, 0) + 1
items = [(v, k) for k, v in wordcounts.items()]
for count, word in sorted(items, reverse=True):
    print(“%5d %s” % (count, word))
# in case we need extract date or month or year
for x in created_ats:
    print x
    print x[4:10]
    print x[26:31]
    print x[4:7]

Output example for last for loop (just one cycle)
Wed Mar 30 02:10:20 +0000 2016
Mar 30

Any comments or suggestions are welcome.

[1] Twitter, From Wikipedia
TWITTER by Abhishanga Upadhyay, Luis Mao, Malavika Goda Krishna

7 Ideas for Building Text Mining Application

It is no doubt that the web is growing at an incredible pace. And as the most documents of the web consist of the text, the applications of text analytics or text mining are getting more use. In such applications the textual data are used for extracting intelligence from a large collection of documents. Here are 7 ideas for building this type of applications. Later on during this year 2016 some online working demo examples on this site will be built to test the ideas. The focus is on applications for personal use. Business applications of text mining can be found in [1]

1. Trending is collecting historical data in order to find pattern or predict future. If the usage of word phrase “python programming” is going up from month to month then it is good signal for paying attention to this. There is the tool Google Trends is a public web facility of Google Inc., based on Google Search, that shows how often a particular search-term is entered relative to the total search-volume across various regions of the world, and in various languages. The horizontal axis of the main graph represents time (starting from 2004), and the vertical is how often a term is searched for relative to the total number of searches, globally. [2]

However what if we more focused on the future and we want know what terms will be popular in the future. For example “data science” or “big data” are now popular search terms but back to the time when the usage was the lowest – if the tool can predict high increase of usage in such situation – that would be very useful.

2. Building news feed is another example of application for text mining. Newly published web based content if it matches the user interest has a great value. So the application should allow for user to set the topics for the desired content.
For the same topic the user might be interesting in the ability to set other filters such as source of content, type of content and some other characteristics of content.
Over the time the user interests will change and so the application should learn and adapt to user interests too.

3.Post editing. While someone online is typing article for blog or paper there is always the need to find something related to topic on the web. Imagine that it will automatically show an additional text box with similar content from the web. This would eliminate switching back to search engine and also can bring something new that the author even did not think about.

4. Automatic creation of data reports is very helpful for people who needs data for their research or business needs. There a lot of data and information freely available on the web however it is tedious go to online on weekly or monthly basis and manually extract data and put in some file or database for further analysis. Such task often will include such things as formatting data, merging the information from different sources and some other processing operations.

5. Sentiment analysis (also known as opinion mining) refers to the use of natural language processing, text analysis and computational linguistics to identify and extract subjective information in source materials. Sentiment analysis is widely applied to reviews and social media for a variety of applications, ranging from marketing to customer service. [3]
Almost each online text analytics web based service is offering sentiment analysis option for users. There are also many online examples how to do sentiment analysis using python or R as programming language. [4]
Sentiment analysis application would be useful for predicting some financial event or collecting opinion about some product.

6. Content Organizing application can organize documents into groups by topic, keywords or by some other means.
We can subscribe and receive email notifications about site news, latest post or new article. The application that is saving links that we liked or decided to review later can help us be more productive. In addition to links it would save some information about links like keywords, topics or description. Such application could also group information by topic or keyword and automatically assign additional keywords.
Text document clustering and classification would be used a lot for this application.

7. Topic detection application can be used for automatic text categorization, for understanding what people are talking about, for automatic processing or preprocessing emails or user submitted online articles, comments.
The task of topic detection might also require the
development of approaches related to the presentation
of topics: topic ranking, relevant image retrieval, title
and keyword extraction. One of the example of using topic detection is shown in [5]
Obviously one document can consist of several segments on different topics. In one of researches simple clustering algorithm was used to group the semantically-related sentences. The distance between two sentences was calculated based on the distance between
all nouns that appear in the sentences. The distance between two nouns was calculated using the Wordnet thesaurus. [6],[7]


1. Text Mining and its Business Applications

2. Google Trends
From Wikipedia

3. Sentiment analysis
From Wikipedia

Analysis of Unstructured Data: Applications of Text Analytics
and Sentiment Mining, Dr. Goutam Chakraborty, Murali Krishna Pagolu

Two-level message clustering for topic detection in
Twitter. Georgios Petkos ,Symeon Papadopoulos, Yiannis Kompatsiaris

6. A Non-Linear Topic Detection Method for Text
Summarization Using Wordnet

Click to access Wksp-Tec-Info-Ling-Silla-2003.pdf

Carlos N. Silla Jr. , Celso A. A. Kaestner, Alex A. Freitas

Topic Detection by Clustering Keywords. Christian Wartena and Rogier Brussee