First the script is establishing connection to Twitter and credentials are being checked by Twitter service. This requires to provide access tokens such as CONSUMER_KEY, CONSUMER_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET. Refer to [1] how to obtain this information from Twitter account.<\/p>\n
2. Searching for Tweets<\/strong><\/p>\n Once access token information verified then the search for tweets related to a\tparticular hashtag “deep learning” in our example is performed and if it is successful we are getting data. The python script then iterates through 5 more batches of results by following the cursor. All results are saved in json data structure statuses<\/em>.<\/p>\n Now we are extracting data such as hashtags, urls, texts and created at date. The date is useful if we need do trending over the time.<\/p>\n In the next step we are preparing data for trending in the format: date word. This allows to view how the usage of specific word in the tweets is changing over the time. 3. Clustering Tweets<\/strong><\/p>\n Now we are preparing tweets data for data clustering. We are converting text data into bag of words data representation. This is called vectorization which is the general process of turning a collection of text documents into numerical feature vectors. [2]<\/p>\n In the next step we output some data and build plot of frequency for hashtags.<\/p>\n Source Code<\/strong> References<\/strong> 2. Feature extraction<\/a> scikit-learn Documentation, Machine Learning in Python<\/p>\n 3. Clustering – Birch<\/a> scikit-learn Documentation, Machine Learning in Python<\/p>\n
\nHere is code example of getting urls and date data:<\/p>\nurls = [ urls['url']\r\n for status in statuses\r\n for urls in status['entities']['urls'] ]\r\n\r\n\r\ncreated_ats = [ status['created_at']\r\n for status in statuses\r\n ]\r\n<\/code><\/pre>\n\r\nmodelvectorizer = CountVectorizer(analyzer = \"word\", \\\r\n tokenizer = None, \\\r\n preprocessor = None, \\ \r\n stop_words='english', \\\r\n max_features = 5000) \r\n\r\ntrain_data_features = vectorizer.fit_transform(texts)\r\ntrain_data_features = train_data_features.toarray()\r\nprint (train_data_features.shape)\r\nprint (train_data_features)\r\n'''\r\nThis will print like this: \r\n[[0 0 0 ..., 0 1 1]\r\n [0 0 1 ..., 0 0 0]\r\n [0 0 0 ..., 0 1 1]\r\n ..., \r\n [0 0 0 ..., 0 0 0]\r\n [0 0 0 ..., 0 0 0]\r\n [0 0 0 ..., 0 0 0]]\r\n'''\r\n\r\nvocab = vectorizer.get_feature_names()\r\nprint (vocab)\r\ndist = np.sum(train_data_features, axis=0)\r\n\r\n#For each, print the vocabulary word and the number of times it appears in the training set\r\n\r\nfor tag, count in zip(vocab, dist):\r\n print (count, tag)\r\n\r\n'''\r\nThis will print something like this\r\n3 ai\r\n1 alexandria\r\n2 algorithms\r\n1 amp\r\n2 analytics\r\n1 applications\r\n1 applied\r\n''''\r\n\r\nNow we are ready to do clustering. We select to use Birch clustering algorithm. [3] Below is the code snippet for this. We specify the number of clusters 6.\r\n\r\nbrc = Birch(branching_factor=50, n_clusters=6, threshold=0.5, compute_labels=True)\r\nbrc.fit(train_data_features)\r\n\r\nclustering_result=brc.predict(train_data_features)\r\nprint (\"\\nClustering_result:\\n\")\r\nprint (clustering_result)\r\n\r\n'''\r\nBelow is the example of printout (each tweet got the number, this number represents the number of cluster associated with this tweet, number of clusters is 6 ):\r\nClustering_result:\r\n\r\n[0 0 0 0 0 4 0 0 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 5 0 0 0 1 4 1 1 1\r\n 2 2]\r\n'''\r\n<\/code><\/pre>\n![\"Frequency](\"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2016\/05\/Frequency-of-Hashtags-300x171.png\")
<\/p>\n
\nThus we explored python coding of data mining for Twitter. We looked at different tasks such as searching tweets, extracting different data from search results, preparing data for trending, converting text results into numerical form, clustering and printing plot of frequency of hashtags.
\nBelow is the source code for all of this. In the future we plan add more functionality. There many possible ways how to data mine Twitter data. Some interesting ideas on the web can be found in [4]. <\/p>\n\r\nimport twitter\r\nimport json\r\n\r\n\r\n\r\n\r\nimport matplotlib.pyplot as plt\r\nimport numpy as np\r\n\r\n\r\n\r\nfrom sklearn.feature_extraction.text import CountVectorizer\r\nfrom sklearn.cluster import Birch\r\n\r\nCONSUMER_KEY =\"xxxxxxxxxxxxxxx\"\r\nCONSUMER_SECRET =\"xxxxxxxxxxxx\"\r\nOAUTH_TOKEN = \"xxxxxxxxxxxxxx\"\r\nOAUTH_TOKEN_SECRET = \"xxxxxxxxxx\"\r\n\r\n\r\nauth = twitter.oauth.OAuth (OAUTH_TOKEN, OAUTH_TOKEN_SECRET, CONSUMER_KEY, CONSUMER_SECRET)\r\n\r\ntwitter_api= twitter.Twitter(auth=auth)\r\nq='#deep learning'\r\ncount=100\r\n\r\n# Do search for tweets containing '#deep learning'\r\nsearch_results = twitter_api.search.tweets (q=q, count=count)\r\n\r\nstatuses=search_results['statuses']\r\n\r\n# Iterate through 5 more batches of results by following the cursor\r\nfor _ in range(5):\r\n print (\"Length of statuses\", len(statuses))\r\n try:\r\n next_results = search_results['search_metadata']['next_results']\r\n except KeyError: \r\n break\r\n # Create a dictionary from next_results\r\n kwargs=dict( [kv.split('=') for kv in next_results[1:].split(\"&\") ])\r\n\r\n search_results = twitter_api.search.tweets(**kwargs)\r\n statuses += search_results['statuses']\r\n\r\n# Show one sample search result by slicing the list\r\nprint (json.dumps(statuses[0], indent=10))\r\n\r\n\r\n\r\n# Extracting data such as hashtags, urls, texts and created at date\r\nhashtags = [ hashtag['text'].lower()\r\n for status in statuses\r\n for hashtag in status['entities']['hashtags'] ]\r\n\r\n\r\nurls = [ urls['url']\r\n for status in statuses\r\n for urls in status['entities']['urls'] ]\r\n\r\n\r\ntexts = [ status['text']\r\n for status in statuses\r\n ]\r\n\r\ncreated_ats = [ status['created_at']\r\n for status in statuses\r\n ]\r\n\r\n# Preparing data for trending in the format: date word\r\n# Note: in the below loop w is not cleaned from #,? characters\r\ni=0\r\nprint (\"===============================\\n\")\r\nfor x in created_ats:\r\n for w in texts[i].split(\" \"):\r\n if len(w)>=2:\r\n print (x[4:10], x[26:31] ,\" \", w)\r\n i=i+1\r\n\r\n\r\n\r\n\r\n# Prepare tweets data for clustering\r\n# Converting text data into bag of words model\r\n\r\nvectorizer = CountVectorizer(analyzer = \"word\", \\\r\n tokenizer = None, \\\r\n preprocessor = None, \\\r\n stop_words='english', \\\r\n max_features = 5000) \r\n\r\ntrain_data_features = vectorizer.fit_transform(texts)\r\n\r\ntrain_data_features = train_data_features.toarray()\r\n\r\nprint (train_data_features.shape)\r\n\r\nprint (train_data_features)\r\n\r\nvocab = vectorizer.get_feature_names()\r\nprint (vocab)\r\n\r\ndist = np.sum(train_data_features, axis=0)\r\n\r\n# For each, print the vocabulary word and the number of times it \r\n# appears in the training set\r\nfor tag, count in zip(vocab, dist):\r\n print (count, tag)\r\n\r\n\r\n# Clustering data\r\n\r\nbrc = Birch(branching_factor=50, n_clusters=6, threshold=0.5, compute_labels=True)\r\nbrc.fit(train_data_features)\r\n\r\nclustering_result=brc.predict(train_data_features)\r\nprint (\"\\nClustering_result:\\n\")\r\nprint (clustering_result)\r\n\r\n\r\n\r\n\r\n\r\n# Outputting some data\r\nprint (json.dumps(hashtags[0:50], indent=1))\r\nprint (json.dumps(urls[0:50], indent=1))\r\nprint (json.dumps(texts[0:50], indent=1))\r\nprint (json.dumps(created_ats[0:50], indent=1))\r\n\r\n\r\nwith open(\"data.txt\", \"a\") as myfile:\r\n for w in hashtags: \r\n myfile.write(str(w.encode('ascii', 'ignore')))\r\n myfile.write(\"\\n\")\r\n\r\n\r\n\r\n# count of word frequencies\r\nwordcounts = {}\r\nfor term in hashtags:\r\n wordcounts[term] = wordcounts.get(term, 0) + 1\r\n\r\n\r\nitems = [(v, k) for k, v in wordcounts.items()]\r\n\r\n\r\n\r\nprint (len(items))\r\n\r\nxnum=[i for i in range(len(items))]\r\nfor count, word in sorted(items, reverse=True):\r\n print(\"%5d %s\" % (count, word))\r\n \r\n\r\n\r\n\r\nfor x in created_ats:\r\n print (x)\r\n print (x[4:10])\r\n print (x[26:31])\r\n print (x[4:7])\r\n\r\n\r\n\r\nplt.figure()\r\nplt.title(\"Frequency of Hashtags\")\r\n\r\nmyarray = np.array(sorted(items, reverse=True))\r\n\r\n\r\nprint (myarray[:,0])\r\n\r\nprint (myarray[:,1])\r\n\r\nplt.xticks(xnum, myarray[:,1],rotation='vertical')\r\nplt.plot (xnum, myarray[:,0])\r\nplt.show()\r\n\r\n<\/code><\/pre>\n
\n1. MINING DATA FROM TWITTER<\/a>
\nAbhishanga Upadhyay, Luis Mao, Malavika\tGoda Krishna<\/p>\n