Month: September 2016

How Can We Use Computer Programming to Increase Effective Thinking

Once a while we might find ourselves in situation when we think “I wish I knew this before” , “Why I did not think about this before” or “Why it took so long to come to this decision or action”. Can computer programs be used to help us to avoid or minimize the situations like this? Having background in computer science I decided to look at human thinking patterns and compare them with the learning computer algorithms.

The situations mentioned above as well as all our actions are result of our learning and thinking. Effective thinking and learning drive good decisions and actions.

As mentioned on Wikipedia [1] – “Learning is the act of acquiring new, or modifying and reinforcing, existing knowledge, behaviors, skills, values, or preferences and may involve synthesizing different types of information.”

Learning very closely connected to thinking. New information often can lead to new thoughts or ideas and during the thinking process we often come to the need to learn something new, to extend our knowledge.

“Thinking is a process of response to external stimuli, and if thinking is effective it results in changes to or strengthening of world views, beliefs, opinions, attitudes, behaviours, skills, understanding, and knowledge.
Thinking and learning have the same outcomes, so have to be very closely related.” [2]

Current computer algorithms can be very intelligent due to the latest advances in computer sciences. Computer programs can learn information and use this information for making intelligent decisions. There are a number of computer fields associated with learning. For example machine learning, deep learning, reinforcement learning successfully provide computer algorithms for learning information in many different applications.

After learning computers make decisions based on learned information and programming instructions created by programmers.
Computers can not think (at least as of right now). Human beings can think and they are very flexible in the process of making decisions. For example they can get new ideas or apply knowledge from totally different domain area.

While computers can not think, the computer programs can be very flexible – nothing stop us from combining several algorithms to cover all or most of all possibilities, nothing stop us to produce more intelligent program.
Just simple example – program can sort apple from pear based on color, or it can use color and shape. In the second case it will be more intelligent and more accurate. If needed we may be could add even more attributes like weight, smell.

Humans have the ability to think and foresee some future situations but not always use this ability. Often people make actions following same patterns or following other people or just picking the easy or random option. It can work well but not always. And here computers can help to humans – as the computer machines can access and process a lot of information and calculate different alternatives and choose optimal solution.

Computer programs use algorithms. Scientists create algorithm and then it coded into program. Can algorithm be created for increasing effective thinking? Different people use different ways of thinking , even for the same problem. However even for different problems, we can see common thinking patterns like following from simple to more complex, dividing the something complex into smaller pieces or using similarity. Some patterns are used often some not. Can we program those patterns? In the next post or posts we will take a look at learning and thinking patterns in the context of how they are programmed for the computers.

References
1. Wikipedia – Learning
2. The Relationship Between Thinking and Learning



Retrieving Emails from POP3 Server Using Python Script

My inbox has a lot of data as many websites are sending notifications and updates. So I tasked myself with creating python script to extract emails from POP3 email server and organize information in better way.
I started from the first step – automatically reading emails from mailbox. Based on examples I have found on Internet I created the script that is retrieving emails and removing not needed information like headers.
I am using web based email (not gmail). In the script I am using poplib module which encapsulates a connection to a POP3 server. Another module that I am using is email – this is a library for managing email messages. As I have many emails I limited for loop to 15 emails.

There are still a few things that can be done. For example I would like to keep “FROM:” data, also some HTML tags still need to be removed. However this code allows to extract body text from emails and can be used as starting point.

Feel free to provide any feedback or suggestions.

Here is the full source code for python script to get body text emails from mailbox.



import poplib
import email


SERVER = "server_name"   
USER = "email_address"
PASSWORD = "email_password"
 

server = poplib.POP3(SERVER)
server.user(USER)
server.pass_(PASSWORD)
 
 
numMessages = len(server.list()[1])
if (numMessages > 15):
    numMessages=15
for i in range(numMessages) :
    (server_msg, body, octets) = server.retr(i+1)
    for j in body:
        try:
            msg = email.message_from_string(j.decode("utf-8"))
            strtext=msg.get_payload()
            print (strtext)
        except:
            pass

References
1. Read Email, pop3
2. poplib — POP3 protocol client includes POP3 Example that opens a mailbox and retrieves and prints all messages
3. email — An email and MIME handling package



Getting WordNet Information and Building Graph with Python and NetworkX

WordNet and Wikipedia are often utilized in text mining algorithms for enriching short text representation [1] or for extracting additional knowledge about words. [2] WordNet’s structure makes it a useful tool for computational linguistics and natural language processing.[3] In this post we will look how to pull information from WordNet using python. Also we will look how to build graph for relations between words using python and NetworkX.

WordNet groups English words into sets of synonyms called synsets, provides short definitions and usage examples, and records a number of relations among these synonym sets or their members. WordNet can thus be seen as a combination of dictionary and thesaurus. While it is accessible to human users via a web browser, its primary use is in automatic text analysis and artificial intelligence applications. [4]

Here is how to get all synsets for the word ‘good’ using NLTK package:


from nltk.corpus import wordnet as wn

print (wn.synsets('good'))

#This is the output of above line:
#[Synset('good.n.01'), Synset('good.n.02'), Synset('good.n.03'), Synset('commodity.n.01'), Synset('good.a.01'), Synset('full.s.06'), Synset('good.a.03'), Synset('estimable.s.02'), Synset('beneficial.s.01'), Synset('good.s.06'), Synset('good.s.07'), Synset('adept.s.01'), Synset('good.s.09'), Synset('dear.s.02'), Synset('dependable.s.04'), Synset('good.s.12'), Synset('good.s.13'), Synset('effective.s.04'), Synset('good.s.15'), Synset('good.s.16'), Synset('good.s.17'), Synset('good.s.18'), Synset('good.s.19'), Synset('good.s.20'), Synset('good.s.21'), Synset('well.r.01'), Synset('thoroughly.r.02')]

All synsets are connected to other synsets by means of semantic relations. These relations, which are not all shared by all lexical categories, include:

hypernyms: Y is a hypernym of X if every X is a (kind of) Y (canine is a hypernym of dog)
hyponyms: Y is a hyponym of X if every Y is a (kind of) X (dog is a hyponym of canine)
meronym: Y is a meronym of X if Y is a part of X (window is a meronym of building)
holonym: Y is a holonym of X if X is a part of Y (building is a holonym of window) [4]

Here is how can we can get hypernyms and hyponyms from WordNet.

car = wn.synset(‘car.n.01’)
print (“HYPERNYMS”)
print (car.hypernyms())
print (“HYPONYMS”)
print (car.hyponyms())

Here is the output from above code:
HYPERNYMS
[Synset(‘motor_vehicle.n.01’)]
HYPONYMS
[Synset(‘ambulance.n.01’), Synset(‘beach_wagon.n.01’), Synset(‘bus.n.04’), Synset(‘cab.n.03’), Synset(‘compact.n.03’), Synset(‘convertible.n.01’), Synset(‘coupe.n.01’), Synset(‘cruiser.n.01’), Synset(‘electric.n.01’), Synset(‘gas_guzzler.n.01’), Synset(‘hardtop.n.01’), Synset(‘hatchback.n.01’), Synset(‘horseless_carriage.n.01’), Synset(‘hot_rod.n.01’), Synset(‘jeep.n.01’), Synset(‘limousine.n.01’), Synset(‘loaner.n.02’), Synset(‘minicar.n.01’), Synset(‘minivan.n.01’), Synset(‘model_t.n.01’), Synset(‘pace_car.n.01’), Synset(‘racer.n.02’), Synset(‘roadster.n.01’), Synset(‘sedan.n.01’), Synset(‘sport_utility.n.01’), Synset(‘sports_car.n.01’), Synset(‘stanley_steamer.n.01’), Synset(‘stock_car.n.01’), Synset(‘subcompact.n.01’), Synset(‘touring_car.n.01’), Synset(‘used-car.n.01’)]

Here is how to get synonyms, antonyms , lemmas and similarity: [5]


synonyms = []
antonyms = []

for syn in wn.synsets("good"):
    for l in syn.lemmas():
        synonyms.append(l.name())
        if l.antonyms():
            antonyms.append(l.antonyms()[0].name())

print(set(synonyms))
print(set(antonyms))
print (syn.lemmas())


w1 = wn.synset('ship.n.01')
w2 = wn.synset('cat.n.01')
print(w1.wup_similarity(w2))

Here is how we can use Textblob package [6] and represent some word relations via graph. The output graph is shown below.


from textblob import Word
word = Word("plant")
print (word.synsets[:5])
print (word.definitions[:5])

word = Word("computer")
for syn in word.synsets:
    for l in syn.lemma_names():
        synonyms.append(l)
        
import networkx as nx
import matplotlib.pyplot as plt
G=nx.Graph()


w=word.synsets[1]


G.add_node(w.name())
for h in w.hypernyms():
      print (h)
      G.add_node(h.name())
      G.add_edge(w.name(),h.name())


for h in w.hyponyms():
      print (h)
      G.add_node(h.name())
      G.add_edge(w.name(),h.name())

print (G.nodes(data=True))
plt.show()
nx.draw(G, width=2, with_labels=True)
plt.savefig("path.png")

Wordnet_graph

Here is the full source code


from nltk.corpus import wordnet as wn

print (wn.synsets('good'))

car = wn.synset('car.n.01')
print ("HYPERNYMS")
print (car.hypernyms())
print ("HYPONYMS")
print (car.hyponyms())

synonyms = []
antonyms = []

for syn in wn.synsets("good"):
    for l in syn.lemmas():
        synonyms.append(l.name())
        if l.antonyms():
            antonyms.append(l.antonyms()[0].name())

print(set(synonyms))
print(set(antonyms))
print (syn.lemmas())



w1 = wn.synset('ship.n.01')
w2 = wn.synset('cat.n.01')
print(w1.wup_similarity(w2))


from textblob import Word
word = Word("plant")
print (word.synsets[:5])
print (word.definitions[:5])

word = Word("computer")
for syn in word.synsets:
    for l in syn.lemma_names():
        synonyms.append(l)


import networkx as nx
import matplotlib.pyplot as plt
G=nx.Graph()


w=word.synsets[1]


G.add_node(w.name())
for h in w.hypernyms():
      print (h)
      G.add_node(h.name())
      G.add_edge(w.name(),h.name())
     



for h in w.hyponyms():
      print (h)
      G.add_node(h.name())
      G.add_edge(w.name(),h.name())



print (G.nodes(data=True))
plt.show()
nx.draw(G, width=2, with_labels=True)
plt.savefig("path.png")

References
1. Enriching short text representation in microblog for clustering
2. Automatic Topic Hierarchy Generation Using WordNet
3. WordNet
4. WordNet
5. WordNet NLTK Tutorial
6. Tutorial: What is WordNet? A Conceptual Introduction Using Python