{"id":966,"date":"2017-02-18T21:03:17","date_gmt":"2017-02-18T21:03:17","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=966"},"modified":"2017-11-30T02:16:01","modified_gmt":"2017-11-30T02:16:01","slug":"building-decision-trees-in-python","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/","title":{"rendered":"Building Decision Trees in Python"},"content":{"rendered":"

A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm.
\nDecision trees are commonly used in operations research, specifically in decision analysis, to help identify a strategy most likely to reach a goal, but are also a popular tool in machine learning. [1]<\/p>\n

Decision trees<\/strong> are widely used since they are easy to interpret, handle categorical features, extend to the multiclass classification setting, do not require feature scaling, and are able to capture non-linearities and feature interactions. [2] Decision trees are also one of the most widely used predictive analytics techniques. <\/p>\n

Recently I decided to build python code for decision tree for AdWords data. This was motivated by the post [3] about visual analytics used for AdWords dataset. Below are the main components that I used for implementing decision tree.<\/p>\n

Dataset<\/strong>
\nAdWords dataset – the dataset was obtained on Internet. Below is the table with few rows to show data. Only the columns that were used for decision tree, are shown in the table.<\/p>\n\n\n\n\n
Keyword<\/td>\nNumber of words<\/td>\nCPC<\/td>\nClicks<\/td>\nCTR<\/td>\nCost<\/td>\nImpressions<\/td>\n<\/tr>\n
car insurance premium<\/td>\n3<\/td>\n176<\/td>\n\t7<\/td>\n\t0.012<\/td>\n\t1399<\/td>\n\t484<\/td>\n<\/tr>\n
AdSense data<\/td>\n2<\/td>\n119<\/td>\n\t13<\/td>\n\t0.025<\/td>\n\t1061<\/td>\n\t466<\/td>\n<\/tr>\n<\/table>\n

The following independent variables were selected:
\nNumber of words in keyword phrase – this column was added based on the keyword phrase column.
\nCTR – click through rate<\/p>\n

For the dependent variable it was selected CPC – Average Cost per Click.<\/p>\n

Python Module<\/strong>
\nAs the dependent variable is numeric and continuos, the regression decision tree from python module sklearn.tree was used in the script:
\n from sklearn.tree import DecisionTreeRegressor<\/p>\n

In sklearn.tree Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. [5]<\/p>\n

Visualization<\/strong>
\nFor visualization of decision tree graphviz was installed. “Graphviz is open source graph visualization software. Graph visualization is a way of representing structural information as diagrams of abstract graphs and networks. It has important applications in networking, bioinformatics, software engineering, database and web design, machine learning, and in visual interfaces for other technical domains.” <\/p>\n

Python Script<\/strong>
\nThe created code consisted of the following steps:
\nreading data from csv data file
\nselecting needed columns
\nsplitting dataset for testing and training
\ninitializing DecisionTreeRegressor
\nvisualizing decision tree via function. Note that the path to Graphviz are specified inside of scripts. <\/p>\n

Decision tree and Python code are shown below. Online resources used for this post are provided in the reference section.<\/p>\n

\"Decision
Decision Tree<\/figcaption><\/figure>\n

Python computer code:<\/p>\n

\r\n# -*- coding: utf-8 -*-\r\n\r\nimport pandas as pd\r\nfrom sklearn.cross_validation import train_test_split\r\nfrom sklearn.tree import DecisionTreeRegressor\r\n\r\n\r\nimport subprocess\r\n\r\nfrom sklearn.tree import  export_graphviz\r\n\r\n\r\ndef visualize_tree(tree, feature_names):\r\n    \r\n    with open(\"dt.dot\", 'w') as f:\r\n        \r\n        export_graphviz(tree, out_file=f, feature_names=feature_names,  filled=True, rounded=True )\r\n\r\n    command = [\"C:\\\\Program Files (x86)\\\\Graphviz2.38\\\\bin\\\\dot.exe\", \"-Tpng\", \"C:\\\\Users\\\\Owner\\\\Desktop\\\\A\\\\Python_2016_A\\\\dt.dot\", \"-o\", \"dt.png\"]\r\n    \r\n        \r\n    try:\r\n        subprocess.check_call(command)\r\n    except:\r\n        exit(\"Could not run dot, ie graphviz, to \"\r\n             \"produce visualization\")\r\n    \r\ndata = pd.read_csv('adwords_data.csv', sep= ',' , header = 1)\r\n\r\n\r\nX = data.values[:, [3,13]]\r\nY = data.values[:,11]\r\n\r\n                       \r\nX_train, X_test, y_train, y_test = train_test_split( X, Y, test_size = 0.3, random_state = 100)                           \r\n                           \r\nclf = DecisionTreeRegressor( random_state = 100,\r\n                               max_depth=3, min_samples_leaf=4)\r\nclf.fit(X_train, y_train)   \r\n\r\n\r\nvisualize_tree(clf, [\"Words in Key Phrase\", \"CTR\"])\r\n<\/code><\/pre>\n
References<\/strong>
\n1. Decision tree<\/a>  Wikipedia
\n2. MLlib – Decision Trees<\/a>
\n3. Visual analysis of AdWords data: a primer<\/a>
\n4. Decision trees in python with scikit_learn and pandas<\/a>
\n5. Decision Tree<\/a>
\n6. Graphviz – Graph Visualization Software<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm. Decision trees are commonly used in operations research, specifically in decision analysis, to help identify a strategy most … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[5,2,9,10],"tags":[],"jetpack_publicize_connections":[],"yoast_head":"\nBuilding Decision Trees in Python - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building Decision Trees in Python - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm. Decision trees are commonly used in operations research, specifically in decision analysis, to help identify a strategy most ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2017-02-18T21:03:17+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-11-30T02:16:01+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/02\/dt_post1_N_CTQ_Cost_regr1-2-use-this-300x103.png\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/\",\"name\":\"Building Decision Trees in Python - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2017-02-18T21:03:17+00:00\",\"dateModified\":\"2017-11-30T02:16:01+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Building Decision Trees in Python\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Building Decision Trees in Python - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/","og_locale":"en_US","og_type":"article","og_title":"Building Decision Trees in Python - Machine Learning Applications","og_description":"A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm. Decision trees are commonly used in operations research, specifically in decision analysis, to help identify a strategy most ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/","og_site_name":"Machine Learning Applications","article_published_time":"2017-02-18T21:03:17+00:00","article_modified_time":"2017-11-30T02:16:01+00:00","og_image":[{"url":"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/02\/dt_post1_N_CTQ_Cost_regr1-2-use-this-300x103.png"}],"author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/","name":"Building Decision Trees in Python - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2017-02-18T21:03:17+00:00","dateModified":"2017-11-30T02:16:01+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Building Decision Trees in Python"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-fA","jetpack-related-posts":[{"id":975,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/","url_meta":{"origin":966,"position":0},"title":"Building Decision Trees in Python – Handling Categorical Data","date":"February 18, 2017","format":false,"excerpt":"In the post Building Decision Trees in Python we looked at the decision tree with numerical continuous dependent variable. This type of decision trees can be called also regression tree. But what if we need to use categorical dependent variable? It is still possible to create decision tree and in\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/02\/dt-235x300.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1516,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/11\/23\/regression-and-classification-decision-trees-building-with-python-and-running-online\/","url_meta":{"origin":966,"position":1},"title":"Regression and Classification Decision Trees – Building with Python and Running Online","date":"November 23, 2017","format":false,"excerpt":"According to survey [1] Decision Trees constitute one of the 10 most popular data mining algorithms. Decision trees used in data mining are of two main types: Classification tree analysis is when the predicted outcome is the class to which the data belongs. Regression tree analysis is when the predicted\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/11\/decision_tree_11_2017-300x283.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":993,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/22\/converting-categorical-text-variable-into-binary-variables\/","url_meta":{"origin":966,"position":2},"title":"Converting Categorical Text Variable into Binary Variables","date":"February 22, 2017","format":false,"excerpt":"Sometimes we might need convert categorical feature into multiple binary features. Such situation emerged while I was implementing decision tree with independent categorical variable using python sklearn.tree for the post Building Decision Trees in Python - Handling Categorical Data and it turned out that a text independent variable is not\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2194,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/08\/11\/applied-machine-learning-classification-for-decision-making\/","url_meta":{"origin":966,"position":3},"title":"Applied Machine Learning Classification for Decision Making","date":"August 11, 2018","format":false,"excerpt":"Making the good decision is the challenge that we often have. So, in this post, we will look at how applied machine learning classification can be used for the process of decision making. The simple and quick approach to make decision is follow our past experience of similar situations. Usually\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"Decision Making","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/08\/signs-1172211_640-e1534384498570.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1446,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/11\/06\/10-new-top-resources-on-machine-learning-from-around-the-web\/","url_meta":{"origin":966,"position":4},"title":"10 New Top Resources on Machine Learning from Around the Web","date":"November 6, 2017","format":false,"excerpt":"For this post I put new and most interesting machine learning resources that I recently found on the web. This is the list of useful resources in such areas like stock market forecasting, text mining, deep learning, neural networks and getting data from Twitter. Hope you enjoy the reading. 1.\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2529,"url":"http:\/\/intelligentonlinetools.com\/blog\/2019\/07\/10\/python-files-tracker\/","url_meta":{"origin":966,"position":5},"title":"Python Files Tracker for Reducing Time Consuming Tasks","date":"July 10, 2019","format":false,"excerpt":"Do you want to know how many python files you create or update each year? Or do you need review actions to be completed next month? Or you maybe run machine learning python models located in different folders, and find that it takes extra time to get back after switching\u2026","rel":"","context":"In "Automate Time Consuming Tasks"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2019\/07\/Example-of-usage-python-files-tracker.jpg?resize=350%2C200","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/966"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=966"}],"version-history":[{"count":34,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/966\/revisions"}],"predecessor-version":[{"id":1567,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/966\/revisions\/1567"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=966"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=966"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=966"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}