{"id":975,"date":"2017-02-18T22:04:31","date_gmt":"2017-02-18T22:04:31","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=975"},"modified":"2017-11-30T01:54:40","modified_gmt":"2017-11-30T01:54:40","slug":"building-decision-trees-in-python-handling-categorical-data","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/","title":{"rendered":"Building Decision Trees in Python – Handling Categorical Data"},"content":{"rendered":"

In the post Building Decision Trees in Python<\/a> we looked at the decision tree with numerical continuous dependent variable. This type of decision trees can be called also regression tree<\/strong>. <\/p>\n

But what if we need to use categorical dependent variable<\/strong>? It is still possible to create decision tree and in this post we will look how to create decision tree if dependent variable is categorical data. In this case the decision tree is called classification tree<\/strong>. Classification trees, as the name implies are used to separate the dataset into classes belonging to the response variable. [4] Classification is a typical problem that can be found in such fields as machine learning, predictive analytics, data mining. <\/p>\n

Getting Data<\/strong>
\nFor simplicity we will use the same dataset as before but will convert numerical target variable into categorical variable as we want build python code for decision tree with categorical dependent variable.
\nTo convert dependent variable into categorical we use the following simple rule:
\n if CPC < 22 then CPC = \"Low\"
\n else CPC = “High”<\/p>\n

For independent variables we will use “keyword” and “number of words” fields.
\nKeyword (usually it is several words) is a categorical variable. In general it is not the problem as in either case (regression or classification tree), the predictors or independent variables may be categorical or numeric. It is the target variable that determines the type of decision tree needed.[4]<\/p>\n

However sklearn.tree (at least the version that is used here) does not support categorical independent variable. See discussion and suggestions on stack overflow [5]. To use independent categorical variable, we will code the categorical feature into multiple binary features. For example, we might code [‘red’,’green’,’blue’] with 3 columns, one for each category, having 1 when the category match and 0 otherwise. This is called one-hot-encoding, binary encoding, one-of-k-encoding or whatever. [5]<\/p>\n

Below are shown few rows from the table data. We added 2 columns for categories “insurance”, “Adsense”. We actually have more categories and therefore we added more columns but this is not shown in the table.<\/p>\n

For small dataset such conversion can be done manually. But we also created python script in the post Converting Categorical Text Variable into Binary Variables<\/a> for this specific task. [10]<\/p>\n\n\n\n\n
Keyword<\/td>\nNumber of words<\/td>\nInsurance<\/td>\nAdsense<\/td>\nCTR<\/td>\nCost<\/td>\nCost (categorical)<\/td>\n<\/tr>\n
car insurance premium<\/td>\n3<\/td>\n1<\/td>\n0<\/td>\n\t0.012<\/td>\n\t20<\/td>\n\tLow<\/td>\n<\/tr>\n
AdSense data<\/td>\n2<\/td>\n0<\/td>\n\t1<\/td>\n\t0.025<\/td>\n\t1061<\/td>\n\tHigh<\/td>\n<\/tr>\n<\/table>\n

Building the Code<\/strong>
\nNow we need to build the code. The call for decision tree is looking like this:<\/p>\n

clf_gini = DecisionTreeClassifier(criterion = “gini”, random_state = 100,
\n max_depth=8, min_samples_leaf=4) <\/p>\n

we use here criterion Gini index for splitting data.
\nIn the call to export_graphviz we specify class names:<\/p>\n

export_graphviz(tree, out_file=f, feature_names=feature_names, filled=True, rounded=True, class_names=[“Low”, “High”] )<\/p>\n

The rest of the code is the same as in previous post for regression tree.<\/p>\n

Here is the python computer code:<\/p>\n

\r\n# -*- coding: utf-8 -*-\r\n\r\n\r\nimport pandas as pd\r\nfrom sklearn.cross_validation import train_test_split\r\nfrom sklearn.tree import DecisionTreeClassifier\r\nimport subprocess\r\n\r\nfrom sklearn.tree import  export_graphviz\r\n\r\n\r\ndef visualize_tree(tree, feature_names):\r\n    \r\n    with open(\"dt.dot\", 'w') as f:\r\n        \r\n        export_graphviz(tree, out_file=f, feature_names=feature_names,  filled=True, rounded=True, class_names=[\"Low\", \"High\"] )\r\n\r\n    command = [\"C:\\\\Program Files (x86)\\\\Graphviz2.38\\\\bin\\\\dot.exe\", \"-Tpng\", \"C:\\\\Users\\\\Owner\\\\Desktop\\\\A\\\\Python_2016_A\\\\dt.dot\", \"-o\", \"dt.png\"]\r\n    \r\n       \r\n    try:\r\n        subprocess.check_call(command)\r\n    except:\r\n        exit(\"Could not run dot, ie graphviz, to \"\r\n             \"produce visualization\")\r\n    \r\ndata = pd.read_csv('adwords_data.csv', sep= ',' , header = 1)\r\n\r\n\r\n\r\nX = data.values[:, [3,17,18,19,20,21,22]]\r\nY = data.values[:,8]\r\n\r\n                           \r\nX_train, X_test, y_train, y_test = train_test_split( X, Y, test_size = 0.3, random_state = 100)                           \r\n                           \r\nclf_gini = DecisionTreeClassifier(criterion = \"gini\", random_state = 100, \r\n                               max_depth=8, min_samples_leaf=4)\r\nclf_gini.fit(X_train, y_train)   \r\n\r\n\r\nvisualize_tree(clf_gini, [\"Words in Key Phrase\", \"AdSense\",\t\"Mortgage\",\t\"Money\",\t\"loan\", \t\"lawyer\", \t\"attorney\"])\r\n\r\n\r\n<\/code><\/pre>\n
\"\"
Decision Tree (partial view)<\/figcaption><\/figure>\n
References<\/strong>
\n1. Decision tree<\/a>  Wikipedia
\n2. MLlib – Decision Trees<\/a>
\n3. Visual analysis of AdWords data: a primer<\/a>
\n4. 2 main differences between classification and regression trees<\/a>
\n5. strings as features in decision tree\/random forest<\/a>
\n6. Decision Trees with scikit-learn<\/a>
\n7. Classification: Basic Concepts, Decision Trees, and Model Evaluation<\/a>
\n8. Understanding decision tree output from export_graphviz<\/a>
\n9. Building Decision Trees in Python<\/a>
\n10. Converting Categorical Text Variable into Binary Variables<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
In the post Building Decision Trees in Python we looked at the decision tree with numerical continuous dependent variable. This type of decision trees can be called also regression tree. But what if we need to use categorical dependent variable? It is still possible to create decision tree and in this post we will look … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[5,2,9,10],"tags":[],"jetpack_publicize_connections":[],"yoast_head":"\nBuilding Decision Trees in Python - Handling Categorical Data - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Building Decision Trees in Python - Handling Categorical Data - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"In the post Building Decision Trees in Python we looked at the decision tree with numerical continuous dependent variable. This type of decision trees can be called also regression tree. But what if we need to use categorical dependent variable? It is still possible to create decision tree and in this post we will look ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2017-02-18T22:04:31+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2017-11-30T01:54:40+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/02\/dt-235x300.png\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/\",\"name\":\"Building Decision Trees in Python - Handling Categorical Data - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2017-02-18T22:04:31+00:00\",\"dateModified\":\"2017-11-30T01:54:40+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Building Decision Trees in Python – Handling Categorical Data\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Building Decision Trees in Python - Handling Categorical Data - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/","og_locale":"en_US","og_type":"article","og_title":"Building Decision Trees in Python - Handling Categorical Data - Machine Learning Applications","og_description":"In the post Building Decision Trees in Python we looked at the decision tree with numerical continuous dependent variable. This type of decision trees can be called also regression tree. But what if we need to use categorical dependent variable? It is still possible to create decision tree and in this post we will look ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/","og_site_name":"Machine Learning Applications","article_published_time":"2017-02-18T22:04:31+00:00","article_modified_time":"2017-11-30T01:54:40+00:00","og_image":[{"url":"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/02\/dt-235x300.png"}],"author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/","name":"Building Decision Trees in Python - Handling Categorical Data - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2017-02-18T22:04:31+00:00","dateModified":"2017-11-30T01:54:40+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python-handling-categorical-data\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Building Decision Trees in Python – Handling Categorical Data"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-fJ","jetpack-related-posts":[{"id":1516,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/11\/23\/regression-and-classification-decision-trees-building-with-python-and-running-online\/","url_meta":{"origin":975,"position":0},"title":"Regression and Classification Decision Trees – Building with Python and Running Online","date":"November 23, 2017","format":false,"excerpt":"According to survey [1] Decision Trees constitute one of the 10 most popular data mining algorithms. Decision trees used in data mining are of two main types: Classification tree analysis is when the predicted outcome is the class to which the data belongs. Regression tree analysis is when the predicted\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/11\/decision_tree_11_2017-300x283.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":966,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/18\/building-decision-trees-in-python\/","url_meta":{"origin":975,"position":1},"title":"Building Decision Trees in Python","date":"February 18, 2017","format":false,"excerpt":"A decision tree is a decision support tool that uses a tree-like graph or model of decisions and their possible consequences, including chance event outcomes, resource costs, and utility. It is one way to display an algorithm. Decision trees are commonly used in operations research, specifically in decision analysis, to\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"Decision Tree","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/02\/dt_post1_N_CTQ_Cost_regr1-2-use-this-300x103.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":993,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/02\/22\/converting-categorical-text-variable-into-binary-variables\/","url_meta":{"origin":975,"position":2},"title":"Converting Categorical Text Variable into Binary Variables","date":"February 22, 2017","format":false,"excerpt":"Sometimes we might need convert categorical feature into multiple binary features. Such situation emerged while I was implementing decision tree with independent categorical variable using python sklearn.tree for the post Building Decision Trees in Python - Handling Categorical Data and it turned out that a text independent variable is not\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2194,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/08\/11\/applied-machine-learning-classification-for-decision-making\/","url_meta":{"origin":975,"position":3},"title":"Applied Machine Learning Classification for Decision Making","date":"August 11, 2018","format":false,"excerpt":"Making the good decision is the challenge that we often have. So, in this post, we will look at how applied machine learning classification can be used for the process of decision making. The simple and quick approach to make decision is follow our past experience of similar situations. Usually\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"Decision Making","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/08\/signs-1172211_640-e1534384498570.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1470,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/11\/13\/getting-data-driven-insights-from-blog-data-analysis-with-feature-selection\/","url_meta":{"origin":975,"position":4},"title":"Getting Data-Driven Insights from Blog Data Analysis with Feature Selection","date":"November 13, 2017","format":false,"excerpt":"Machine learning algorithms are widely used in every business - object recognition, marketing analytics, analyzing data in numerous applications to get useful insights. In this post one of machine learning techniques is applied to analysis of blog post data to predict significant features for key metrics such as page views.\u2026","rel":"","context":"In "Data Mining"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/11\/feature_selection-300x253.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1446,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/11\/06\/10-new-top-resources-on-machine-learning-from-around-the-web\/","url_meta":{"origin":975,"position":5},"title":"10 New Top Resources on Machine Learning from Around the Web","date":"November 6, 2017","format":false,"excerpt":"For this post I put new and most interesting machine learning resources that I recently found on the web. This is the list of useful resources in such areas like stock market forecasting, text mining, deep learning, neural networks and getting data from Twitter. Hope you enjoy the reading. 1.\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/975"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=975"}],"version-history":[{"count":28,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/975\/revisions"}],"predecessor-version":[{"id":1560,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/975\/revisions\/1560"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=975"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=975"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=975"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}