{"id":2481,"date":"2019-01-02T01:22:04","date_gmt":"2019-01-02T01:22:04","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=2481"},"modified":"2019-01-07T15:33:12","modified_gmt":"2019-01-07T15:33:12","slug":"reinforcement-learning-python-dqn-application-resource-allocation","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/","title":{"rendered":"Reinforcement Learning Python DQN Application for Resource Allocation"},"content":{"rendered":"

In the previous post Reinforcement Learning Example for Planning Tasks Using Q Learning and Dyna-Q<\/a> we applied Dyna-Q algorithm for planning of actions<\/b> to complete tasks. This problem can be viewed as resource allocation<\/b> task. In this post we will use reinforcement learning python DQN (Deep Q-network)<\/b> for the same problem. In case you did not read previous post the problem is described below. <\/p>\n

The Problem<\/h2>\n

Given<\/b> some goals (projects to complete) and set of actions<\/b> (number of hours to put for each project per day) we are interesting to know<\/b> what action we need to take (how many hours to put per project on each day) in order to get the best result<\/b> in the end (we have reward for completion project in time). <\/p>\n

So we are trying to allocate resource (time) for each project for each day in such way that it produces maximum reward<\/b> in the end of given period. We have reward data and time needed to complete for each project.<\/p>\n

The diagram of one of possible path would look like this:<\/p>\n

\"Planning
Planning Diagram<\/figcaption><\/figure>\n

On this diagram the green indicates path that produces the max reward 13 as the agent was able to complete both goals.<\/p>\n

Deep Q-Network<\/h2>\n

Deep Q-Networks, abbreviated DQN, use deep neural networks<\/b> as function approximation of the
\naction-value function q(s, a). The input of the artificial neural network used is the state and the output is the estimated q-values of the state-action pairs.<\/p>\n

In DQN the replay memory<\/b> simply stores the transitions such that they can be used at later times. By sampling transitions from the replay memory the network increases its ability to generalize<\/b>. This also allows the network to predict the correct values in states which might be visited less frequently when the agent\u2019s strategy gets better.<\/p>\n

Also we add a second network, a target network<\/b>, which is a copy of the first network, which we call the training network<\/b>. The target network is only used to predict the value of taking the optimal action from s0 when updating the training network. The target network is updated with a certain frequency by copying the weights from the training network. This prevents instability when s and s0 are equal or even similar which is often the case. [1]<\/p>\n

Solution<\/h2>\n

The code here is based on DQN with Tensorflow for maze problem[2] and previous code for Dyna-Q mentioned in the beginning of the post. It has 2 modules for programming environment and Reinforcement Learning Tensorflow DQN algorithm. Additionally it has main module which run the loop with episodes.<\/p>\n

To run this reinforcement learning example you can use reinforcement learning python source code from the links below:<\/p>\n

Reinforcement Learning DQN Planning Environment<\/a>
\n Reinforcement Learning DQN<\/a>
\n Reinforcement Learning DQN Run Planning<\/a><\/p>\n

Results<\/h2>\n

Below are charts obtained from running program. Performance (achieving max possible reward) with DQN is a little higher (but not significantly) than with Dyna-Q example on the same problem.<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

References<\/strong><\/p>\n

1. Reinforcement learning for planning of a simulated production line<\/a> Gustaf Ehn, Hugo Werner, February 27, 2018 <\/p>\n

2. Reinforcement Learning Methods and Tutorials<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

In the previous post Reinforcement Learning Example for Planning Tasks Using Q Learning and Dyna-Q we applied Dyna-Q algorithm for planning of actions to complete tasks. This problem can be viewed as resource allocation task. In this post we will use reinforcement learning python DQN (Deep Q-network) for the same problem. In case you did … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[10,97],"tags":[120,118,119,98],"jetpack_publicize_connections":[],"yoast_head":"\nReinforcement Learning Python DQN Application for Resource Allocation - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Reinforcement Learning Python DQN Application for Resource Allocation - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"In the previous post Reinforcement Learning Example for Planning Tasks Using Q Learning and Dyna-Q we applied Dyna-Q algorithm for planning of actions to complete tasks. This problem can be viewed as resource allocation task. In this post we will use reinforcement learning python DQN (Deep Q-network) for the same problem. In case you did ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2019-01-02T01:22:04+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-01-07T15:33:12+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/10\/RL_paths-e1541117896388.png\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/\",\"name\":\"Reinforcement Learning Python DQN Application for Resource Allocation - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2019-01-02T01:22:04+00:00\",\"dateModified\":\"2019-01-07T15:33:12+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Reinforcement Learning Python DQN Application for Resource Allocation\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Reinforcement Learning Python DQN Application for Resource Allocation - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/","og_locale":"en_US","og_type":"article","og_title":"Reinforcement Learning Python DQN Application for Resource Allocation - Machine Learning Applications","og_description":"In the previous post Reinforcement Learning Example for Planning Tasks Using Q Learning and Dyna-Q we applied Dyna-Q algorithm for planning of actions to complete tasks. This problem can be viewed as resource allocation task. In this post we will use reinforcement learning python DQN (Deep Q-network) for the same problem. In case you did ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/","og_site_name":"Machine Learning Applications","article_published_time":"2019-01-02T01:22:04+00:00","article_modified_time":"2019-01-07T15:33:12+00:00","og_image":[{"url":"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/10\/RL_paths-e1541117896388.png"}],"author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/","name":"Reinforcement Learning Python DQN Application for Resource Allocation - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2019-01-02T01:22:04+00:00","dateModified":"2019-01-07T15:33:12+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2019\/01\/02\/reinforcement-learning-python-dqn-application-resource-allocation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Reinforcement Learning Python DQN Application for Resource Allocation"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-E1","jetpack-related-posts":[{"id":2327,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/10\/28\/reinforcement-learning-example-planning-using-q-learning-dyna\/","url_meta":{"origin":2481,"position":0},"title":"Reinforcement Learning Example for Planning Tasks Using Q Learning and Dyna-Q","date":"October 28, 2018","format":false,"excerpt":"What is Planning Process Planning is the process of finding a sequence of actions (steps), which if executed by an agent result in the achievement of a set of predefined goals. The sequence of actions mentioned above is also referred to as plan. Planning is studied within Reinforcement Learning and\u2026","rel":"","context":"In "Reinforcement Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/10\/labyrinth-1015643_640.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1446,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/11\/06\/10-new-top-resources-on-machine-learning-from-around-the-web\/","url_meta":{"origin":2481,"position":1},"title":"10 New Top Resources on Machine Learning from Around the Web","date":"November 6, 2017","format":false,"excerpt":"For this post I put new and most interesting machine learning resources that I recently found on the web. This is the list of useful resources in such areas like stock market forecasting, text mining, deep learning, neural networks and getting data from Twitter. Hope you enjoy the reading. 1.\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2253,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/09\/06\/ml-applications\/","url_meta":{"origin":2481,"position":2},"title":"Everyday Examples of Machine Learning Applications","date":"September 6, 2018","format":false,"excerpt":"Artificial Intelligence and Machine Learning applications is one of the most hottest topics in the industry today. Robots, self driving cars, intelligent chatbots and many other innovations are coming to our work and life. In this post we will look at few machine learning less known applications that were covered\u2026","rel":"","context":"In "Machine learning applications"","img":{"alt_text":"Topic modeling with textacy","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/09\/Topic-modeling-with-textacy-e1536508581929.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":586,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/09\/15\/how-can-we-use-computer-technology-to-improve\/","url_meta":{"origin":2481,"position":3},"title":"How Can We Use Computer Programming to Increase Effective Thinking","date":"September 15, 2016","format":false,"excerpt":"Once a while we might find ourselves in situation when we think \"I wish I knew this before\" , \"Why I did not think about this before\" or \"Why it took so long to come to this decision or action\". Can computer programs be used to help us to avoid\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":227,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/05\/28\/using-python-for-mining-data-from-twitter\/","url_meta":{"origin":2481,"position":4},"title":"Using Python for Mining Data From Twitter","date":"May 28, 2016","format":false,"excerpt":"Twitter is increasingly being used for business or personal purposes. With Twitter API there is also an opportunity to do data mining of data (tweets) and find interesting information. In this post we will take a look how to get data from Twitter, prepare data for analysis and then do\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"Frequency of Hashtags","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2016\/05\/Frequency-of-Hashtags-300x171.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":621,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/10\/09\/online-resources-for-neural-networks-with-python\/","url_meta":{"origin":2481,"position":5},"title":"Online Resources for Neural Networks with Python","date":"October 9, 2016","format":false,"excerpt":"The neural network field enjoys now a resurgence of interest. New training techniques made training deep networks feasible. With deeper networks, more training data and powerful new hardware to make it all work, deep neural networks (or \u201cdeep learning\u201d systems) suddenly began making rapid progress in areas such as speech\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/2481"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=2481"}],"version-history":[{"count":23,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/2481\/revisions"}],"predecessor-version":[{"id":2508,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/2481\/revisions\/2508"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=2481"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=2481"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=2481"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}