{"id":1754,"date":"2018-02-27T11:23:30","date_gmt":"2018-02-27T11:23:30","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=1754"},"modified":"2018-04-17T23:29:56","modified_gmt":"2018-04-17T23:29:56","slug":"time-series-prediction-lstm-keras","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/","title":{"rendered":"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead"},"content":{"rendered":"

In this post I will share experiment with Time Series Prediction with LSTM and Keras. LSTM neural network is used in this experiment for multiple steps ahead for stock prices data. The experiment is based on the paper [1]. The authors of the paper examine independent value prediction approach. With this approach a separate model is built for each prediction step. This approach helps to avoid error accumulation problem that we have when we use multi-stage step prediction.<\/p>\n

LSTM Implementation<\/h3>\n

Following this approach I decided to use Long Short-Term Memory network or LSTM network for daily data stock price prediction. LSTM is a type of recurrent neural network used in deep learning. LSTMs have been used to advance the state-of the-art for many difficult problems. [2]<\/p>\n

For this time series prediction I selected the number of steps to predict ahead = 3 and built 3 LSTM models with Keras in python. For each model I used different variable (fit0, fit1, fit2) to avoid any “memory leakage” between models.
\nThe model initialization code is the same for all 3 models except changing parameters (number of neurons in LSTM layer)
\nThe architecture of the system is shown on the fig below. <\/p>\n

\"Multiple
Multiple step prediction with separate neural networks<\/figcaption><\/figure>\n

Here we have 3 LSTM models that are getting same X input data but different target Y data. The target data is shifted by number of steps. If model is forecasting the data stock price for day 2 then Y is shifted by 2 elements.
\nThis happens in the following line when i=1:<\/p>\n

\r\nyt_ = yt.shift (-i - 1  ) \r\n<\/pre>\n
The data were obtained from stock prices from Internet. <\/p>\n
The number of unit was obtained by running several variations and chosen based on MSE as following:<\/p>\n
   \r\n    if i==0:\r\n        units=20\r\n        batch_size=1\r\n    if i==1:\r\n        units=15\r\n        batch_size=1\r\n    if i==2:\r\n         units=80\r\n         batch_size=1\r\n<\/pre>\n
If you want run more than 3 steps \/ models you will need to add parameters to the above code. Additionally you will need add model initialization code shown below. <\/p>\n
Each LSTM network was constructed as following:<\/p>\n
\r\n\r\n if i == 0 :\r\n          fit0 = Sequential ()\r\n          fit0.add (LSTM (  units , activation = 'tanh', inner_activation = 'hard_sigmoid' , input_shape =(len(cols), 1) ))\r\n          fit0.add(Dropout(0.2))\r\n          fit0.add (Dense (output_dim =1, activation = 'linear'))\r\n          fit0.compile (loss ="mean_squared_error" , optimizer = "adam")  \r\n   \r\n          fit0.fit (x_train, y_train, batch_size =batch_size, nb_epoch =25, shuffle = False)\r\n          train_mse[i] = fit0.evaluate (x_train, y_train, batch_size =batch_size)\r\n          test_mse[i] = fit0.evaluate (x_test, y_test, batch_size =batch_size)\r\n          pred = fit0.predict (x_test) \r\n          pred = scaler_y.inverse_transform (np. array (pred). reshape ((len( pred), 1)))\r\n             # below is just fo i == 0\r\n          for j in range (len(pred)) :\r\n                   prediction_data[j] = pred[j] \r\n<\/pre>\n
For each model the code is saving last forecasted number.
\nAdditionally at step i=0 predicted data is saved for comparison with actual data:<\/p>\n
\r\nprediction_data = np.asarray(prediction_data)\r\nprediction_data = prediction_data.ravel()\r\n\r\n# shift back by one step\r\nfor j in range (len(prediction_data) - 1 ):\r\n    prediction_data[len(prediction_data) - j - 1  ] =  prediction_data[len(prediction_data) - 1 - j - 1]\r\n\r\n# combine prediction data from first model and last predicted data from each model\r\nprediction_data = np.append(prediction_data, forecast)\r\n<\/pre>\n
The full python source code<\/strong> for time series prediction with LSTM in python  is shown here<\/a><\/p>\n
Data<\/strong> can be found here<\/a><\/p>\n
Experiment Results<\/h3>\n
The LSTM neural network was running with the following performance:<\/p>\n
\r\ntrain_mse\r\n[0.01846262458218137, 0.009637593373373323, 0.0018845983509225203]\r\ntest_mse\r\n[0.01648362025879952, 0.026161141224167357, 0.01774421124347165]\r\n<\/pre>\n
Below is the graph of actual data vs data testing data, including last 3 stock data prices from each model.<\/p>\n
\"Multiple
Multiple step prediction – actual data vs predictions<\/figcaption><\/figure>\n
Accuracy of prediction 98% calculated for last 3 data stock prices (one from each model).<\/p>\n
The experiment confirmed that using models (one model for each step) in multistep-ahead time series prediction has advantages. With this method we can adjust parameters of needed LSTM for each step. For example, number of neurons for i=2 was modified to decrease prediction error for this step. And it did not affect predictions for other steps. This is one of machine learning techniques for stock prediction that is described in [1]<\/p>\n
References<\/strong>
\n1. Multistep-ahead Time Series Prediction<\/a>
\n2. LSTM: A Search Space Odyssey<\/a>
\n3. Deep Time Series Forecasting with Python: An Intuitive Introduction to Deep Learning for Applied Time Series Modeling<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
In this post I will share experiment with Time Series Prediction with LSTM and Keras. LSTM neural network is used in this experiment for multiple steps ahead for stock prices data. The experiment is based on the paper [1]. The authors of the paper examine independent value prediction approach. With this approach a separate model … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[9,10,3,60],"tags":[42,49,48,50,27,14,16],"jetpack_publicize_connections":[],"yoast_head":"\nTime Series Prediction with LSTM and Keras for Multiple Steps Ahead - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"In this post I will share experiment with Time Series Prediction with LSTM and Keras. LSTM neural network is used in this experiment for multiple steps ahead for stock prices data. The experiment is based on the paper [1]. The authors of the paper examine independent value prediction approach. With this approach a separate model ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2018-02-27T11:23:30+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-04-17T23:29:56+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/02\/multiple-step-prediction.png\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/\",\"name\":\"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2018-02-27T11:23:30+00:00\",\"dateModified\":\"2018-04-17T23:29:56+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/","og_locale":"en_US","og_type":"article","og_title":"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead - Machine Learning Applications","og_description":"In this post I will share experiment with Time Series Prediction with LSTM and Keras. LSTM neural network is used in this experiment for multiple steps ahead for stock prices data. The experiment is based on the paper [1]. The authors of the paper examine independent value prediction approach. With this approach a separate model ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/","og_site_name":"Machine Learning Applications","article_published_time":"2018-02-27T11:23:30+00:00","article_modified_time":"2018-04-17T23:29:56+00:00","og_image":[{"url":"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/02\/multiple-step-prediction.png"}],"author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/","name":"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2018-02-27T11:23:30+00:00","dateModified":"2018-04-17T23:29:56+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-si","jetpack-related-posts":[{"id":1783,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/01\/19\/machine-learning-stock-prediction-lstm-keras\/","url_meta":{"origin":1754,"position":0},"title":"Machine Learning Stock Prediction with LSTM and Keras","date":"January 19, 2018","format":false,"excerpt":"In this post I will share experiments on machine learning stock prediction with LSTM and Keras with one step ahead. I tried to do first multiple steps ahead with few techniques described in the papers on the web. But I discovered that I need fully understand and test the simplest\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"Forecasting one step ahead LSTM 60","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/01\/Forecasting-one-step-ahead-LSTM-60-3_1_2018.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1798,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/01\/20\/machine-learning-stock-prediction-lstm-keras-python-source-code\/","url_meta":{"origin":1754,"position":1},"title":"Machine Learning Stock Prediction with LSTM and Keras – Python Source Code","date":"January 20, 2018","format":false,"excerpt":"Python Source Code for Machine Learning Stock Prediction with LSTM and Keras - Python Source Code with LSTM and Keras Below is the code for machine learning stock prediction with LSTM neural network. References 1. Machine Learning Stock Prediction with LSTM and Keras - Python Source Code with LSTM and\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1974,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/03\/machine-learning-stock-market-prediction-lstm-keras\/","url_meta":{"origin":1754,"position":2},"title":"Machine Learning Stock Market Prediction with LSTM Keras","date":"April 3, 2018","format":false,"excerpt":"In the previous posts [1,2] I created script for machine learning stock market price on next day prediction. But it was pointed by readers that in stock market prediction, it is more important to know the trend: will the stock go up or down. So I updated the script to\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"Stock Data Prices Prediction with LSTM","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/04\/timeseries_differenced_and_inverted_back.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1995,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/","url_meta":{"origin":1754,"position":3},"title":"LSTM Neural Network Training – Few Useful Techniques for Tuning Hyperparameters and Saving Time","date":"April 17, 2018","format":false,"excerpt":"Neural networks are among the most widely used machine learning techniques.[1] But neural network training and tuning multiple hyper-parameters takes time. I was recently building LSTM neural network for prediction for this post Machine Learning Stock Market Prediction with LSTM Keras and I learned some tricks that can save time.\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"LSTM NN Training Value Loss Charts with High Number and Adjusted","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/04\/LSTM-NN-Training-Value-Loss-with-High-Number-and-adjusted-e1524097124417.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1178,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/05\/14\/time-series-prediction-with-convolutional-neural-networks\/","url_meta":{"origin":1754,"position":4},"title":"Forecasting Time Series Data with Convolutional Neural Networks","date":"May 14, 2017","format":false,"excerpt":"Convolutional neural networks(CNN) is increasingly important concept in computer science and finds more and more applications in different fields. Many posts on the web are about applying convolutional neural networks for image classification as CNN is very useful type of neural networks for image classification. But convolutional neural networks can\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/05\/time-series-LSTM-300x164.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1628,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/12\/17\/time-series-analysis-python-prophet\/","url_meta":{"origin":1754,"position":5},"title":"Time Series Analysis with Python and Prophet","date":"December 17, 2017","format":false,"excerpt":"Recently Facebook released Prophet - open source software tool for forecasting time series data. Facebook team have implemented in Prophet two trend models that can cover many applications: a saturating growth model, and a piecewise linear model. [4] With growth model Prophet can be used for prediction growth\/decay - for\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/12\/time-series-analysis-python-300x180.png?resize=350%2C200","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1754"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=1754"}],"version-history":[{"count":15,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1754\/revisions"}],"predecessor-version":[{"id":1993,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1754\/revisions\/1993"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=1754"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=1754"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=1754"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}