{"id":1995,"date":"2018-04-17T23:53:45","date_gmt":"2018-04-17T23:53:45","guid":{"rendered":"http:\/\/intelligentonlinetools.com\/blog\/?p=1995"},"modified":"2018-04-20T23:43:25","modified_gmt":"2018-04-20T23:43:25","slug":"lstm-neural-network-training-techniques-tuning-hyperparameters","status":"publish","type":"post","link":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/","title":{"rendered":"LSTM Neural Network Training – Few Useful Techniques for Tuning Hyperparameters and Saving Time"},"content":{"rendered":"

Neural networks are among the most widely used machine learning techniques.[1] But neural network training and tuning multiple hyper-parameters takes time. I was recently building LSTM neural network for prediction for this post Machine Learning Stock Market Prediction with LSTM Keras <\/a> and I learned some tricks that can save time. In this post you will find some techniques that helped me to do neural net training more efficiently.<\/p>\n

1. Adjusting Graph To See All Details<\/h2>\n

Sometimes validation loss is getting high value and this prevents from seeing other data on the chart. I added few lines of code to cut high values so you can see all details on chart. <\/p>\n

\r\nimport matplotlib.pyplot as plt\r\nimport matplotlib.ticker as mtick\r\n\r\nT=25\r\nhistory_val_loss=[]\r\n\r\nfor x in history.history['val_loss']:\r\n      if x >= T:\r\n             history_val_loss.append (T)\r\n      else:\r\n             history_val_loss.append( x )\r\n\r\nplt.figure(6)\r\nplt.plot(history.history['loss'])\r\nplt.plot(history_val_loss)\r\nplt.title('model loss adjusted')\r\nplt.ylabel('loss')\r\nplt.xlabel('epoch')\r\nplt.legend(['train', 'test'], loc='upper left')\r\n<\/pre>\n
Below is the example of charts. Left graph is not showing any details except high value point because of the scale. Note that graphs are obtained from different tests.
\n
\"LSTM
LSTM NN Training Value Loss Charts with High Number and Adjusted<\/figcaption><\/figure><\/p>\n
2.  Early Stopping<\/h2>\n
Early stopping is allowing to save time on not running tests when a monitored quantity has stopped improving.  Here is how it can be coded:<\/p>\n
\r\nearlystop = EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=80,  verbose=1, mode='min')\r\ncallbacks_list = [earlystop]\r\n\r\nhistory=model.fit (x_train, y_train, batch_size =1, nb_epoch =1000, shuffle = False, validation_split=0.15, callbacks=callbacks_list)\r\n<\/pre>\n
Here is what arguments mean per Keras documentation [2].<\/p>\n
min_delta: minimum change in the monitored quantity to qualify as an improvement, i.e. an absolute change of less than min_delta, will count as no improvement.
\npatience: number of epochs with no improvement after which training will be stopped.
\nverbose: verbosity mode.
\nmode: one of {auto, min, max}. In min mode, training will stop when the quantity monitored has stopped decreasing; in max mode it will stop when the quantity monitored has stopped increasing; in auto mode, the direction is automatically inferred from the name of the monitored quantity.<\/p>\n
3.  Weight Regularization<\/h2>\n
Weight regularizer can be used to regularize neural net weights. Here is the example. <\/p>\n
\r\nfrom keras.regularizers import L1L2\r\nmodel.add (LSTM ( 400,  activation = 'relu', inner_activation = 'hard_sigmoid' , bias_regularizer=L1L2(l1=0.01, l2=0.01),  input_shape =(len(cols), 1), return_sequences = False ))\r\n<\/pre>\n
Below are the charts that are showing impact of weight regularizer on loss value :
\n
\"LSTM
LSTM NN Training Value Loss without weigh regularization<\/figcaption><\/figure><\/p>\n
\"LSTM
LSTM NN Training Value Loss without weigh regularization<\/figcaption><\/figure>\n
Without weight regularization validation loss is going more up during the neural net training.<\/p>\n
 4. Optimizer <\/h2>\n
Keras software allows to use different optimizers.  I was using adam optimizer which is widely used. Here is how it can be used:<\/p>\n
\r\nadam=optimizers.Adam(lr=0.01, beta_1=0.91, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=True)\r\nmodel.compile (loss ="mean_squared_error" , optimizer = "adam") \r\n<\/pre>\n
I found that beta_1=0.89 performed better then suggested 0.91 or other tested values. <\/p>\n
5. Rolling Window Size <\/h2>\n
Rolling window (in case we use it) also can impact on performance. Too small or too big will drive higher validation loss. Below are charts for different window size (N=4,8,16,18, from left to right). In this case the optimal value was 16 which resulted in 81% accuracy.<\/p>\n
\"LSTM
LSTM Neural Net  Loss Charts with Different N<\/figcaption><\/figure>\n
I hope you enjoyed this post on different techniques for tuning hyper parameters.  If you have any tips or anything else to add, please leave a comment below in the comment box.<\/p>\n
Below is the full source code:<\/p>\n
\r\nimport numpy as np\r\nimport pandas as pd\r\nfrom sklearn import preprocessing\r\n\r\nimport matplotlib.pyplot as plt\r\nimport matplotlib.ticker as mtick\r\n\r\nfrom keras.regularizers import L1L2\r\n\r\nfname="C:\\\\Users\\\\stock data\\\\GM.csv"\r\ndata_csv = pd.read_csv (fname)\r\n\r\n#how many data we will use \r\n# (should not be more than dataset length )\r\ndata_to_use= 150\r\n\r\n# number of training data\r\n# should be less than data_to_use\r\ntrain_end =120\r\n\r\n\r\ntotal_data=len(data_csv)\r\n\r\n#most recent data is in the end \r\n#so need offset\r\nstart=total_data - data_to_use\r\n\r\n\r\nyt = data_csv.iloc [start:total_data ,4]    #Close price\r\nyt_ = yt.shift (-1)   \r\n\r\nprint (yt_)\r\n\r\ndata = pd.concat ([yt, yt_], axis =1)\r\ndata. columns = ['yt', 'yt_']\r\n\r\n\r\nN=16    \r\ncols =['yt']\r\nfor i in range (N):\r\n  \r\n    data['yt'+str(i)] = list(yt.shift(i+1))\r\n    cols.append ('yt'+str(i))\r\n    \r\ndata = data.dropna()\r\ndata_original = data\r\ndata=data.diff()\r\ndata = data.dropna()\r\n    \r\n    \r\n# target variable - closed price\r\n# after shifting\r\ny = data ['yt_']\r\nx = data [cols]\r\n\r\n   \r\nscaler_x = preprocessing.MinMaxScaler ( feature_range =( -1, 1))\r\nx = np. array (x).reshape ((len( x) ,len(cols)))\r\nx = scaler_x.fit_transform (x)\r\n\r\nscaler_y = preprocessing. MinMaxScaler ( feature_range =( -1, 1))\r\ny = np.array (y).reshape ((len( y), 1))\r\ny = scaler_y.fit_transform (y)\r\n\r\n    \r\nx_train = x [0: train_end,]\r\nx_test = x[ train_end +1:len(x),]    \r\ny_train = y [0: train_end] \r\ny_test = y[ train_end +1:len(y)]  \r\n\r\nx_train = x_train.reshape (x_train. shape + (1,)) \r\nx_test = x_test.reshape (x_test. shape + (1,))\r\n\r\nfrom keras.models import Sequential\r\nfrom keras.layers.core import Dense\r\nfrom keras.layers.recurrent import LSTM\r\nfrom keras.layers import  Dropout\r\nfrom keras import optimizers\r\n\r\nfrom numpy.random import seed\r\nseed(1)\r\nfrom tensorflow import set_random_seed\r\nset_random_seed(2)\r\n\r\nfrom keras import regularizers\r\n\r\nfrom keras.callbacks import EarlyStopping\r\n\r\n\r\nearlystop = EarlyStopping(monitor='val_loss', min_delta=0.0001, patience=80,  verbose=1, mode='min')\r\ncallbacks_list = [earlystop]\r\n\r\nmodel = Sequential ()\r\nmodel.add (LSTM ( 400,  activation = 'relu', inner_activation = 'hard_sigmoid' , bias_regularizer=L1L2(l1=0.01, l2=0.01),  input_shape =(len(cols), 1), return_sequences = False ))\r\nmodel.add(Dropout(0.3))\r\nmodel.add (Dense (output_dim =1, activation = 'linear', activity_regularizer=regularizers.l1(0.01)))\r\nadam=optimizers.Adam(lr=0.01, beta_1=0.89, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=True)\r\nmodel.compile (loss ="mean_squared_error" , optimizer = "adam") \r\nhistory=model.fit (x_train, y_train, batch_size =1, nb_epoch =1000, shuffle = False, validation_split=0.15, callbacks=callbacks_list)\r\n\r\n\r\ny_train_back=scaler_y.inverse_transform (np. array (y_train). reshape ((len( y_train), 1)))\r\nplt.figure(1)\r\nplt.plot (y_train_back)\r\n\r\n\r\nfmt = '%.1f'\r\ntick = mtick.FormatStrFormatter(fmt)\r\nax = plt.axes()\r\nax.yaxis.set_major_formatter(tick)\r\nprint (model.summary())\r\n\r\nprint(history.history.keys())\r\n\r\nT=25\r\nhistory_val_loss=[]\r\n\r\nfor x in history.history['val_loss']:\r\n      if x >= T:\r\n             history_val_loss.append (T)\r\n      else:\r\n             history_val_loss.append( x )\r\n\r\n\r\nplt.figure(2)\r\nplt.plot(history.history['loss'])\r\nplt.plot(history.history['val_loss'])\r\nplt.title('model loss')\r\nplt.ylabel('loss')\r\nplt.xlabel('epoch')\r\nplt.legend(['train', 'test'], loc='upper left')\r\nfmt = '%.1f'\r\ntick = mtick.FormatStrFormatter(fmt)\r\nax = plt.axes()\r\nax.yaxis.set_major_formatter(tick)\r\n\r\n\r\n\r\nplt.figure(6)\r\nplt.plot(history.history['loss'])\r\nplt.plot(history_val_loss)\r\nplt.title('model loss adjusted')\r\nplt.ylabel('loss')\r\nplt.xlabel('epoch')\r\nplt.legend(['train', 'test'], loc='upper left')\r\n\r\n\r\nscore_train = model.evaluate (x_train, y_train, batch_size =1)\r\nscore_test = model.evaluate (x_test, y_test, batch_size =1)\r\nprint (" in train MSE = ", round( score_train ,4)) \r\nprint (" in test MSE = ", score_test )\r\n\r\npred1 = model.predict (x_test) \r\npred1 = scaler_y.inverse_transform (np. array (pred1). reshape ((len( pred1), 1)))\r\n \r\nprediction_data = pred1[-1]     \r\nmodel.summary()\r\nprint ("Inputs: {}".format(model.input_shape))\r\nprint ("Outputs: {}".format(model.output_shape))\r\nprint ("Actual input: {}".format(x_test.shape))\r\nprint ("Actual output: {}".format(y_test.shape))\r\n\r\nprint ("prediction data:")\r\nprint (prediction_data)\r\n\r\ny_test = scaler_y.inverse_transform (np. array (y_test). reshape ((len( y_test), 1)))\r\nprint ("y_test:")\r\nprint (y_test)\r\n\r\nact_data = np.array([row[0] for row in y_test])\r\n\r\nfmt = '%.1f'\r\ntick = mtick.FormatStrFormatter(fmt)\r\nax = plt.axes()\r\nax.yaxis.set_major_formatter(tick)\r\n\r\nplt.figure(3)\r\nplt.plot( y_test, label="actual")\r\nplt.plot(pred1, label="predictions")\r\n\r\nprint ("act_data:")\r\nprint (act_data)\r\n\r\nprint ("pred1:")\r\nprint (pred1)\r\n\r\nplt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.05),\r\n          fancybox=True, shadow=True, ncol=2)\r\n\r\n\r\nfmt = '$%.1f'\r\ntick = mtick.FormatStrFormatter(fmt)\r\nax = plt.axes()\r\nax.yaxis.set_major_formatter(tick)\r\n\r\ndef moving_test_window_preds(n_future_preds):\r\n\r\n    ''' n_future_preds - Represents the number of future predictions we want to make\r\n                         This coincides with the number of windows that we will move forward\r\n                         on the test data\r\n    '''\r\n    preds_moving = []                                    # Store the prediction made on each test window\r\n    moving_test_window = [x_test[0,:].tolist()]          # First test window\r\n    moving_test_window = np.array(moving_test_window)    \r\n   \r\n    for i in range(n_future_preds):\r\n      \r\n      \r\n        preds_one_step = model.predict(moving_test_window) \r\n        preds_moving.append(preds_one_step[0,0]) \r\n                       \r\n        preds_one_step = preds_one_step.reshape(1,1,1) \r\n        moving_test_window = np.concatenate((moving_test_window[:,1:,:], preds_one_step), axis=1) # new moving test window, where the first element from the window has been removed and the prediction  has been appended to the end\r\n        \r\n\r\n    print ("pred moving before scaling:")\r\n    print (preds_moving)\r\n                                         \r\n    preds_moving = scaler_y.inverse_transform((np.array(preds_moving)).reshape(-1, 1))\r\n    \r\n    print ("pred moving after scaling:")\r\n    print (preds_moving)\r\n    return preds_moving\r\n    \r\nprint ("do moving test predictions for next 22 days:")    \r\npreds_moving = moving_test_window_preds(22)\r\n\r\n\r\ncount_correct=0\r\nerror =0\r\nfor i in range (len(y_test)):\r\n    error=error + ((y_test[i]-preds_moving[i])**2) \/ y_test[i]\r\n\r\n \r\n    if y_test[i] >=0 and preds_moving[i] >=0 :\r\n        count_correct=count_correct+1\r\n    if y_test[i] < 0 and preds_moving[i] < 0 :\r\n        count_correct=count_correct+1\r\n\r\naccuracy_in_change =  count_correct \/ (len(y_test) )\r\n\r\nplt.figure(4)\r\nplt.title("Forecast vs Actual, (data is differenced)")          \r\nplt.plot(preds_moving, label="predictions")\r\nplt.plot(y_test, label="actual")\r\nplt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.05),\r\n          fancybox=True, shadow=True, ncol=2)\r\n\r\n\r\nprint ("accuracy_in_change:")\r\nprint (accuracy_in_change)\r\n\r\nind=data_original.index.values[0] + data_original.shape[0] -len(y_test)-1\r\nprev_starting_price = data_original.loc[ind,"yt_"]\r\npreds_moving_before_diff =  [0 for x in range(len(preds_moving))]\r\n\r\nfor i in range (len(preds_moving)):\r\n    if (i==0):\r\n        preds_moving_before_diff[i]=prev_starting_price + preds_moving[i]\r\n    else:\r\n        preds_moving_before_diff[i]=preds_moving_before_diff[i-1]+preds_moving[i]\r\n\r\n\r\ny_test_before_diff = [0 for x in range(len(y_test))]\r\n\r\nfor i in range (len(y_test)):\r\n    if (i==0):\r\n        y_test_before_diff[i]=prev_starting_price + y_test[i]\r\n    else:\r\n        y_test_before_diff[i]=y_test_before_diff[i-1]+y_test[i]\r\n\r\n\r\nplt.figure(5)\r\nplt.title("Forecast vs Actual (non differenced data)")\r\nplt.plot(preds_moving_before_diff, label="predictions")\r\nplt.plot(y_test_before_diff, label="actual")\r\nplt.legend(loc='upper center', bbox_to_anchor=(0.5, -0.05),\r\n          fancybox=True, shadow=True, ncol=2)\r\nplt.show()\r\n\r\n<\/pre>\n
References<\/strong>
\n1. Enhancing Neural Network Models for Knowledge Base Completion<\/a>
\n2. Usage of callbacks<\/a>
\n3. Rolling Window Regression: a Simple Approach for Time Series Next value Predictions<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Neural networks are among the most widely used machine learning techniques.[1] But neural network training and tuning multiple hyper-parameters takes time. I was recently building LSTM neural network for prediction for this post Machine Learning Stock Market Prediction with LSTM Keras and I learned some tricks that can save time. In this post you will … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[9,10,60],"tags":[48,63,18,27,16],"jetpack_publicize_connections":[],"yoast_head":"\nLSTM Neural Network Training - Few Useful Techniques for Tuning Hyperparameters and Saving Time - Machine Learning Applications<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"LSTM Neural Network Training - Few Useful Techniques for Tuning Hyperparameters and Saving Time - Machine Learning Applications\" \/>\n<meta property=\"og:description\" content=\"Neural networks are among the most widely used machine learning techniques.[1] But neural network training and tuning multiple hyper-parameters takes time. I was recently building LSTM neural network for prediction for this post Machine Learning Stock Market Prediction with LSTM Keras and I learned some tricks that can save time. In this post you will ... Read more\" \/>\n<meta property=\"og:url\" content=\"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/\" \/>\n<meta property=\"og:site_name\" content=\"Machine Learning Applications\" \/>\n<meta property=\"article:published_time\" content=\"2018-04-17T23:53:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2018-04-20T23:43:25+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/04\/LSTM-NN-Training-Value-Loss-with-High-Number-and-adjusted-e1524097124417.png\" \/>\n<meta name=\"author\" content=\"owygs156\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"owygs156\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/\",\"name\":\"LSTM Neural Network Training - Few Useful Techniques for Tuning Hyperparameters and Saving Time - Machine Learning Applications\",\"isPartOf\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\"},\"datePublished\":\"2018-04-17T23:53:45+00:00\",\"dateModified\":\"2018-04-20T23:43:25+00:00\",\"author\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\"},\"breadcrumb\":{\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"http:\/\/intelligentonlinetools.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"LSTM Neural Network Training – Few Useful Techniques for Tuning Hyperparameters and Saving Time\"}]},{\"@type\":\"WebSite\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#website\",\"url\":\"http:\/\/intelligentonlinetools.com\/blog\/\",\"name\":\"Machine Learning Applications\",\"description\":\"Artificial intelligence, data mining and machine learning for building web based tools and services.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478\",\"name\":\"owygs156\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"contentUrl\":\"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g\",\"caption\":\"owygs156\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"LSTM Neural Network Training - Few Useful Techniques for Tuning Hyperparameters and Saving Time - Machine Learning Applications","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/","og_locale":"en_US","og_type":"article","og_title":"LSTM Neural Network Training - Few Useful Techniques for Tuning Hyperparameters and Saving Time - Machine Learning Applications","og_description":"Neural networks are among the most widely used machine learning techniques.[1] But neural network training and tuning multiple hyper-parameters takes time. I was recently building LSTM neural network for prediction for this post Machine Learning Stock Market Prediction with LSTM Keras and I learned some tricks that can save time. In this post you will ... Read more","og_url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/","og_site_name":"Machine Learning Applications","article_published_time":"2018-04-17T23:53:45+00:00","article_modified_time":"2018-04-20T23:43:25+00:00","og_image":[{"url":"http:\/\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/04\/LSTM-NN-Training-Value-Loss-with-High-Number-and-adjusted-e1524097124417.png"}],"author":"owygs156","twitter_card":"summary_large_image","twitter_misc":{"Written by":"owygs156","Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/","url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/","name":"LSTM Neural Network Training - Few Useful Techniques for Tuning Hyperparameters and Saving Time - Machine Learning Applications","isPartOf":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#website"},"datePublished":"2018-04-17T23:53:45+00:00","dateModified":"2018-04-20T23:43:25+00:00","author":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478"},"breadcrumb":{"@id":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/17\/lstm-neural-network-training-techniques-tuning-hyperparameters\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"http:\/\/intelligentonlinetools.com\/blog\/"},{"@type":"ListItem","position":2,"name":"LSTM Neural Network Training – Few Useful Techniques for Tuning Hyperparameters and Saving Time"}]},{"@type":"WebSite","@id":"http:\/\/intelligentonlinetools.com\/blog\/#website","url":"http:\/\/intelligentonlinetools.com\/blog\/","name":"Machine Learning Applications","description":"Artificial intelligence, data mining and machine learning for building web based tools and services.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"http:\/\/intelligentonlinetools.com\/blog\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/7a886dc5eb9758369af2f6d2cb342478","name":"owygs156","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"http:\/\/intelligentonlinetools.com\/blog\/#\/schema\/person\/image\/","url":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","contentUrl":"http:\/\/2.gravatar.com\/avatar\/b351def598609cb4c0b5bca26497c7e5?s=96&d=mm&r=g","caption":"owygs156"}}]}},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7h1IJ-wb","jetpack-related-posts":[{"id":1783,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/01\/19\/machine-learning-stock-prediction-lstm-keras\/","url_meta":{"origin":1995,"position":0},"title":"Machine Learning Stock Prediction with LSTM and Keras","date":"January 19, 2018","format":false,"excerpt":"In this post I will share experiments on machine learning stock prediction with LSTM and Keras with one step ahead. I tried to do first multiple steps ahead with few techniques described in the papers on the web. But I discovered that I need fully understand and test the simplest\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"Forecasting one step ahead LSTM 60","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/01\/Forecasting-one-step-ahead-LSTM-60-3_1_2018.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1754,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/02\/27\/time-series-prediction-lstm-keras\/","url_meta":{"origin":1995,"position":1},"title":"Time Series Prediction with LSTM and Keras for Multiple Steps Ahead","date":"February 27, 2018","format":false,"excerpt":"In this post I will share experiment with Time Series Prediction with LSTM and Keras. LSTM neural network is used in this experiment for multiple steps ahead for stock prices data. The experiment is based on the paper [1]. The authors of the paper examine independent value prediction approach. With\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"Multiple step prediction with separate neural networks","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/02\/multiple-step-prediction.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1798,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/01\/20\/machine-learning-stock-prediction-lstm-keras-python-source-code\/","url_meta":{"origin":1995,"position":2},"title":"Machine Learning Stock Prediction with LSTM and Keras – Python Source Code","date":"January 20, 2018","format":false,"excerpt":"Python Source Code for Machine Learning Stock Prediction with LSTM and Keras - Python Source Code with LSTM and Keras Below is the code for machine learning stock prediction with LSTM neural network. References 1. Machine Learning Stock Prediction with LSTM and Keras - Python Source Code with LSTM and\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1178,"url":"http:\/\/intelligentonlinetools.com\/blog\/2017\/05\/14\/time-series-prediction-with-convolutional-neural-networks\/","url_meta":{"origin":1995,"position":3},"title":"Forecasting Time Series Data with Convolutional Neural Networks","date":"May 14, 2017","format":false,"excerpt":"Convolutional neural networks(CNN) is increasingly important concept in computer science and finds more and more applications in different fields. Many posts on the web are about applying convolutional neural networks for image classification as CNN is very useful type of neural networks for image classification. But convolutional neural networks can\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2017\/05\/time-series-LSTM-300x164.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":1974,"url":"http:\/\/intelligentonlinetools.com\/blog\/2018\/04\/03\/machine-learning-stock-market-prediction-lstm-keras\/","url_meta":{"origin":1995,"position":4},"title":"Machine Learning Stock Market Prediction with LSTM Keras","date":"April 3, 2018","format":false,"excerpt":"In the previous posts [1,2] I created script for machine learning stock market price on next day prediction. But it was pointed by readers that in stock market prediction, it is more important to know the trend: will the stock go up or down. So I updated the script to\u2026","rel":"","context":"In "Machine Learning"","img":{"alt_text":"Stock Data Prices Prediction with LSTM","src":"https:\/\/i0.wp.com\/intelligentonlinetools.com\/blog\/wp-content\/uploads\/2018\/04\/timeseries_differenced_and_inverted_back.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":621,"url":"http:\/\/intelligentonlinetools.com\/blog\/2016\/10\/09\/online-resources-for-neural-networks-with-python\/","url_meta":{"origin":1995,"position":5},"title":"Online Resources for Neural Networks with Python","date":"October 9, 2016","format":false,"excerpt":"The neural network field enjoys now a resurgence of interest. New training techniques made training deep networks feasible. With deeper networks, more training data and powerful new hardware to make it all work, deep neural networks (or \u201cdeep learning\u201d systems) suddenly began making rapid progress in areas such as speech\u2026","rel":"","context":"In "Artificial Intelligence"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1995"}],"collection":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/comments?post=1995"}],"version-history":[{"count":17,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1995\/revisions"}],"predecessor-version":[{"id":1997,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/posts\/1995\/revisions\/1997"}],"wp:attachment":[{"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/media?parent=1995"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/categories?post=1995"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/intelligentonlinetools.com\/blog\/wp-json\/wp\/v2\/tags?post=1995"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}