DeepTrading with TensorFlow VII

Making prediction of GOLD price

For the first time, we will use the features of multiple financial instruments. In this case, we will use the main Forex pairs and the SP500 to perform our forecasts on Gold. This Jupyter notebook will be your guide for more complex calculations. Obviously, you can change the features and instruments as you want or you need. That will be your own research. I have given you the guide.

Problem definition

Experiment conditions:

• features = [“open”, “high”, “low”, “close”] for each Forex pair and SPXUSD
• Features Normalization

We have the previous features in candles of one minute, but, to simplify, we will only use the opening price of each of the currency pairs. With these features, we will calculate the opening price of the next one hour candle for GOLD (XAUUSD).D).

Finally, we have to say that in some parts of the code, we have abused of pandas dataframes only with the aim of getting the cleare printable landscape of the working data. These lines of code are indicated with the comment “…as dataframe” and may be avoided if you translate this notebook to python scripts.

Again, we are going to follow our Supervised Learning Flowchart as we see below:

Load configuration

Below, it is an example of .env file for your customization:

PROJ_DIR=.
DATA_DIR=../data/
RAW_DIR=../data/raw/
INTERIM_DIR=../data/interim/
PROCESSED_DIR=../data/processed/
FIGURES_DIR=../reports/figures/
MODEL_DIR=../models/
EXTERNAL_DIR=../data/external/
PRODUCTION_DIR=/home/PRODUCTION/

In [1]:

## 1. Import libraries and modules. Load env variables
import os
import os
import sys
module_path = os.path.abspath(os.path.join('..'))
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
#import seaborn as sns
import zipfile
import sqlite3
from datetime import date, datetime, timezone
from dotenv import find_dotenv, load_dotenv

import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import signature_constants
from tensorflow.python.saved_model import signature_def_utils
from tensorflow.python.saved_model import tag_constants
from tensorflow.python.saved_model.utils import build_tensor_info

from sklearn.preprocessing import MinMaxScaler
#from IPython.display import Image

#Reads the key,value pair from .env and adds them to environment variable 
load_dotenv(find_dotenv())

# Check the env variables exist. Check as many variables as you need
raw_msg = "Set your raw data absolute path in the .env file at project root"
assert "RAW_DIR" in os.environ, raw_msg
data_msg = "Set your processed data absolute path in the .env file at project root"
assert "DATA_DIR" in os.environ, data_msg
interim_msg = "Set your interim data absolute path in the .env file at project root"
assert "INTERIM_DIR" in os.environ, interim_msg

# Load env variables
proj_dir = os.path.expanduser(os.environ.get("PROJ_DIR"))
data_dir = os.path.expanduser(os.environ.get("DATA_DIR"))
raw_dir = os.path.expanduser(os.environ.get("RAW_DIR"))
interim_dir = os.path.expanduser(os.environ.get("INTERIM_DIR"))
processed_dir = os.path.expanduser(os.environ.get("PROCESSED_DIR"))
figures_dir = os.path.expanduser(os.environ.get("FIGURES_DIR"))
model_dir = os.path.expanduser(os.environ.get("MODEL_DIR"))
external_dir = os.path.expanduser(os.environ.get("EXTERNAL_DIR"))
production_dir = os.path.expanduser(os.environ.get("PRODUCTION_DIR"))

# Import our project modules
#
#Add src/app to the PATH
#Ram sys.path.append(os.path.join(proj_dir,"src/app"))

#Add src/data to the PATH
sys.path.append(os.path.join(proj_dir,"src/data"))
#Ram import make_dataset as md

#Add src/visualization to the PATH
#Ram sys.path.append(os.path.join(proj_dir,"src/visualization"))
#Ram import visualize as vs

#Data files
#raw_data = 
#interim_data = 

#Global configuration variables
# Send models to production env. folder (True: send. False: Do not send)
to_production = True

/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)
/home/parrondo/anaconda3/envs/deeptrading/lib/python3.5/importlib/_bootstrap.py:222: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
  return f(*args, **kwds)

Ingest raw data

Download raw datasets

The great people at HistData.com have set up the infrastructure necessary to provide you FOREX data for free. This is awesome and if possible, you should donate or purchase some of their services to help them. There exist several tools contained on the internet to download the data, but all of them need your careful attention. For example:

• https://github.com/philipperemy/FX-1-Minute-Data
• https://github.com/xeb/forex-histdata-etl
• https://gist.github.com/EvianZhow/93b30edb5e1ac44f3dd2de7ef9a543d9

I include forex time series zipped files needed in this tutorial

In [2]:

## 2. Download raw data sets

#This point must be adapted for each project

Download .csv data file from HistData.com and save in ../data/raw dir

Basic pre-process data

Machine Learning time-series algorithms usually require data to be into a single text file in tabular format, with each row representing a timestamp of the input dataset and each column one of its features.

“Prepare” data for Machine Learning is a complex task depending on where the data is stored and where it is obtained from. And doubtless, it is one of the most time-consuming task. Often the Forex data is not available in a single file. They may be distributed across different sources like multiple compressed CSV files, spreadsheets or plain text files, normalized in database tables, or even in NoSql database like MongoDB. So we need a tool to stage, filter, transform when necessary, and finally export to a single flat, text CSV file.

If your Forex data is small and the changes are simple such as adding a derived field or new events you can use a spreadsheet, make the necessary changes, and then export it to a CSV file. Certainly, not too professional. But when the changes are more complex; e.g., joining several sources, filtering a subset of the data, or anaging a large number of timestamp rows, you might need a more powerful tool like an RDBMS. MySQL is a great one and it’s free and opensourced. In this tutorial, we have selected SQLite which is enough for our purpose and data size. Here we treat several compressed .csv files distributed in different folders, which is very usual in real trading. If the data size that we are managing is in the terabytes, then we should consider Hadoop. But trust us, that is another story.

Clean Data

The steps and techniques for data cleaning will vary from dataset to dataset. However, we provide a reliable starting framework that can be used every time. The common steps are:

1. Remove unwanted observations
   • Duplicate observations
   • Irrelevant observations: those that don’t actually fit the problem that we’re trying to solve.

1. Fixing structural errors: mislabeled classes, i.e. separate classes that should really be the same
2. Filtering observations: filter Unwanted Outliers

In [3]:

# All available instruments to trade with
all_symbols = ["EURUSD" ,
           "AUDNZD" ,
           "AUDUSD" ,
           "AUDJPY" ,
           "EURCHF" ,
           "EURGBP" ,
           "EURJPY" ,
           "GBPCHF" ,
           "GBPJPY" ,
           "GBPUSD" ,
           "NZDUSD" ,
           "USDCAD" ,
           "USDCHF" ,
           "USDJPY" ,
           "CADJPY" ,
           "EURAUD" ,
           "CHFJPY" ,
           "EURCAD" ,
           "AUDCAD" ,
           "AUDCHF" ,
           "CADCHF" ,
           "EURNZD" ,
           "GBPAUD" ,
           "GBPCAD" ,
           "GBPNZD" ,
           "NZDCAD" ,
           "NZDCHF" ,
           "NZDJPY" ,
           "XAGUSD" ,
           "XAUUSD" ,
           "SPXUSD" ] # type=str, symbol list using format "EURUSD" "EURGBP"

In [4]:

#
# Here we select the features to work with
#
features = ['timestamp', 'open', 'high', 'low', 'close', 'volume']

In [5]:

# Clean database table
DATABASE_FILE = processed_dir+"Data.db"
def initialize_db(self):
    with sqlite3.connect(DATABASE_FILE) as connection:
        cursor = connection.cursor()
        cursor.execute('CREATE TABLE IF NOT EXISTS History (timestamp INTEGER,'
                       'symbol VARCHAR(20), high FLOAT, low FLOAT,'
                       'open FLOAT, close FLOAT, volume FLOAT, '
                       'quoteVolume FLOAT, weightedAverage FLOAT,'
                       'PRIMARY KEY (timestamp, symbol));')
        connection.commit()

initialize_db(DATABASE_FILE)
conn = sqlite3.connect(DATABASE_FILE)

In [6]:

# Create the dataframe
columns = ["timestamp", "symbol", "open", "high", "low", "close", "volume", "quoteVolume", "weightedAverage"]
dtype = {"timestamp":"INTEGER",
         "symbol":"VARCHAR(20)",
         "open":"FLOAT",
         "high":"FLOAT",
         "low":"FLOAT",
         "close":"FLOAT",
         "volume":"FLOAT",
         "quoteVolume":"FLOAT",
         "weightedAverage":"FLOAT"}
df0 = pd.DataFrame(columns=columns)

# Write dataframe to sqlite database
df0.to_sql("History", conn, if_exists="replace", index=False, dtype=dtype)

Remove unwanted observations

Duplicate observations

Left in blanck intentionally

Irrelevant observations:

Those that don’t actually fit the problem that we’re trying to solve.

Fixing structural errors: mislabeled classes, i.e. separate classes that should really be the same

Left in blanck intentionally

Filtering observations: filter Unwanted Outliers

Left in blanck intentionally

Missing Data

Handling missing data

Missing categorical data

The best way to handle missing data for categorical features is to simply label them as ’Missing’! (Not used in this notebook).

• We add a new class for the feature.
• This tells the algorithm that the value was missing.
• This gets around the technical requirement for no missing values.

Missing numeric data

For missing numeric data, we flag and fill the values.

• Flag the observation with an indicator variable of missingness: “Missingness Indicator”.
• Fill the original missing value with 0 just to meet the technical requirement of no missing values.

By using this technique of flagging and filling, we are allowing the algorithm to estimate the optimal constant for missingness, instead of just filling it in with some other stimated value.

Features extraction

Following are the features consider for training the model to.

timestamp: INTEGER,
symbol: VARCHAR(20)
open: FLOAT
high: FLOAT
low: FLOAT
close: FLOAT

In [7]:

#
# Database population
#
#
# All price instrument in cash currency base
#

# Symbol selection
symbols = all_symbols

# Initialicing dataframes           
df1 = pd.DataFrame().iloc[0:0]
df2 = pd.DataFrame().iloc[0:0]

# Managing diferent raw data files from several folders
for symbol in symbols:
    compressedfile = os.path.join(raw_dir,symbol.lower(),'HISTDATA_COM_ASCII_'+symbol+'_M1_2017.zip')
    zf = zipfile.ZipFile(compressedfile) # having .csv zipped file
    inputfile = 'DAT_ASCII_'+symbol+'_M1_2017.csv'
    print("inputfile: ",inputfile)
    #df1 = pd.read_csv(inputfile, names=['date', 'open', 'high', 'low', 'close', 'volume'],index_col=0, parse_dates=True, delimiter=";")
    
    df1 = pd.read_csv(zf.open(inputfile), header=None,
                      names=['timestamp', 'open', 'high', 'low', 'close', 'volume'],
                      index_col=0, parse_dates=True,sep=';') # reads the csv and creates the dataframe called "df1"
    
    # Resampling data from 1Min to desired Period
    df2 =  df1["open"].resample('60Min').ohlc()
        
    # Convert pandas timestamps in Unix timestamps:
    df2.index = df2.index.astype(np.int64) // 10**9

    # Insert new columns with the instrument name and their values
    df2.insert(loc=0, column='symbol', value=symbol)

    #Only for compatibility with stocks code (optional, you may want to remove this fields from database)
    df2['volume']=1000.
    df2['quoteVolume']=1000.
    df2['weightedAverage']=1.

    # Reset index to save in database
    df2=df2.reset_index()
    
    #Filling gaps forward
    df2 = df2.fillna(method='pad')
        
    # Save to database (Populate database)
    df2.to_sql("History", conn, if_exists="append", index=False, chunksize=1000)

# Liberate memory
del df1
del df2

inputfile:  DAT_ASCII_EURUSD_M1_2017.csv
inputfile:  DAT_ASCII_AUDNZD_M1_2017.csv
inputfile:  DAT_ASCII_AUDUSD_M1_2017.csv
inputfile:  DAT_ASCII_AUDJPY_M1_2017.csv
inputfile:  DAT_ASCII_EURCHF_M1_2017.csv
inputfile:  DAT_ASCII_EURGBP_M1_2017.csv
inputfile:  DAT_ASCII_EURJPY_M1_2017.csv
inputfile:  DAT_ASCII_GBPCHF_M1_2017.csv
inputfile:  DAT_ASCII_GBPJPY_M1_2017.csv
inputfile:  DAT_ASCII_GBPUSD_M1_2017.csv
inputfile:  DAT_ASCII_NZDUSD_M1_2017.csv
inputfile:  DAT_ASCII_USDCAD_M1_2017.csv
inputfile:  DAT_ASCII_USDCHF_M1_2017.csv
inputfile:  DAT_ASCII_USDJPY_M1_2017.csv
inputfile:  DAT_ASCII_CADJPY_M1_2017.csv
inputfile:  DAT_ASCII_EURAUD_M1_2017.csv
inputfile:  DAT_ASCII_CHFJPY_M1_2017.csv
inputfile:  DAT_ASCII_EURCAD_M1_2017.csv
inputfile:  DAT_ASCII_AUDCAD_M1_2017.csv
inputfile:  DAT_ASCII_AUDCHF_M1_2017.csv
inputfile:  DAT_ASCII_CADCHF_M1_2017.csv
inputfile:  DAT_ASCII_EURNZD_M1_2017.csv
inputfile:  DAT_ASCII_GBPAUD_M1_2017.csv
inputfile:  DAT_ASCII_GBPCAD_M1_2017.csv
inputfile:  DAT_ASCII_GBPNZD_M1_2017.csv
inputfile:  DAT_ASCII_NZDCAD_M1_2017.csv
inputfile:  DAT_ASCII_NZDCHF_M1_2017.csv
inputfile:  DAT_ASCII_NZDJPY_M1_2017.csv
inputfile:  DAT_ASCII_XAGUSD_M1_2017.csv
inputfile:  DAT_ASCII_XAUUSD_M1_2017.csv
inputfile:  DAT_ASCII_SPXUSD_M1_2017.csv

In [8]:

# Committing changes and closing the connection to the database file
conn.commit()
conn.close()

Load data from database

In [9]:

# Load dataset (In this case reading the database)
DATABASE_FILE=processed_dir+"Data.db"
conn = sqlite3.connect(DATABASE_FILE)
df = pd.read_sql_query("select * from History;", conn)
df=df.drop(["index"], axis=1, errors="ignore")

In [10]:

df

Out[10]:

269001 rows × 9 columns In [11]:

# Create the apropiate features dataframe (In this case is open prices of all symbols)
symbols = all_symbols
df2 = pd.DataFrame()
for symbol in symbols:
    df1 = df.loc[(df['symbol'] == symbol),['timestamp','open']]
    df1.columns=['timestamp',symbol]
    # Setting the timestamp as the index
    df1.set_index('timestamp', inplace=True)
    
    # Convert timestamps to dates but it's not mandatory
    #df1.index = pd.to_datetime(df1.index, unit='s')

    # Just perform a join and that's it
    df2 = df2.join(df1, how='outer')

# Filling the remaining gaps backguards (the initial gaps has not before value)
df2 = df2.fillna(method='bfill')

# Independent variables data
X_raw = df2

# Dimensions of dataset
print("Dimensions of dataset")
n = X_raw.shape[0]
p = X_raw.shape[1]
print("n=",n,"p=",p)

# Drop timestamp variable (only when necessary)
#Ram print("Drop timestamp variable")
#Ram X_raw = X_raw.drop(['timestamp'], 1)

Dimensions of dataset
n= 8679 p= 31

In [12]:

X_raw

Out[12]:

8679 rows × 31 columns In [13]:

#A quick look at the dataframe time series using pyplot.plot(X_raw['XAUUSD']):
plt.plot(X_raw['XAUUSD'])

Out[13]:

[<matplotlib.lines.Line2D at 0x7fad67dd6128>]

In [14]:

# Target
# We use as target one of the symbols rate, i.e. "XAUUSD". That is we try to predict next value of XAUUSD
lag = -1
y_raw = df2.loc[:,"XAUUSD"].shift(periods=lag)

In [15]:

y_raw

Out[15]:

timestamp
1483322400    1150.67
1483326000    1150.67
1483329600    1150.67
1483333200    1150.67
1483336800    1150.67
1483340400    1150.67
1483344000    1150.67
1483347600    1150.67
1483351200    1150.67
1483354800    1150.67
1483358400    1150.67
1483362000    1150.67
1483365600    1150.67
1483369200    1150.67
1483372800    1150.67
1483376400    1150.67
1483380000    1150.30
1483383600    1150.69
1483387200    1156.09
1483390800    1157.15
1483394400    1156.88
1483398000    1157.33
1483401600    1157.77
1483405200    1157.46
1483408800    1153.10
1483412400    1151.48
1483416000    1148.10
1483419600    1146.78
1483423200    1148.56
1483426800    1148.72
               ...   
1514458800    1294.27
1514462400    1294.02
1514466000    1294.31
1514469600    1294.68
1514473200    1294.38
1514476800    1294.69
1514480400    1294.12
1514484000    1293.97
1514487600    1294.64
1514491200    1295.58
1514494800    1295.80
1514498400    1295.40
1514502000    1295.65
1514505600    1295.94
1514509200    1295.56
1514512800    1295.06
1514516400    1296.39
1514520000    1296.81
1514523600    1297.02
1514527200    1297.02
1514530800    1296.58
1514534400    1299.53
1514538000    1301.33
1514541600    1302.56
1514545200    1303.34
1514548800    1306.05
1514552400    1303.65
1514556000    1302.93
1514559600    1303.20
1514563200        NaN
Name: XAUUSD, Length: 8679, dtype: float64

In [16]:

#
# Removal of Null values**
# Now since there still exists 'NaN' values in our target dataframe, and these are Null values,
# we have to do something about them. In here, I will just do the naive thing of replacing these NaNs
# with previous value because it is only the last value an error is negligible as such:

# Filling gaps forward
y_raw = y_raw.fillna(method='pad')
y_raw

# Drop timestamp variable (only when necessary)
#Ram print("Drop timestamp variable")
#Ram y_raw = data.drop(['timestamp'], 1)

Out[16]:

timestamp
1483322400    1150.67
1483326000    1150.67
1483329600    1150.67
1483333200    1150.67
1483336800    1150.67
1483340400    1150.67
1483344000    1150.67
1483347600    1150.67
1483351200    1150.67
1483354800    1150.67
1483358400    1150.67
1483362000    1150.67
1483365600    1150.67
1483369200    1150.67
1483372800    1150.67
1483376400    1150.67
1483380000    1150.30
1483383600    1150.69
1483387200    1156.09
1483390800    1157.15
1483394400    1156.88
1483398000    1157.33
1483401600    1157.77
1483405200    1157.46
1483408800    1153.10
1483412400    1151.48
1483416000    1148.10
1483419600    1146.78
1483423200    1148.56
1483426800    1148.72
               ...   
1514458800    1294.27
1514462400    1294.02
1514466000    1294.31
1514469600    1294.68
1514473200    1294.38
1514476800    1294.69
1514480400    1294.12
1514484000    1293.97
1514487600    1294.64
1514491200    1295.58
1514494800    1295.80
1514498400    1295.40
1514502000    1295.65
1514505600    1295.94
1514509200    1295.56
1514512800    1295.06
1514516400    1296.39
1514520000    1296.81
1514523600    1297.02
1514527200    1297.02
1514530800    1296.58
1514534400    1299.53
1514538000    1301.33
1514541600    1302.56
1514545200    1303.34
1514548800    1306.05
1514552400    1303.65
1514556000    1302.93
1514559600    1303.20
1514563200    1303.20
Name: XAUUSD, Length: 8679, dtype: float64

Split data

In [17]:

# split into train and test sets

# Total samples
nsamples = n

# Splitting into train (70%) and test (30%) sets
split = 70 # training split% ; test (100-split)%
jindex = nsamples*split//100 # Index for slicing the samples

# Samples in train
nsamples_train = jindex

# Samples in test
nsamples_test = nsamples - nsamples_train
print("Total number of samples: ",nsamples,"\nSamples in train set: ", nsamples_train,
      "\nSamples in test set: ",nsamples_test)

# Here are train and test samples
X_train = X_raw.values[:jindex, :]
y_train = y_raw.values[:jindex]

X_test = X_raw.values[jindex:, :]
y_test = y_raw.values[jindex:]

print("X_train.shape = ", X_train.shape, "y_train.shape =", y_train.shape, "\nX_test.shape =  ",
      X_test.shape, "y_test.shape = ", y_test.shape)

Total number of samples:  8679 
Samples in train set:  6075 
Samples in test set:  2604
X_train.shape =  (6075, 31) y_train.shape = (6075,) 
X_test.shape =   (2604, 31) y_test.shape =  (2604,)

In [18]:

#X_train as dataframe (optional, only for printing. See note in the beginning)
X_Train = pd.DataFrame(data=X_train)
X_Train.columns = X_raw.columns
print("X_train")
X_Train

X_train

Out[18]:

6075 rows × 31 columns In [19]:

#X_test as dataframe (optional, only for printing. See note in the beginning)
X_Test = pd.DataFrame(data=X_test)
X_Test.columns = X_raw.columns
print("X_test")
X_Test

X_test

Out[19]:

2604 rows × 31 columns

Transform features

Note

Be careful not to write X_test_std = sc.fit_transform(X_test) instead of X_test_std = sc.transform(X_test). In this case, it wouldn’t make a great difference since the mean and standard deviation of the test set should be (quite) similar to the training set. However, this is not always the case in Forex market data, as has been well established in the literature. The correct way is to re-use parameters from the training set if we are doing any kind of transformation. So, the test set should basically stand for “new, unseen” data.

In [20]:

# Scale data
from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
X_train_std = sc.fit_transform(X_train)
X_test_std = sc.transform(X_test)

y_train_std = sc.fit_transform(y_train.reshape(-1, 1)) #Reshape to get the right tensor dimension 
y_test_std = sc.transform(y_test.reshape(-1, 1)) #Reshape to get the right tensor dimension

In [21]:

print("Mean:",sc.mean_)
print("Variance",sc.var_)

Mean: [1248.6151177]
Variance [1253.0292182]

In [22]:

#X_train_std as dataframe (optional, only for printing. See note in the beginning)
X_Train_std = pd.DataFrame(data=X_train_std)
X_Train_std.columns = X_Train.columns
print("X_train_std")
X_Train_std

X_train_std

Out[22]:

6075 rows × 31 columns In [23]:

#X_train_std as dataframe (optional, only for printing. See note in the beginning)
X_Test_std = pd.DataFrame(data=X_test_std)
X_Test_std.columns = X_Test.columns

print("X_test_std")
X_Test_std

X_test_std

Out[23]: