reproducible-experiments archivos

Tag: <span>reproducible-experiments</span>

Quant Trading Project Structure

2019-05-252019-05-26
by parrondo

The Quant Trading Project Structure is a logical and reasonably simple project structure for developing quantitative trading work like strategies and research works.

Why use this simple project structure?

When we think about quantitative trading research, we often think just about the resulting strategies, reports, or graph visualizations. While these final products are generally the main objects, it’s easy to focus on making the products look nice and convincing to the third party, even for yourself and forgetting the quality of the code that generates them. Trading research science code quality is about correctness, standardization, and reproducibility.

It is not a secret that good analysis is often the result of very scattered and random explorations. Tentative experiments and rapid testing approaches that may not work are part of the process to get good trading strategies.

Having said that, once it is started, it is not a process that lends itself to thinking carefully about the structure of your code or project design, so it is best to start with a logical and clean structure and stick to it at all times.

If you ever tried to reproduce an analysis you or your team did a few months ago or even a few years ago you know what I mean. You may have written the code, but now it is impossible to decrypt whether you should use first_test.py, first_test_01.py or new_first_test.py to get things done.

Welcome to Quant Trading Project Structure!

Below we expose a simple project structure for quantitative traders who want to develop new strategies or make research and development works. It is based in the following works and we use either of them, depending on the work‘s aim:

cookiecutter-data-science

cookiecutter-deeplearning

luigi_data_science_project_cookiecutter

Index

Reproducibility is the name of the game

There are two basic reasons to make your research reproducible. The first is to show evidence of the correctness of your results. The second one is to enable others to make use of our methods and results. So to achieve it we need to recognize that:

A well-defined standard project structure means that a newcomer can begin to understand an analysis without having to look into an extensive documentation.
It also means that you do not necessarily have to read 100% of the code before you know where to look for very specific things.
A well-organized code tends to self-document, since the organization itself provides the context for its code without too much overhead.
Your team collaborates more easily with you on this analysis
Learn yourself from your analysis about the research process
Feel more confident in the conclusions of the project

You will get out what you put in: Provenance

The term “data provenance” refers to a record that explains the origin of a piece of data (in a database, document or repository) along with an explanation of how and why it arrived at the current location. Trading quants need the provenance like the trees the sunlight.

Example: In an application like quantitative strategies, a lot of data is derived from public databases or brokers, which in turn might be derived from papers but after some clearing and transformations (only the most significant data are put into the public database), which are derived from data markets. A good provenance record will keep the whole history for each piece of data. Here are some questions we have learned to ask with a sense of existential fear:

Are we supposed to go in and join the feature column X to the data before we get started (raw data) or did that come from one of our notebooks?
Come to think of it, which notebook do we have to run first before running the plotting code: was it “process data” or “clean data”?
Where did the shapefiles get downloaded from for the price plots?
Etcetera, etcetera, …

These types of questions are painful and are symptoms of a disorganized, foolish project. A good project structure encourages practices that make it easier to come back to old work. You should make sure that it is clear on how you came to certain conclusions. Just because something makes sense to you, it is not clear that it does for others (yourself next month). When making claims based on your analysis, you must link them to the data on which you base them. This is to ensure that you or others who read your work can easily follow the steps you took from the data to the conclusions. Having a complete context makes it easy to understand the work process and relate to it by reproducing it.

Another benefit is that all raw data remains available to you and, if you wish, for others, to access and consult. This helps transparency and, again, makes it easier for others to understand your workflow.

Getting started

We’ve created a data science like bash script for Linux to start the template for projects in Python. This script contains the three above template types. Your analysis doesn’t have to be in Python, but the templates do provide some Python boilerplates.

Requirements

Python 2.7 or 3.5
cookiecutter Python package >= 1.4.0: pip install cookiecutter
Linux OS (We use Centos 7 but whatever you want will do the job)

name: sphinx
dependencies:
  - python=3.5   # or 2.7
  - pip:
    - sphinx==1.5.6
    - Flask-Sphinx-Themes *(optional, is only an example!)*
    - recommonmark

and execute:

$ conda env create --file environment.yml
$ source activate sphinx

The Script

In the new_project.sh script there are simple instructions to start a new project from scratch.

#!/bin/bash

#2018, Parrondo https://github.com/parrondo/quant-trading-project-structure
#Starting a new project is as easy as running this command at the command line. No need to create a directory first,
#the cookiecutter will do it for you.

#Conda python environment activation where cookiecutter lives.
source activate cookiecutter



echo -e "What kind of new project do you want? \nData-Science [ds] \nDeepLearning [dl] \nData-Science-Luigi [lu] \nSphinx [sp]: "
read answer
case $answer in

        ds|Ds|dS|DS)
                echo "Agreed, you want a Data-Science new project"
                #Creating files and project folder structure with cookiecutter-data-science
                cookiecutter https://github.com/drivendata/cookiecutter-data-science
                ;;

        dl|Dl|dL|DL)
                echo echo "Agreed, you want a DeepLearning new project"
                #Creating files and project folder structure with cookiecutter-deeplearning
                cookiecutter https://github.com/tdeboissiere/cookiecutter-deeplearning.git
                ;;

        lu|Lu|lU|LU)
                echo "Agreed, you want a Data-Science with Luigi new project"
                cookiecutter https://github.com/ffmmjj/luigi_data_science_project_cookiecutter
                ;;

        sp|Sp|sP|SP)
                echo "Agreed, you want a Sphinx documentation new project"
                cookiecutter https://github.com/carschar/cookiecutter-sphinx.git
                ;;

        *) echo "Invalid input"
            ;;
esac

Starting a new project

Starting a new project is as easy as running this command at the command line. No need to create a directory first, the cookiecutter you select in the dialog will do it for you. Remember to change the file new_project.sh as an executable file, also you’ll place the file into the parent folder of the new project structure. Then, execute:

$ ./new_project.sh

this command will create my_project structure (see example below)

Example of Directory structure (Data Science type project)

After finishing setup, some build folders should be ignored during git commit. By default, these folders are as follow:

The resulting directory structure

The directory structure of your new project looks like this:

my_project
│
├── LICENSE
├── Makefile           <- Makefile with commands like `make data` or `make train`
├── README.md          <- The top-level README for developers using this project.
├── data
│   ├── external       <- Data from third party sources.
│   ├── interim        <- Intermediate data that has been transformed.
│   ├── processed      <- The final, canonical data sets for modeling.
│   └── raw            <- The original, immutable data dump.
│
├── docs               <- A default Sphinx project; see sphinx-doc.org for details
│
├── models             <- Trained and serialized models, model predictions, or model summaries
│
├── notebooks          <- Jupyter notebooks. Naming convention is a number (for ordering),
│                         the creator's initials, and a short `-` delimited description, e.g.
│                         `1.0-jqp-initial-data-exploration`.
│
├── references         <- Data dictionaries, manuals, and all other explanatory materials.
│
├── reports            <- Generated analysis as HTML, PDF, LaTeX, etc.
│   └── figures        <- Generated graphics and figures to be used in reporting
│
├── requirements.txt   <- The requirements file for reproducing the analysis environment, e.g.
│                         generated with `pip freeze > requirements.txt`
│
├── src                <- Source code for use in this project.
│   ├── __init__.py    <- Makes src a Python module
│   │
│   ├── data           <- Scripts to download or generate data
│   │   └── make_dataset.py
│   │
│   ├── features       <- Scripts to turn raw data into features for modeling
│   │   └── build_features.py
│   │
│   ├── models         <- Scripts to train models and then use trained models to make
│   │   │                 predictions
│   │   ├── predict_model.py
│   │   └── train_model.py
│   │
│   └── visualization  <- Scripts to create exploratory and results oriented visualizations
│       └── visualize.py
│
└── tox.ini            <- tox file with settings for running tox; see tox.testrun.org

Data is immutable

Most important: don’t ever edit your carefully and usually expensive downloaded raw data, especially not manually, and especially not in Excel or whatever worksheet.
Don’t overwrite your raw data ever.
Don’t save multiple versions of the raw data.
Treat the data (and its format) as immutable.
The code you write should move the raw data through a pipeline to your final analysis (Luigi is perfect for this task).
You shouldn’t have to run all of the steps every time you want to make a new figure, but anyone should be able to reproduce the final products with only the code in src and the data in data/raw.

Also, if data is immutable, it doesn’t need source control in the same way that code does. Therefore, by default, the data folder is included in the .gitignore file. If you have a small amount of data that rarely changes, you may want to include the data in the repository. Github currently warns if files are over 50MB and it reject files over 100MB.

Notebooks are for exploration and communication

Notebook packages like the Jupyter notebook, are very effective for exploratory data analysis. However, these tools can be less effective for reproducing an analysis. When we use notebooks in our work, we identify the notebooks files. For example, 01-ram-exploratory.ipynb contains initial explorations, whereas 01-ram-reports.ipynb is final work that can be exported as html to the reports directory.

Since notebooks are challenging objects for source control (e.g., diffs of the json are often not human-readable and merging is near impossible), we recommended not collaborating directly with others on Jupyter notebooks.

There are two steps we recommend for using notebooks effectively:

Follow a naming convention that shows the owner and the order the analysis was done in. We use the format: <step>-<user>-<description>.ipynb (e.g., “01-ram-data-preparation.ipynb”, “03-ram-returns-visualization.ipynb”).
Refactor the good parts. Don’t write code to do the same task in multiple notebooks. If it’s a data preprocessing task, put it in the pipeline at src/data/make_dataset.py and load data from data/interim.

Build from a Conda environment

The first step in reproducing an analysis is always reproducing the computational environment it was run in. You need the same tools, the same libraries, and the same versions to make everything play nicely together. One effective approach to this is by using Anaconda. By listing all of your requirements in the repository (it is include a requirements.txt file) you can easily track the packages needed to recreate the analysis. Here is an example of a good workflow:

1. Create an environment.yml file in the project folder. Typically the environment name, ENV_NAME, will be the same as the folder name. At minimum, it will specify the version of Python you want to use.
2. $ conda env create.
3. $ source activate ENV_NAME (to activate the environtment)
4. (myenv)$ pip install the packages that your analysis needs
5. Run pip freeze > requirements.txt (to pin the exact package versions used to recreate the analysis). Update environment.yml with requirements.txt (delete requirements.txt).
6. If you find you need to install another package:
   run pip freeze > requirements.txt again, update environment.yml with requirements.txt (delete requirements.txt), and commit the changes to version control.

Usually, you have more complex requirements for recreating your environment, therefore you should consider a virtual machine based approach such as Docker or Vagrant. Both of these tools use text-based formats (Dockerfile and Vagrantfile respectively) and consequently you can easily add to source control to describe how to create a virtual machine with the requirements you need. Once you are familiarized with this virtual machine techniques, they work like a charm.

If you need orchestration then use, for example, Kubernetes, Mesos, and Docker Swarm. These last ones are more for production scale and so, out of the scope of this post.

Secrets and configuration should be kept out of version control

You really don’t want to leak your username and password on Github. Here’s how to do this:

Store your secrets and config variables in a special file

Create a .env file in the project root folder. Thanks to the .gitignore, this file should never get committed into the version control repository. Here’s an example:

# example .env file
DATABASE_URL=mongodb://username:password@localhost:5432/dbname
AWS_ACCESS_KEY=myaccesskey
AWS_SECRET_ACCESS_KEY=mysecretkey
OTHER_VARIABLE=something

Use a package to load these variables automatically

If you look at the stub script in src/data/make_dataset.py, it uses a package called python-dotenv to load up all the entries in this file as environment variables so they are accessible with os.environ.get. Here’s an example snippet adapted from the python-dotenv documentation:

# src/data/dotenv_example.py
import os
from dotenv import load_dotenv, find_dotenv

# find .env automagically by walking up directories until it's found
dotenv_path = find_dotenv()

# load up the entries as environment variables
load_dotenv(dotenv_path)

database_url = os.environ.get("DATABASE_URL")
other_variable = os.environ.get("OTHER_VARIABLE")

Default folder structure quasi-immutability

if you want to keep this structure broadly applicable to many different kinds of projects, then, the best approach is to adapt the folders to your project. Nonetheless, you must be very conservative in changing the default basic structure for all projects. This is what we say quasi-immutable.

Now you have a folder-layout label specifically for your trading development works to add, subtract, rename, or move folders around.

Be careful and be consistent!

Don’t forget to clone or download the script from here: quant-trading-project-structure

Be careful and be consistent!

Final notes

This post is an actualization of my Github work:

https://parrondo.github.io/quant-trading-project-structure/

Don’t forget to clone or download the script from here: quant-trading-project-structure

References

GitHub. (2019). drivendata/cookiecutter-data-science. [online] Available at: https://github.com/drivendata/cookiecutter-data-science [Accessed 25 May 2019].
GitHub. (2017). ffmmjj/luigi_data_science_project_cookiecutter. [online] Available at: https://github.com/ffmmjj/luigi_data_science_project_cookiecutter [Accessed 23 Oct. 2017].
GitHub. (2017). tdeboissiere/cookiecutter-deeplearning. [online] Available at: https://github.com/tdeboissiere/cookiecutter-deeplearning [Accessed 23 Oct. 2017]. Ropensci.github.io. (2017).
Reproducibility Guide. [online] Available at: http://ropensci.github.io/reproducibility-guide/sections/introduction/ [Accessed 23 Oct. 2017]. Conda.io. (2017).
Conda — Conda documentation. [online] Available at: https://conda.io/docs/index.html [Accessed 25 May 2019].
GitHub. (2017). theskumar/python-dotenv. [online] Available at: https://github.com/theskumar/python-dotenv [Accessed 25 May 2019].

Black Belt

Artificial Intelligence Trading Systems

2019-05-142019-09-02
by parrondo

Introduction

“By far, the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.”

Eliezer Yudkowsky

A few days ago I was talking with a good friend about artificial intelligence (AI). He told me “I think the AI is overvalued and quite inflated, really AI has not had big changes since I left the university (30 years ago!).

I was surprised by that statement because this friend is a cultured person with a good university education. So I thought that artificial intelligence is still a great unknown even in our trading world.

This post is the first of a series on AI fundamentals. After reading it you will be able to better understand the naming and structure of AI and its application to financial markets.

Continue reading to have a clear vision of the AI.

Nomenclature

Let’s get acquainted a little with the necessary terminology.

The names or terms comprising AI is very extensive. Perhaps the better compilation is that of Wikipedia:

https://en.wikipedia.org/wiki/Glossary_of_artificial_intelligence

This glossary is very broad and will increase since the field is very active. The good news is that you do not have to learn it all, only what you have to work with.

Reading what follows below will give you a good starting point.

Artificial Intelligence

Since the first imagining of so-called “thinking machines” or “artificial intelligence” (AI), there has been a rich debate as to whether machines would, could, or should supplant humans or only enhance humans. In theory, AI could be designed to do either task [1].

The Artificial Intelligence Waves

DARPA’s John Launchbury first coined the notion of the “three waves of AI”[2], and Six Kin Development’s Scott Jones[3] recently expounded upon that to add timelines, examples, and a fourth wave to account for AI, we have adapted it to trading systems as illustrated in the chart below.

The First Wave: PURELY REACTIVE

DARPA defines the first wave of AI as enabling “reasoning over narrowly defined problems,” but with a poor level of certainty.

This is the most basic form of AI. It doesn’t store memories or past experiences and It acts directly for what it sees. Overall, It doesn’t have a concept of a wider world.

To date, AI has developed many tools and techniques to perform very specific functions. Basically in the form of programs. However, programs and expert systems can be very fragile and must follow specific rules. Human designers write logic line by line.

This is the so-called “1st wave AI”. Even a simple standard computer can be considered to fall into this category, as they are designed and coded to fulfill a specific purpose. For example, a Python program that makes a simple trend following strategy based on exponential averages. It is a rudimentary form of AI because it replaces what the human brain would do with a computer algorithm: a first step to emulate human cognition.

Examples: Trading system program coded with strategies like reveled in the book:

“151 Trading Strategies” of Kakushadze and Serur (2018)[4].

And you want something of software, here there is an example of an open source trading platform writing in python with trading strategies of the first wave:

https://www.backtrader.com/docu/strategy.html

This last software is excellent and you can use it to implement more modern systems as those related to the below “Second Wave”.

Nowadays, implementing strategies of the First Wave type is almost always an absolute waste of time.

The Second Wave: LIMITED MEMORY

“Econometrics might be good enough to succeed in financial academia (for now), but succeeding in practice requires ML.”

Marcos López de Prado (2018)

The second wave enables creating statistical models and training them on big data, albeit with minimal reasoning.

The second wave of AI is currently in full swing. It consists of statistical systems that use new and powerful techniques and computer resources to perform the statistical recognition of objects and search for patterns in large batches of data.

Neural network algorithms were developed for the first time in the 1950s [5]. These systems tried to reflect the organization of the human brain by mimicking the connectivity of neurons and synapses at a very simplistic level.

Artificial neural networks (ANN) is also called connectionist models. “Neurons” within ANN have a simple input / output structure and perform a processing layer between each subsequent “synaptic” connection layer.

However, the analogy with human brain systems is so weak that most system practitioners now talk more about nodes and layers rather than neurons and synapses.

With the development of the latest deep neural network (DNN) algorithms and the increase of computing power, these systems have become powerful enough to recognize categorical objects and faces.

While it is extremely powerful, there are several drawbacks in these statistical systems.

Drawbacks

There is the requirement of tens of thousands of examples for the system to learn. And Only historical data are available in financial markets.
The limited capacity of the system once trained to perform only specific tasks. For example, a system trained to trade a trending market can do quite well in recognizing trends and trade it, but it will fail completely if it is suddenly asked to trade a sideways market.
These systems do not have general or real-time interactive learning capacity, cannot work with dynamic objectives and contexts, or deal with abstract reasoning or the understanding of language.
These systems develop a black box solution for your specific problem, which makes it extremely difficult to understand and debug errors.

An alternative approach lies in the so-called 3rd wave of AI, which is designed to supplant the problem and develop cognitive reasoning systems based on machines more analogous to human reasoning and capabilities.

Example: Trading system program coded with strategies like revealed in the book:

“Advances in Financial Machine Learning” [6]

Marcos López de Prado (2018)

Here you are some excellent Github examples implemented in python:

hudson-and-thames/mlfinlab

BlackArbsCEO/Adv_Fin_ML_Exercises

The Third Wave: THEORY OF MIND

The third wave which enables machines to adapt to changing situations.

For instance, adaptive reasoning will enable computer algorithms to discern the difference between the use of ‘principal’ and ‘principle’ based on the analysis of surrounding words to help determine context.

The third wave of artificial intelligence designs systems that adapt to the context.

Systems that can reason, learn and adapt to their environments. The Third Wave AI is based on the paradigm of cognitive systems [7] that moves away from the current statistical AI and seeks to understand the fundamental processes.

The main focus in the development of cognitive systems is in the development of cognitive architectures.

An architecture can be thought of as a fixed set of structures and mechanisms or processes. Cognitive behaviors have certain aspects: they are objective-oriented, reflect a rich and complex environment, require a large amount of knowledge, require the use of symbols and abstractions, are flexible and require experience and learning [8].

These cognitive architectures form the template for the development of intelligent software agents to solve problems of the highest level as financial market problems.

Laird et al. [10] have recently published a Standard Model of the Mind that seeks to formally unify the components and processing design of all human-type cognitive functions. This model includes systems based on AI, neuroscience or robotics.

In the figure below I have created a trading system architecture based on the standard model of mind.

This standard model integrates memory, perception and action components with learning, communication and representation processes. The basic fundamentals of the standard models are:

Structure and Processing

Processing produces bounded rationality, not optimality.
There is significant parallel processing (within and across the modules).
Complex behavior arises from a sequence of independent cognitive cycles that operate in its local context.
A cognitive cycle drives behavior via sequential action selection.

Memory and Content

Declarative and procedural long-term memories (LTMs) contain symbol structures and associated quantitative metadata.
Short-term working memory (WM) provides global communication:
1. Composed of rule-like conditions and actions
2. Exerts control by altering contents of WM
Procedural LTM provides global control.
Declarative LTM provides factual knowledge.

Learning

The content of LTM is learnable.
Learning occurs online and incrementally, as a side effect of performance and is often based on an inversion of the flow of information from performance
Procedural learning involves, at least, reinforced learning and procedural composition:
1. Reinforcement learning yields weights over action selection.
2. Procedural composition yields behavioral automation.
Declarative learning involves the acquisition of facts and tunning of metadata.
More complex forms of learning involve combinations of the fixed set of simpler forms of learning.

Perception (Input data) and Action (Motor)

Perception (input data) produces symbol structures with associated metadata in specific WM buffers.
1. There can be many different perception modules with different input modalities and its own buffer.
2. Perceptual learning acquires new patterns and tunes of existing ones.
3. The bottleneck of attention limits the information available in the WM.
4. Top-down information from WM can influence the perception (input).
Action control converts symbol structures in its buffers into external actions
1. There can be multiple action modules.
2. Action learning acquires new patterns and tunes of existing ones.

Standard model reflects a real initial consensus about how could work AI:

Serial and parallel processing
Symbolic and representations
Procedural and declarative memories
Pervasive learning

But still remain incomplete.

In next pots we will continue deepening in this system model.

Example: C-3P0 and R2D2 from Star Wars

The Fourth Wave: SELF- AWARE

The fourth wave accounts for what many believe will be the ultimate phase of AI: human-level intelligence and Super-intelligence.

Talking about this wave is highly speculative, so I won’t. But here you have some indications if you want to:

The Singularity Is Near

Superintelligence: Paths, Dangers, Strategies

How to Create a Mind

You may be wondering at this point…Where are we at this moment?

Obviously, In the second wave!

So let’s look at it more closely.

AI Taxonomy

I have elaborated a machine learning taxonomy picture in order to clarify it. This taxonomy is based on multiples sources from academic and web sites.

One of the classes in taxonomy is Reinforcement Learning which is a very active researching field, so I have adopted a graph from the page [11] which is quite complete. In this site, you can start programming your Reinforcement Learning trading systems. You have an excellent idea of this kind of system looking at the .github repository of Andrew Kismuz [12]

References

1) Grigsby S.S. (2018) Artificial Intelligence for Advanced Human-Machine Symbiosis. In: Schmorrow D., Fidopiastis C. (eds) Augmented Cognition: Intelligent Technologies. AC 2018. Lecture Notes in Computer Science, vol 10915. Springer, Cham

2) Launchbury, J.: A DARPA Perspective on Artificial Intelligence (2017).

https://www. youtube.com/watch?time_continue=5&v=-O01G3tSYpU

3) https://www.sixkin.com/posts/3rd-wave-ai/

4)Z. Kakushadze and J.A. Serur. 151 Trading Strategies. Cham, Switzerland:

Palgrave Macmillan, an imprint of Springer Nature, 1st Edition (2018), XX,

480 pp; ISBN 978-3-030-02791-9. Full version:

https://www.springer.com/us/book/9783030027919.

5) Kleene, S.C.: Representation of events in nerve nets and finite automata. Ann. Math. Stud. 34, 3–41 (1956)

6) ”Advances in Financial Machine Learning” Marcos López de Prado (2018)

7) Langley, P.: The cognitive systems paradigm. Adv. Cogn. Syst. 1, 3–13 (2012)

8) Laird, J.E.: The Soar Cognitive Architecture. MIT Press, Cambridge (2012)

9) Profanter, S.: Cognitive Architectures. Hauptseminar Human-Robot Interaction (2012).

http://profanter.me/static/publications/SeminarCogArch/elaboration.pdf

10) Laird, J.E., Lebiere, C., Rosenbloom, P.S.: A standard model of the mind: toward a common

11) https://github.com/NervanaSystems/coach

12) https://github.com/Kismuz/btgym

Brown Belt

Bases of Scientific Trading Systems

2019-04-282021-04-08
by parrondo

Introduction

What is the secret to get a consistent trading systems?

It’s not an easy question to answer.

But you probably know other people that are able to generate incomes from every strategy. Perhaps not in person but on the web. They are very good individual traders.

How do they do it? Do they know some trading system building secret that you don’t?

Actually, yes, they do!

It may happen behind the scenes, but some people know how to get consistent trading systems with ease.

I’m one of those people ?

And today I’m going to show you how: the bases of the scientific trading systems and their design.

The Scientific trading system (STS)

In the trading literature, there are the following terms and some similar ones: Trading strategy, trading system, trading plan, portfolios theory, asset analysis, High-frequency trading, pricing models or algorithmic trading are some of the terms used in trading. They are methods that traders use to determine when to buy and sell assets in the financial markets.

We will call all of them as Trading Systems. A variety of techniques base them. Technical and fundamental analysis, quantitative methods, statistical, mathematical or any combination of factors.

Scientific trading systems are systems based on the application of the scientific method.

But what exactly is a scientific trading system?

“A scientific trading system is an accurate (that is, reliable, consistent and non-arbitrary) representation of the financial market in order to determine when to buy and sell assets that can be repeatedly tested and verified in accordance with the scientific method, using accepted protocols of market observation, measurement, and evaluation of results.”

Let me elaborate the idea behind STS.

Scientific Trading System as model

A model is a logical framework intended to represent the financial markets reality.

So STS are a specific category of models that fulfill the necessary criteria (see above). One can use language to describe a model, yet the STS is the model and not the description of the model. Black-Scholes Model is a model of future fluctuations (volatility) of a security’s price. It’s a well-known option pricing model. It’s based on factors in current stock price, options strike price, time until expiration, and risk-free interest rates. These objects have associated properties (parameters). The model parameters determine how the theoretical value of European-style options change. This could be used to elaborate a further model (STS) to trade options. The last model can then be tested to see whether it accurately predicts future observations.

Keep reading because I’m about to explain how to design STS to achieve consistency.

Scientific Trading System Design (STSD)

A quick search of Internet resources demonstrates that in reality there is no single uniform description of a trading system design. Some descriptions have four steps; others have six or eight. Some start with a market hypothesis; others start with market observation. Some emphasize one step of the process at the expense of the others. None of them establish a frontier between consistent and not consistent trading results. The following definition represents a fair and accurate definition of the design process. Their outcome is STS:

“The scientific trading system design is the process by which traders, individually or collectively and over time, try to construct STS”

Let us examine the statement in more detail:

The Scientific Trading System Design is a process
The STSDesign is used by traders
The Scientific Trading System Design’s goal is to construct STS.

The process is not a cookbook recipe. That is to say, there is no owner’s manual or user’s guide. Many traders don’t even know the process details! …The loser ones.

The Process: Scientific Trading System Explained

In 2018, Quantopian (Obsolete) sponsored a new series of a trading contest with total prize money of 5000 USD/month (at the present time). The goal of the contest is to build a stock trading system. This should trade a significant time amount without trader intervention. Contest rules are hard. All these systems must earn money otherwise they will be disqualified.

Only the top ten earn prizes on a daily basis. More of the contestants do not win in first attempt. The contestants carefully monitored what happened in that contest, analyzed the data, and made corrections to their trading systems. Many contestants improve their positions in second attempt. Again, the contestants carefully monitored what happened in the contest, analyzed the data, and made corrections to their systems. Frequently contestants improve their positions in the third contest.

This is an example of the scientific trading system design in action. Notice the key elements of the process:

Observation

A trading reality check occurs (e.g. a contest with an undesirable result).

Hypothesis

Data is collected and analyzed based on (1).

Predictions

Improvements and corrections are made based on errors found in (2).

Experiment

Something better occurs next time (more desirable contest results) based on (3).

One could draw the process as follows:

What makes the process so effective is the “feedback loop” shown by the heavy red dot line. “Feedback” is a technical term for “learning.” Everybody do that: They make a mistake, figure out what they did wrong, and hopefully do not repeat the mistake. Each time a trader do that, she is using the scientific method! If she never learned, she would continue to make the same mistakes over and over entering into a loser loop. That is the power of the scientific method.

Some of the contestants didn’t finish any of the contests. Did they fail all the contests because they didn’t apply the scientific method? Not at all. They were wrong. They created errors in the process, and mistakes overwhelmed or camouflaged the truth, resulting in failure.

But, are traders scientists?

Keep reading and we’ll see.

The Traders

Most descriptions of the scientific method state that it is used by scientists. Many descriptions further restrict its use to “hard” science (physics, chemistry, and so on). Besides, other descriptions restrict its use to situations which involve physical magnitudes. Such statements are incorrect and false. Everybody learns; thus everyone uses the scientific method. Even learning machines!

“Learning” is essential to success, moreover for a trader success. For example, consider the situation of a compulsive gambler. Step 1 is to recognize the problem as a problem: “I cannot control my impulse to gamble,.” Step 2 is to examine the options, “I can cure myself or I cannot cure myself” and then take appropriate action. If the answer is “I can’t cure myself,” then I must devise a corrective plan (Step 3) – decide to start a residential treatment and rehab programs. I will make improvements after I starting treatment and begin executing their program. Before graduating from the program, I will have to pass some kind of test (Step 4). When completed, I will be successful and ready to face reality (Step 1). The process is more difficult than that, but the loop is correct.

The Goal of Trading Systems Design

The goal of the scientific trading system design is to construct an accurate picture of financial market reality to determine when to buy and sell assets. We can define the Trading Truth as “the more accurate picture of financial market reality to best determine when to buy and sell assets” = Trading Truth (TT). So the goal of the STSD is to seek the Trading Truth. There is only one Trading Truth. TT is an entelechy and imply the perfect knowledge of future events.

I challenge you to apply the scientific method!

We will examine in more detail how the scientific method can be applied to discern the Trading Truth.

Some Sources of Error in Trading Systems

Below are the main sources of error that can arise when you design a scientific trading system

1) Observation to create the trading systems: In the contest

Observation phase of the scientific trading system design

Failure to recognize and admit failure

An extreme example of this failure is the “everybody wins” syndrome. If a competitor does not recognize that not finishing the contest, or not finishing the contest earning enough to beat another competitor is a failure, that trader will not be motivated to learn and improve performance in the next contest. One would think that this is very rare, but the syndrome is wild in our trading world.

Failure to collect all the information needed

You need enough data to complete a thorough analysis, or to collect erroneous information for the analysis. Insufficient or erroneous information will result in erroneous conclusions and problem identification which exclude effective corrective action.

2) During hypothesis funding the trading systems

Incorrect assumptions

We can’t know everything and we can’t get full information. Thus we must make some assumptions about how things are, work, and so on. If these assumptions are wrong, they could result in wrong conclusions and results. A common mistake is to presume an outcome before complete the analysis and then use the analysis to support or justify the presumption. This is a fatal mistake that prevents all possibility of learning.

Sloppy, incomplete or incompetent analysis

These conditions will result in erroneous analytic results, usually identifying the wrong problem as the cause of failure. Again, since we can’t know everything, shortcomings in analytical approach and rigor result in identifying the wrong problem as the cause of failure. For example, assume that the contest failed because the trading system didn’t earn money. Why didn’t the trading system earn? It placed the wrong orders. Why did it place wrong orders? The order manager malfunctioned. Why? Etc. Sometimes we have to ask “Why?” dozens of times before we reach the root cause. The process is hard and very painful. So it requires extreme discipline to continue questioning, and many don’t. The incorrect diagnosis is one of the most common source of error and failure. Can you believe it?

Error and sensitivity analysis

One of the most critical, yet ignored, elements of this activity is error analysis, also known as a sensitivity analysis. This is very difficult. There is nothing perfect in this world; thus errors abound everywhere. One must identify both the sources of such errors and their possible effects on the outcome. For example, let’s assume that the order manager of the trading system uses the bid price to place orders (both types sell and buy). And the usual spread is less than 1%.

On several month contests, we’ll have to face volatility variations. This could result in spreads greater than 25%; too much to succeed. How much error can we tolerate? How good an order manager can we actually design? But, the order manager accuracy may not matter if we can depend on a risk controller to recalculate the risk in contest. So, when an error analysis is not performed, we may be wasting a lot of energy, time, and resources trying to solve a problem that doesn’t exist!

3) In the predictions (corrective plan)

Corrective plan is part of prediction phase

Bad problem identification

The success of the corrective action plan depends on how accurately the root cause problem was identified in the hypothesis step. What good is an improvement plan that does not address the real root cause of the problem? Why fix something that is not broken, and leave broken something that is broken? It’s the best recipe for disaster.

Assumptions and rigor

The success of the corrective plan depends on the same factors as the analysis of the problem: Accuracy of assumptions, rigor, completeness, and veracity of the planning process. What good is a corrective action plan that relies on uncontrollable factors?, “We failed because EUR/USD is raising and it won’t rise next time”? The same rigor and diligence are required in developing the corrective plan as was required for analysis. Make sure plan errors achieve acceptable tolerances – or failure will be a fact!

4) During the trading systems experiment

Experiment phase in scientific trading system design

Exhaustive and honest tests

Improvements must be made honestly and thoroughly tested against the requirements of the corrective plan. For example, our corrective plan calls for a wheat-corn pair trading with frequent rebalances to control the risk. But, both markets have different liquidities and we could have to face the risk of losing one of the legs. If we don’t test the market volatility before entering the contest, the trading system will place incomplete orders and we will fail. If we do test it and find out the truth, we have an opportunity to take other corrective actions and still succeed. We know that few things work exactly as planned. So Exhaustive and honest testing is an essential element of success.

Stress and limit testing

The only certainty is that we will meet the unexpected. Therefore we need to conduct stress testing to see how the trading system will function under the usual adverse conditions. We are likely to encounter such as extreme volatility, important news, debt exposure and so on. Long-Term Capital lost $4 billion after a debt Management default by Russia. We need to validate that our trading system will work fine in the presence of high impact news like Fukushima disaster or Brexit. otherwise it will lost and fail the contest. The same logic applies to limit testing, which validates that the trading system will always work within the boundaries of its design limits.

The Worst Failure about Trading Systems

Lack of honesty and intellectual integrity: The foundation for the entire process is honesty, accuracy, and truthfulness for each step of the process. Lacking these human traits, the process is sure to fail because results, reports, and even tests will be falsified. But such falsification inevitably results in a failure during market reality check. Thus personal honesty and integrity are the key foundations of the entire process. The process is nothing more than a futile recipe for failure without intellectual integrity,

Conclusion

These concepts and design principles are all you need to build a formidable scientific trading system.

These systems are reliable, consistent and non-arbitrary.

Here you are the steps and the possible failure sources:

Observation.
- Failure to recognize and admit failure
- Failure to collect all the information needed
Hypothesis
- Incorrect assumptions
- Sloppy, incomplete or incompetent analysis
- Error and sensitivity analysis
Predictions
- Bad problem identification
- Assumptions and rigor
Experiment
- Exhaustive and honest tests
- Stress and limit testing

Scientific trading system design in action

Apply yourself to these principles and in a few months you will have a consistent system and you could fight to be among the top ten of any contest.

Make sure to visit the site of Quantopian (Obsolete) and other interesting contests as Darwinex to test your scientific trading system design abilities.

In the next posts, I will show you how to implement these concepts and principles step by step.

Meanwhile do not forget to read my post about Social trading