As you can see in the above central box, our trading system abstracts several core functionalities.

• Settings Management – the entire trading system is configured via a straightforward xml configuration file. The actual storage and management of this is abstracted by the particular profile. For live running, these settings are version controlled and managed in a central replicated sql database. For simulation, these are stored as a simple file provided to a console based simulator. For optimization purposes, these files serve as the basis for chromosomes in the genetic optimizer (with an optimization file providing the constraints for the search space). At the end of the day, develop a simple generic settings management system that can be abstracted for different targets.
• Contract Manager / Base Contract – The core component of any system is the instrument that you are trading / measuring. The contract manager provides position management and risk management abstractions, as well as contract locating functionalities. Ultimately any object that requires a contract, goes through the contract manager, and is given an abstraction of a base contract. The base contract can be a futures contract, equity, bond etc. This provides for a universal interface to subscribe to market data, and issue / monitor orders.
• Strategy Engine / Base Strategies – The strategy engine is the very heart of any trading system. This basic class subscribes to message pumps and processes the messages to handle orders. It is the most versatile object in the trading system, allowing for nearly any type of strategy.
• Charting – Few systems put enough emphasis on thorough charting, but I find it critical for visualizing the results of a simulation, as well as determining what is happening during live trading. All contracts and strategies implement a simple IChartable interface that allows them to output highly configurable charts, right down to the Graphics handles. This allows the charts to be presented in a live windows forms view, or painted to a Bitmap class for saving to disk.
• Logging – At the end of the day, traceability is critical. Every trade made needs to be serialized to disk / database in order to reconcile with your clearing house. Furthermore, every strategy needs to output useful tracing information to aid in debugging. Beyond the obvious tracing, strategies also need to implement a reporting interface to provide live state information to the user interface in order to determine how it is behaving, and if necessary to modify its parameter set, or to debug the strategy. This again is abstracted, just like settings and charting to go to different destinations based on the target of the trading engine. For simulation it outputs to the simulation results, whereas in live trading we work against easily queried database engines.

Next up I want to cut into application design and multithreading. There is a lot to cover, and I am swamped, so expect the articles to continue to appear as I have time. And if you have any questions feel free to email email hidden; JavaScript is required.

• Live Trading Engine – Any given system will start with the live trading engine. This is the piece of software which runs in real time and actually places orders and reacts to market data.
• Simulation Engine – When developing strategies, you often need to back test them. In an ideal world back testing would demonstrate profitability, but in reality it is just used to verify that your strategy does what you think it does. The key to a good simulation engine is that you run the exact same code in simulation as you do in production. I can’t understate that last sentence, so I’ll state it again – the key to a good simulation engine is that you run the exact same code in simulation as you do in production.
• Historical Service – this runs hand in hand with the simulation engine. You need a tick database for simulation. This is the backbone of all research applications, from back testing strategies to developing market models, you need a thorough, indexed, tick database. You can also build bar data from ticks, but you better have ticks available for simulation.
• Optimization Engine – All of your automated strategies require parameterization. Generally speaking these are best optimized by hand through selection of sensible variables. Sometimes however, you need to parameterize a simple strategy for a large number of symbols, in which case you want an automated system for optimization. Our system uses a cloud computing service to distribute instances of our simulation engine which run chromosomes from a centralized genetic optimization engine.
• Analytics – You need to ruthlessly track your trading performance. At the core of any solid trading engine is a solid analytics engine which tracks your various strategies.
• Reconciler – This was the biggest surprise coming from retail brokers to institutional brokers, but everyone makes mistakes. Sometimes the exchange will fail to tell your clearing house about trades you made, other times your clearing house will accidentally include another clients trades in your account. At the end of every day you need to reconcile every fill you think you made with the statements you receive from your clearing house and immediately reconcile any errors with your clearing house and the exchange.

Next up, I will cover the major components of the Trading Engine.

Because of this website, nearly every automated trader I have met has started this way:

1. Manual trading with interactive brokers
2. Seeing a strange phenomena in the market
3. Building a simple widget to hook up to Ib
4. Exploiting the mispricing until their trading volume grows and they realize Ib does not negotiate fees and on a bum day auto liquidates your account without any notice.
5. Decides they need to go the institutional route, and figure out what their other options are.

This is exactly how I got started, and it is only after developing a full institutional system that I appreciated how good Interactive Brokers is, and why they can charge such a premium.

If you are still in steps 1 – 3, then you will want to stick with IB (in fact after step 5, for many people, sticking with IB will still be the right call). For those who have gotten to step 5, my next series of posts will cover your basic decision space.

Your first step is to get your requirements figured out. What contracts do you trade? Equities, Options, Futures, Bonds? Interactive Brokers is incredible in their support for cross asset class trading in a single account. Starting with them, it seems trivial to expect all clearing houses to let you short shares on the ASX and use your acquired buying power to buy futures on Eurex. This is simply not the case. Interactive Brokers provides both clearing services, and execution services. In institutional land these services are broken up, and the most important piece is partnering with the right clearing house.

Your clearing house determines which exchanges you can trade. They establish relationships with each exchange, and guarantee your trades. If you place a loosing bet in a leveraged account, you stand to loose more money than you have in the account, and if the clearing house mistakenly allows you to do this, they will have to cover your losses. I mentioned you have to choose your products, this is because if you want to trade futures on the LME and NYMEX, you will have to find a clearing house with relationships to both exchanges. Likewise if you would like to trade equities, you will need a clearing house that specializes in equities. Generally, clearing houses do not support all asset classes, and execution platforms certainly don’t. Example clearing houses are Advantage Futures, MF Global, NewEdge, Goldman Sachs etc.

So if clearing houses are the first piece of the puzzle, then execution platforms are the second. The clearing house will clear your trades and manage your account on behalf of an exchange, but to actually place trades, you will need an execution platform. For interactive brokers, this is their Trader Workstation front end, and their timberhill backend. In the futures market, there are a few major players, Trading Technologies, RTS (Real Time Systems Group), CQG, and Pats. These are the interfaces you will use to actually trade, and are generally all supported by the major clearing houses and exchanges, so you can use the same execution platform regardless of who you clear with.

The above diagram generally illustrates the basic relationship between the parties. What you will notice is that the only time money exchanges hands is at night. Intraday there is no relationship between your cash and your maximum position. That is strictly a risk management function of your clearing house. This is where your clearing house relationship becomes incredibly important. It is not uncommon for clearing houses to let you have much lower intraday margins than required by the exchange (how do you think velocity futures gives out $500 margin on intraday e-minis), but at night time the clearing house has to settle with the exchange. Here again, the clearing house will frequently lend you money to increase your margin , but in general over night is dangerous enough, no reason to over leverage it.

There is a lot more to this, but I want to cut to the technology. If you are going through this process, and want a lot more detail on the various fee structures and technology tradeoffs, please email hidden; JavaScript is required. I have worked with or thoroughly researched nearly every major futures clearing house and execution platform, as well as most of the equities options.

My next posts are going to continue to be overview posts, but I am particularly excited to cut to the technical / code, so stay tuned.

I’d like to break these up into three major pieces:

1. Vendors and Organization – this includes clearing houses, execution platforms etc, fees etc.
2. Technology – These are your basic platform decisions, Windows vs *nix, x86 vs x64 etc.
3. Software – the meat of our system – cover the various components required to build a fully functional ATS, as well as a research system.

My next post will be an overview of what is required to do automated trading.

Stay Tuned…

If you have any problems, email hidden; JavaScript is required.

Download the release here, or go to the utilities page.

This is a non-relational database, which is well suited for things like raw tick data, where you are not querying them by values, but instead by an indexed time and symbol field.

To give you an idea of why this is important, take a look at their white paper from 2006 on performance. In a transacted environment they achieved 125,486 single record writes per second. With modern 2009 hardware, and multiple CPU / Solid State Disk systems, this could readily record every single tick coming off of the NYSE and NASDAQ (multiple million per second).

Check them out at Oracle Berkeley DB. I’m in the process of testing this out for our historical logging, and am very impressed so far, even on commodity hardware.

Please download release 9.3.0.3 here.

I chose to make the RightType enumeration translate to "PUT" and "CALL". A bug report from the yahoo forums illustrated that this is no longer true, and that "P" and "C" are the only accepted values.

Please download the bug fix versions

9.2.0.2 and 9.1.0.2

Both also available under utilities.

Changes

The beta API adds support for five second bars (similar API to the request historical bars), adds two new tick types, Shortable and Last Timestamp, adds a new current timestamp method, and updates the order status event to add a whyHeld field. The release notes are available from Interactive Brokers here.

I have updated the online documentation to the 9.20 release at http://ibhelp.dinosaurtech.com.

I have also included a Visual Basic sample with this release. The C# client module is fully CLR compliant, and can be accessed from any language.

Download

So without further ado, download the latest release here.

Please submit any issues on the forums, or email me at email hidden; JavaScript is required.

Why did you do this?

I’m sure many of you are wondering why I would spend so much time on this, when Interactive Brokers now makes a .net version available through their J# compiler. The reasons are many, but mainly the J# component breaks a lot of C# design rules, and is poorly documented. My port places the documentation inline, so Visual Studio will bring up the messages, and so you are not working with cryptic const values.

Support / Stability

I plan to simultaneously support two C# clients, corresponding to the latest release of the TWS API as well as the TWS Beta API. My assembly versions will match the TWS API with an additional revision to correspond to my changes. For example, this initial release’s version is 9.1.0.1 corresponding to TWS API 9.10 and my release version 1. Please note that this is release version 1. I am considering this an alpha release as I have mechanically ported and cleaned up a lot of the code, but have not thoroughly tested every function yet.

License

The source is free for you to use in any application you like. I ask, but do not require, you to give me credit and a link back if you like it. I also ask, but do not require that if you improve the library, or find any problems that you send me your changes / problem report, so I can continue to make this better. I want to be clear that I do not hold any liability or responsibility for any use / content of this code.

Documentation

I have published MSDN style documentation from generated from the library at http://ibdoc.dinosaurtech.com. This documentation was populated from Interactive Broker’s User Guide, and notes from the IB Yahoo Forum and was generated using Microsoft Sandcastle with DocProject. I will continue to revise it on feedback from users, and as I find room for improvement.

Forum

I have created a forum, where I would like to collect problem reports / feature requests at http://www.dinosaurtech.com/forums/ and ask that any users register and provide feedback there. I will try to be very responsive, and incorporate your feedback.

Credits

This library is a port of the Interactive Brokers Java library, so of course I would like to credit the interactive broker’s developers for their API. I also used code from an article by Javier Compos "Description Enum TypeConverter" in order to transform the ugly consts to pretty enumerations with overridden ToString() functions.

Downloads

• Version 9.1.0.1 Library – July 8, 2007 – Alpha – Corresponds to IB API 9.10
• Documentation

Please note that I will maintain the latest on my utilities page.

What Does the Fitness Function Do?

The fitness function is the basis of the “survival of the fittest” premise of genetic algorithms. It is responsible for evaluating the parameter set, and choosing which parameter sets mate. The most difficult part of the fitness function is designing the function to produce parameters that are reliable and effective on data outside of the training set.

It helps to consider nature’s fitness function, we are the result of millions of years of genetic optimization, yet do not retain the brawn of a gorilla, nor the size of a sauropods (dinosaur that weighed 209 tons), nor the predatorial skills of a Tyrannosaurus. A genetic function does not just optimize for the strongest creature, but for the creature that can survive and thrive in all circumstances. Dinosaurs were clearly at the top of the food chain and thriving 65 million years ago, but were easily outlived by insects for their ability to survive the harsh aftermath of the Cretaceous-Tertiary extinction event. (Can you tell I have been researching a lot about dinosaurs since starting this blog?).

My point is that you need a fitness function which results in a set of parameters that performs well during a bull run, bear run, and also survives a market crash. A parameter set that makes a fortune on rallies, but bleeds on sideways patterns and reversals is no better than the dinosaurs, ultimately they will perish, taking a lot of your equity with them.

What Makes a Good Fitness Function?

A fitness function can be as simple as the profit generated by running your rules over training data, but this is likely to exploit onetime events in the data, and not to place an emphasis on reliability.

A good fitness function does the following

• Understands Risk – does not evaluate only profit, but how much capital the rules placed at risk to earn that profit
• Punishes Losses Heavily – by punishing the parameter set more heavily for losses than profits, you are training it to focus on consistent profits over volatile returns.
• Punishes High Risk – any rules can earn a lot on a good day by loading up on beta, you want to train your algorithm to seek true alpha.
• Does not punish zero gains – it is important to let your algorithm learn when to enter the market, and when to stay clear. Providing some incentive to simply not take a loss can be just as important as proving incentives to take a large gain.
• Run on a reasonable time frame – A fitness function should evaluate each day (or possibly shorter) of sample data on its own, accumulating the results for a particular parameter set.

Following these guidelines the fitness function must rank each parameter set, and select mates.

Mate Selection

Once the parameter sets have been ranked they must undergo selection. The obvious solution would be to only select the top ranked parameters to mate, but this may ignore other minima that lesser parameter sets are exploring.

The chart above illustrates the importance of occasionally exploring lesser ranked parameter sets. The green lines represent the highest ranked parameter sets, but as we can see on the parameter space the red line is at the base of the global minima, while the green lines are just exploring local minima. The best way to allow for this is to select mates with an absolute valued normal distribution. The choice of probability distribution and standard deviation has a large effect on how fast a genetic algorithm converges, an analysis of which will be in a future article. For now the normal distribution proves to be more than adequate.

As you can see the fitness function has a huge impact on the output of your maximization, it defines what the ideal function should do.

Tune in for more Genetic Optimization in Part 4 where I will talk about Training.

This is part two in a series of articles on Genetic Optimization / Maximization. Please visit part1 to get up to speed.

What is it?

Genetic based optimization uses the principals of survival of the fittest in order to determine the ideal parameters. This happens by first creating a random population of parameter sets (or starting with a seed population), and evaluating each parameter set’s performance. The algorithm then “mates” the highest performing parameter sets together, and includes the genetic concepts of mutation and crossover to create “children”. Wikipedia does a great job of covering what Genetic Algorithms are, so I won’t repeat the theory, but will dive into their application to a rules based system.

The basic components of Genetic Algorithm are:

1. Encoding & Mating
2. The Fitness Function
3. Training

I will attack each of these in a different article, and begin here with encoding & mating.

Applying the Genetic Algorithm to our Rules

The basic concept here is that we have to “encode” our parameters in such a way that we can apply crossover and mutation during each mating. This will depend largely on the number of parameters you are tuning for, but there are really only two options: binary encoding or chain encoding.

Binary Encoding

Binary encoding is when you convert all of your parameters to their raw binary form and string them together to effectively create a genome for your parameter set. In our example above, we are using two double types, which encoded into binary form would consist of 128 bits of data. Thus to create the random seed population, a uniform distribution is used to randomly select each of the 128 bits of data. The data is then read back as doubles and each parameter set is tested, and their results are fed into the fitness function to select the parents of the next generation.

The nice part of binary encodings is that they are very easily mated. Once the two parents have been selected (I will cover the fitness function below) the mating is performed with cross over, effectively selecting groupings from each parent, and then with mutation, randomly changing some of the children’s bits. The algorithm for this is pseudocoded below.

Function bit[] BitEncodeMate(bit[] mom, bit[] dad,
double crossoverPercent, double mutationPercent)
{
bit[] child = new bit[mom.length];
bool fromMom = true;
for(int ix = 0; ix < mom.length; ix++)
{
if(RandDouble(0,1) < crossoverPercent)
fromMom = !fromMom;
if(fromMom)
child[ix] = mom[ix];
else
child[ix] = dad[ix];
if(RandDouble(0,1) < mutationPercent)
child[ix] = !child[ix];
}
return child;
}

As you can see, this function is a very simplistic model to mate two parents. I introduced two variables, crossoverPercent, and mutationPercent. These variables are critical to the performance of the genetic algorithm, and are the topic for another blog post. Suffice it to say I am using a simple uniform distribution model (RandDouble), and crossoverPercent will drive how many crossovers occur, and the mutation percent should be quite low (think 5% or below) in order to provide good coverage, but still to converge.

The trouble with the bitwise encoding is that you cannot bound your variables easily. This encoding will randomly cover the entire range of the double variables, including infinity and several uninterpretable(NaN) values. It could be bounds checked after a mate to check validity and simply constrain all of the children to values within acceptable ranges, but as you can see this allows you very little control over the distribution of each parameter.

To summarize, bit encoding provides a very easy way to encode binary data and allows for crossover to occur despite a small parameter space. Unfortunately, bit encoding provides very little control over the ranges of the parameters, nor the distribution that the mutations will take. Despite this, it is still the most common form of encoding, and works much better for integers than for floating point numbers. You can thus constrain the number of bits for a particular integer to constrain its range, but cannot control its distribution.

Chain Encoding

Chain encoding works best for a very high parameter space where large speed gains may be had with bounding each parameter. This is the more typical case for rules based engines. While the function I provided above only has two parameters, it is often much more common to have 10 to 20 parameters which need tuning.

Here the parameter sets are simply encoded as an array of doubles (or whatever datatypes you want), and the crossover operation is taken by parameter rather than by bit. The big advantage now is that mutation can be carefully controlled. If you are able to provide a close starting point, the algorithm may use a normal probability distribution to mutate about the current value. You may also continue to operate on a uniform distribution, and simply constrain each parameter to its range. The following is the pseudocode for a function which mates two parents that are encoded with chain encoding, and whose mutations are constrained to a range.

Function double[] ChainEncodeMate(double[] mom, double[] dad,
double crossoverPercent, double mutationPercent,
double[] maxRange, double[] minRange)
{
double[] child = new double[mom.length];
bool fromMom = true;
for(int ix = 0; ix < mom.length; ix++)
{
if(RandDouble(0,1) < crossoverPercent)
fromMom = !fromMom;
if(fromMom)
child[ix] = mom[ix];
else
child[ix] = dad[ix];
if(RandDouble(0,1) < mutationPercent)
child[ix] = RandDouble(minRange[ix], maxRange[ix]);
if(child[ix] < minRange[ix])
child[ix] = minRange[ix];
if(child[ix] > maxRange[ix])
child[ix] = maxRange[ix];
}
return child;
}

Clearly this code is very similar to that used for binary encoding, but the power to control each parameter can make a dramatic effect on run time (again, this will be analyzed in a later article). This algorithm would have some trouble with narrow parameter scopes though, as in the example above, everytime a parent mates, effectively the child would very closely resemble the parents, or simply conduct a random walk of the parameter space. It is again also important to control the mutationPercent, while we are now in control of the distribution, we do not want to modify all of the parameters very often. As much as possible, we would prefer to allow the mutation to happen in one dimension, and to let the parental variability converge to the solution.

The above chart illustrates the results of 1,674 generations of running against a rules system for one of seven parameters. As you can see, the mutations caused the parameter to be exposed over a large part of its given range (0 to 100) while generally sticking with values it has found to be profitable (hence the vertical piers).

That’s all for encoding and Mating – next up we will cover the fitness function!

Before I discuss my second Automated Trading System (ATS), I need to explain the principal in which it operates. A rule based ATS depends on carefully chosen thresholds and parameters to determine when a particular stock should be entered and exited (long or short). Experience and theory can provide an excellent starting point, but to perform really well for a particular stock, it is useful to maximize these parameters on historical data.

Think about your set of rules as though they are a function.

function rules (double shortMATime, double longMATime)
{
“Enter Long when the shortMATime minute moving average crosses above the longMATime minute moving average.”
“Exit when the longMATime minute moving average crosses above the shortMATime minute moving average.”
}

In this example, there are two rules which are executed over live or historical data, with two parameters, shortMATime and longMATime. We would like to select values for these parameters such that they would have made the most money over the last week (or any time frame), assuming this represents closely what values will make the most money tomorrow. This function is very difficult to maximize, as it is not continuous. Small adjustments to either parameter can cause huge swings in the profitability of the system.

In this particular case, you may consider running the entire variable space through the function, setting each parameter from 0 to 1000 minutes, incrementing by one second, and taking the maximum output when you are done. This turns out to be roughly (1000 * 60)² = 3,600,000,000 runs. Assuming a very fast 5 seconds per run for a week’s worth of data, this would take 570.77 years to process. Clearly a better maximization function is needed, but it cannot depend on the derivative of the function, nor can it require continuity of the first order. This is exactly where genetic algorithms shine.

My next series of articles will cover Genetic Optimization in detail. Stay tuned for more updates.

In any case, this spurred a lot of research on my part. Up until now I had been investing in the best discount broker available, scottrade, and as a software engineer, the first thing I did was to write an html parsing interface to scottrade to allow my soon to be automated framework to make buy and sell orders. It wasn’t long before I did my research on the difference between a direct access broker and a brokerage firm. The former allowing me to place direct orders with an exchange, and the latter attempting to take a piece of my cake by being a market maker. My research immediately led me to Interactive Brokers, which offers an API (what you say? No HTML parsing to interface?), as well as much lower fees (we’re talking half a penny per share baby!). Later I learned that they do not offer the best margin rates, nor do they offer the highest leverage… but lets just say for my level of sophistication, they are perfect.

After establishing my new brokerage account, I needed to test the waters. I developed an incredibly simple trading infrastructure in C# using the interface provided in http://finance.groups.yahoo.com/group/twsapi/. This first cut interface simply parsed news events at the fastest possible speed, allowing my application to track it, and rapidly trade on it. As you can see from the chart below, there was some news event at 12:00. If the news event was detected at 12:00:10, one could make a tidy profit simply by going long early, and selling at around 12:04 with a stop at 43.55, this would represent a profit of 45 cents/share at a risk of 15 cents/share, or a nice 3R.

I implemented this strategy and ran for two months before trading live. It worked flawlessly, I back tested it on old news releases for about two years, and there was not a single case where the momentum did not follow the news. Then I turned it on last July, and got one good month of running, and come August it started out alright, right before the bull came. Then it hit (possibly like triceratops butting heads), the uptrend in all of the stocks I was following was proceeded by erratic trading behavior. Momentum on the in the news stocks was no longer carrying. After a news release I may see a large down tick followed be an instant up tick, so fast that I could not liquidate and close positions in time to make money. Clearly my strategy had to change. I shutdown my strategy in September, and went to work on version 2.0 of my ATS. (Part II to follow…)