Monthly Archive for September, 2009

Berkeley DB C# Bindings

Berkeley DB is an excellent Transactional Data Store. Historically it has had a strong C and Java following, but with the recent 4.8 release, they have added C# bindings, finally opening up Berkeley DB to C#.

This is a non-relational database, which is well suited for things like raw tick data, where you are not querying them by values, but instead by an indexed time and symbol field.

To give you an idea of why this is important, take a look at their white paper from 2006 on performance. In a transacted environment they achieved 125,486 single record writes per second. With modern 2009 hardware, and multiple CPU / Solid State Disk systems, this could readily record every single tick coming off of the NYSE and NASDAQ (multiple million per second).

Check them out at Oracle Berkeley DB. I’m in the process of testing this out for our historical logging, and am very impressed so far, even on commodity hardware.

Multithreading made easy

I love when Robotics innovations overlap with Finance. Our trading engine is a discombobulated series of threads interfacing to their respective data sources and execution platforms, with careful locking in between. Making changes to this system has become a nightmare due to all the careful locking concerns. Enter Microsoft CCR. It provides a very clean and thoroughly tested set of primitives for a large scale thread pool with inter task messaging. It even has adapters for using tasks on Windows Forms threads, as well as WPF dispatches. It was first distributed with the Microsoft Robotics Studio, as they suffer from the same problems we do in finance, talking to a host of sensors and actuators with careful locking constraints.

To illustrate a common problem with trading systems. You have a producer consumer model, where your data is coming in from the exchange on a single thread. This data is going to a large number of contracts, and it would be nice to do this in a multithreaded model. Under the ordinary paradigm, you would create a queue, have your producer thread lock on the queue, add the new market ticks, and unlock, letting your consumers take locks on the queue, and beginning to work on this. There are special concerns though, for a given contract you need to make sure all ticks are handled synchronously, and whenever the queue hits zero for a particular contract after data has changed, you want to tell your strategy to process the new data. Here you would need a second producer consumer framework, and things continue expanding from there.

With the CCR, you would break things down into tasks. There would be a task that consumes raw market data and posts it to a port. There would be a task that takes a tick from the market data and posts it to a contract. There would be a task that takes a tick in a contract and updates the contract, checking at the end if the contract’s queue is empty, if so, it would post the entire contract to the strategy saying it has been updated. There would be a task that consumes a contract and updates the strategy. All of these are small work tasks that are setup to run in a common dispatcher pool, and are all invoked every time a work product is posted to a port. No complicated setup, just arbitrated registrations.

I’d love to spend a lot more time talking about this, but here is one short example.

</code>using (var dispatcher = new Dispatcher(0, "Master Dispatcher"))
var dispatcherQueue = new DispatcherQueue("Master Queue");
var tickPort = new Port<int>();

Arbiter.Activate(dispatcherQueue, Arbiter.Receive(true, tickPort, tick=>
if(tick > 25)
Market.PlaceOrder(tick, 1);
tickPort.Post(new Random().Next(50));

I realize there are many simplifications made in this post – I just wanted to convey how cool the CCR framework is. If it were free, I’d post my Ib wrapper for it! Easily worth the $400, and if you are a student you can get it from the Robotics Studio for free. There is talk of merging it with the parallels framework added in C# 4.0 (another great invention… waiting for VS 2010 to fully integrate it though).