Berkeley DB is an excellent Transactional Data Store. Historically it has had a strong C and Java following, but with the recent 4.8 release, they have added C# bindings, finally opening up Berkeley DB to C#.
This is a non-relational database, which is well suited for things like raw tick data, where you are not querying them by values, but instead by an indexed time and symbol field.
To give you an idea of why this is important, take a look at their white paper from 2006 on performance. In a transacted environment they achieved 125,486 single record writes per second. With modern 2009 hardware, and multiple CPU / Solid State Disk systems, this could readily record every single tick coming off of the NYSE and NASDAQ (multiple million per second).
Check them out at Oracle Berkeley DB. I’m in the process of testing this out for our historical logging, and am very impressed so far, even on commodity hardware.
It would be great if you’ll post your conclusions, thanks!
It would be great if you’ll post your conclusions, thanks!
This looks very interesting, but I was wondering if there
was a good interface to construct the DB with. One gets a bit spoiled with Micrsoft’s interface in MS Access and SQL.
The next question becomes: can one use the DB as a data feed. (pump it to the DB and let the apps attach to the DB rather than to IB for quotes). It would be really nice to route prices to multiple apps when one hits IB’s ticker limit. (Suppose you want to track Intel in your SP500, QQQ and DIA application). (Or one could use a abandon IB to different data vendor)
Berkeley Db has definitely proven to be useful, but as Robert pointed out – there is no easy interface. Your entire wOO wrapper has to be hand coded, and all of your indices are hand maintained. It is much more difficult than SQL / linq-to-sql, but that’s the price you pay for ridiculous speed.
If you are looking at Berkely DB on Windows you could also evaluate the ESENT database engine that is part of Windows. ESENT is the database engine that Active Directory, Microsoft Exchange and several other Windows applications use. Its performance characteristics are similar to Berkely DB. In terms of features the two products have different approaches: Berkeley DB supports replication and multi-process access but is a simple key/value store while ESENT is an embedded database which supports sophisticated meta-data (thousands of tables/columns/indexes, sparse columns, sophisticated indexes), snapshot isolation and non-blocking reads.
The managed wrapper to the ESENT API is here:
http://managedesent.codeplex.com/
Interesting, I will have to take a closer look. I am a fan of Berkeley Db’s blob nature…