Sanjay Arora <sanjay.k.arora@xxxxxxxxx> writes: > Do you mean something like storing one month worth tick data in a blob > type field and giving the contents to the CEP engine for further > building of required data streams? Either that or put the tick data in an external file and store that file's pathname in the database row. As I said, you can find plenty of argumentation on both sides of that in the PG archives. When SSS were doing this, we had the raw tick data in one set of files and pre-aggregated bar data in other files (I think we stored 5-min and daily bars, but it was a long time ago). The analysis functions would automatically build the bar width they wanted from the widest stored form that divided the desired width, so as to minimize what they had to read from disk. It does sound like you are doing pretty much exactly what we were doing. One thing to think about is that the real-time case is actually much slower and easier to deal with than back-testing. When you are back-testing, you'd like to test a trading strategy over say a year's worth of data, and you'd like that to require a bit less than a year to run, no? So the path you have to optimize is the one feeding stored, pre-aggregated data into your test engine. The people hollering about real-time performance are just betraying that they've never built one of these things. I'm not familiar with this CEP software, but it sounds to me like you want that as part of the real-time data collection process and nowhere near your back-testing data path. Another little tip: if your trading strategies are like ours were, they need to ramp up for actual trading by pre-scanning some amount of historical data. So you're going to need a mechanism that starts by reading the historical data at high speed and smoothly segues into reading the real-time feed (at which the passage of time in the model suddenly slows to one-to-one with real time). Also consider what happens when your machine crashes (it happens) and you need to not only redo that startup process, but persuade the model that its actual trading position is whatever you had open. Or you changed the model a bit and restart as above. The model might now wish it was in a different position, but it has to cope with your actual position. The crash reliability of a DBMS is a strong reason why you track your live positions in one, btw ... your brokerage isn't going to forget you were short IBM, even if your model crashes. regards, tom lane -- Sent via pgsql-general mailing list (pgsql-general@xxxxxxxxxxxxxx) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-general
