On Mon, May 10, 2010 at 8:20 AM, Matt Garman <matthew.garman@xxxxxxxxx> wrote: > On Sat, Mar 06, 2010 at 03:05:46PM -0800, Mark Knecht wrote: >> On Sat, Mar 6, 2010 at 2:33 PM, Greg Freemyer <greg.freemyer@xxxxxxxxx> wrote: >> > You haven't said what kind of i/o rates you expect, nor how much >> > storage you need. >> >> Good points. I guess I was assuming I'd want 1TB storage and I'd >> buy 3/5/6 1TB drives to get it. Honestly I probably don't need >> anything close to that. My weekly backups of stock data run about >> 1GB to 1TB should hold me for quite awhile I think. > > Where are you sourcing the stock data? 1GB/week seems awful low. > You must be getting top of book only? > > We get real-time full depth stock data. NYSE and Nasdaq data are > each about 100 GB/month, compressed. > > Just something to keep in mind if you ever start working with > full-depth feeds. > > Also: for backups, you might want to consider par2. You can use it > to create a specified amount of parity data for each file or group > of files. Useful in the case of media errors. > Yeah, my description was a bit lacking. Sorry. Currently I only auto-trade index futures so I get only the indexes (tick & 1 minute) and then a few other general things (VIX, A/D, Gold, Oil, 3 month T bill, etc.) for correlation purposes. I don't mess much with individual stocks at all although I have it in my mind to start creating some private out of individual stock data so if I do that storage requirements will certainly go up. Thanks, Mark -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
