On 6/15/07, Brian Dunning <brian@xxxxxxxxxxxxxxxx> wrote:
Thanks for the replies but I'm not sure I know what to do with them. Is the problem with the number of files, or is the problem with the activity? Can you dumb down your answers at all for me? :) On Jun 15, 2007, at 11:51 AM, Daniel Brown wrote: > 6/15/07, Jay Blanchard <jblanchard@xxxxxxxxxx> wrote: >> [snip] >> Server is running Linux, and PHP is constantly creating and modifying >> images in a directory. Apache is constantly serving these same >> images. There are about 250,000 of them in the same directory. Seems >> to be running OK, no problematic CPU load, but I'm wondering if >> anyone knows whether I'm living dangerously having that many docs in >> one directory with that much activity. It is an extremely busy >> server. >> >> Sorry is this seems like more of a linux sysad question than a PHP >> question. Thanks... :) >> [/snip] >> >> Two words. Beware the inode. >> >> -- >> PHP General Mailing List (http://www.php.net/) >> To unsubscribe, visit: http://www.php.net/unsub.php >> >> > > If that directory is all on one single partition, God help you if > there's any corruption. Check into RAID 0+1/RAID 5 disk striping if > you've got that many images. And remember, the fact that they're all > in one directory doesn't matter at all to the system, as directories, > folders, et cetera, are just representations for human readability and > organization. In fact, those files reside on several sectors > throughout the drive, and each file itself is probably fragmented many > times. > > Just for fun, though, if you want to crash your server, you > could always do: > `while [ 1 = 1 ]; do ls -l;done` > > -- > Daniel P. Brown > [office] (570-) 587-7080 Ext. 272 > [mobile] (570-) 766-8107 > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
Brian, I stand behind my suggestion, though I also suggest you take whatever everyone else says into consideration just as much as me. I have well over a decade of experience, but I'll be the first to admit that sometimes a teenage computer geek is more correct than I am. Yes, list, believe it or not, I'm not always correct. I'm just going to let that sink in a little bit. Okay, now that everyone is done laughing at me (well, mostly everyone), let's get on with it. I still think it's best to split the files across a RAID. Ask your datacenter to discuss the pros and cons of both RAID 0+1 and RAID 5 with you, and research it on the web. It would be better to split the files into different directories, as well, but *probably* is not necessary. Take that as you will. That said, one way of doing it that is very common is to create a minimum of 36 directories, from 0 - 9 and a - z. Each directory should contain only files where the first character is the same as the directory within which it resides. However, if you have a specific character that comes up much more frequently than others, or even all the time, this won't work, of course. The point is, in my experience, holding even a terabyte of data in smaller files within one directory is not a Bad Thing [tm], per se. I've administered MySQL databases on servers that were in the two-to-three terabyte range, and one of which had well over 3,000 separate tables. Each of those tables is three separate files (minimum) on disk, mind you: table.frm, table.MYD, table.MYI. Also, keep in mind that I didn't create the database structure, but was instead hired to ensure that it ran smoothly. In the four months and n-million queries that were run on it while it was in use, the server responded as if it were a standard 50MB database in under a dozen files (if memory serves, it was a Dual Xeon 2.6GHz with 4GB RAM) and only used for MySQL). After the project was completed, however, we did upgrade the systems to a distributed network for further expansion, but I think you get the point there. Bottom line, however, is that just because you're not experiencing problems now doesn't mean you won't soon in the future. Seriously consider, with that many separate files and that much data, how much it's worth to you in the long run, and if a RAID setup and directory-splitting is a Good Thing [tm] for your situation. -- Daniel P. Brown [office] (570-) 587-7080 Ext. 272 [mobile] (570-) 766-8107 -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
