> -----Original Message----- > From: Stuart Dallas [mailto:stuart@xxxxxxxx] > > Seriously? Errors like this should not be getting anywhere near your > production servers. This is especially true if you're really getting 30k > hits/s. Don't get me started. I joined here almost a year ago. They didn't even have a RCS or Wiki at the time. Nothing was OOP. There were no PHPDoc or even comments in the code. They used to make each site by copying an existing one and modifying it (ie. no shared libraries or resources). I could go on and on. Suffice it to say we've made HUGE leaps and bounds (thanks to me!), but there is only 3 of us developers here and no official test person let alone a test team. It is what it is. I'm doing the best I can with the limited resources available to me. And let me tell you a little secret, when you get to that scale, you see all kinds of errors you don't see on your VM or even with a test team. DB connections go away. Funny things happen to memcache. Concurrency issues arise. Web bots and search engines rape, pillage and ravage your site in ways that make you feel dirty. So yeah, you do hit weird situations and cases you can't possibly test for, but show up in error logs. > For a commercial, zero-hassle solution I can't recommend > http://newrelic.com/ highly enough. Simple installation followed by highly > detailed reports with zero issues (so far). They do a free trial of all the > pro features so you can see if it gets you what you need. And no, I don't > work for them, I just think they've built a freakin' awesome product that's > invaluable when diagnosing issues that only occur in production. I've never > used it on a site with that level of traffic, and I'm sure it won't be a > problem, but you may want to only deploy it to a fraction of your > infrastructure. A quick look at that product seems interesting, but not what I really need. We have a ton of monitoring solutions in place to get metrics and performance data. I just need a good 'hook' to get details when errors occur. > If you want a homemade solution, the uncaught exceptions are easily dealt > with... CATCH THEM, do something useful with them, and then die gracefully. > Rocket science this ain't! Thanks captain obvious. :) I can do that (and did do that), but again, at these scales, all the text-book code you think you know starts to go out the window. Frameworks break down. RDBMS topple over. You have to write things creatively, leanly (and sometimes error on the side of 'assume something is there' rather than 'assume the worst' or your code spends too much time checking the edge cases). Hit it and quit it! Get in. Get out. I can't put try/catch around everything everywhere, it's just not efficient or practical. Even the SQL queries we write are 'wide' and we pull in as much logical stuff as we can in one DB call, get it into a memcache slab and then pull it out of there over and over, rather than surgical queries to get small chunks of data which would murder mySQL. Part of the reason I took this job is exactly because of these challenges and I've learned an incredible amount here (I've also had to wash the guilt off of me some nights, as some code I've written goes against everything I was taught and thought I knew for the past decade -- but it works and works well -- it just FEELS wrong). We do a lot of things that would make my college professors cringe. THAT is the difference between the REAL world and the LAB. ;-) > See the set_exception_handler function for an > easy way to set up a global function to catch uncaught exceptions if you > don't have a limited number of entry points. > > You can similarly catch the warnings using the set_error_handler function, > tho be aware that this won't be triggered for fatal errors. And this is the meat of the solution. Thanks! I'll look into these handlers and see if I can inject it into someplace useful. I have high hopes for this now. > But seriously... a minimal level of structured testing would prevent issues > like this being deployed to your production servers. Sure, instrument to > help resolve these issues now, but if I were you I'd be putting a lot of > effort into improving your development process. Contact me off-list if you'd > like to talk about this in more detail. See above. I have begged for even a single dedicated tester. I have offered to sacrifice the open req I had for a junior developer to get a tester. That resulted in them taking away the req because "clearly I didn't need the developer then" and "we can just test it ourselves". You're preaching to the choir my friend. I've been doing this for 15+ years at various companies. ;-) d. -- PHP General Mailing List (http://www.php.net/) To unsubscribe, visit: http://www.php.net/unsub.php
