Scott Guthrie has some very interesting things to tell us about the release process Beta 2 went through.
We basically repeat this process over and over again 24 hours a day until each of the 40 different variations passes 300 10-hour runs in a row without incident on each different hardware and OS configuration we support (Windows 2000 Single Proc, Windows 2000 2P, Windows 2003 1P, Windows 2003 2P — with WS03 repeated for x86, x64 and IA64 processors). We then also do longer-haul runs that run on higher-end 4P hardware that runs for a week under extreme load using a combination of different variations (we also then on the IIS side do long-haul runs that take a full 21 days and put extremely heavy load testing our worker process reliability).
Needless to say, it can take awhile to get everything passing. In the early days of the stabilization for Beta2 it was pretty easy to find issues. As we lock down changes in the overall stack, it takes longer and longer for them to surface. Sometimes we had to-do 200-250 runs for a bug to surface, and even then it might take multiple “hits” before we could figure out what exactly was causing it. The stress team did an awesome job driving this process forward and chasing down the final bugs through long days and late nights (it was not uncommon for them to be in the lab until 4-5am and then have to be back by 9am to give updates to our war team).