Stressful Stories

When we started getting close to the first pilot of our browser based application we put aside a couple of weeks for stress testing and performance tuning. At first we started to look around for some tools out there that would stress the system for us. Then we realised that we already had one:

As with any good XP team we write story tests to cover the stories and their acceptance criteria. We've got pretty good at them and we feel as though we know how to implement them. We implement them in our own scripting language, so we could extend the language and its application so that it could do the stress tests for us.

So we went to speak to our customers. They knew how people used the system we were replacing and how they expected (well, wanted) them to use the new one. We decided that there were three types of user, and we put profiles together for their usage. We then put together versions of the story tests that behaved in accordance with these profiles. We added some random elements to make sure there were some variations. And then we ran multiples of these pseudo users in parallel, checking how the system behaved as we upped the load.
We made sure that every major method in the application logged its start and finish with a singleton profiler class and we threw the numbers into a database once the tests finished running.

By getting total timings for parts of the system based on real world user behavior we could concentrate on the areas that would have best overall impact, not just those that we thought took the longest in isolation. We could re-run the tests and check that system performance had improved, and then pick the next area to focus on, feeling comfortable that we were ready to move on.

The result was that performance tuning was very focused. We knew which bits needed attention first, and we knew when to stop tinkering. We knew how every part of the system behaved under stress, and so and we really felt we knew the performance of the system in advance of the pilot.

We stated that the database would have no problem with the quadruple the expected load with confidence. We stated that the web server would handle it without any load balanced clusters, or other such complications. We could give figures for the expected page response times, and the customer had faith in those figures.

And when we went to pilot, we found out we were spot on.