Monday, November 06, 2006

Testing doesn't have to be formal to be automated

Something I hear quite a bit from people who don't 'do' automated testing is the set of excuses that goes something like this:

"Look, we know it's a really good idea and everything but we just can't afford the start-up time to bring in all these automated test tools, set up a continuous integration server and write all the regression tests we'd need in order to get it up and running. And even if WE thought we could, there's no way we'd get it past the management team."

Now let's for a second assume that the standard arguments haven't worked in response: How can you afford not to; It'll save you time in the long run; Yada yada yada. When people are in that mind set there's not much you can do...

For reasons that I'm going to explain to you right now, we have a project on the go that isn't covered by automated tests. It's an inherited system that can't have tests retro-fitted in the kind of time we have. In reality, most of the work we're doing is actually removing functionality, with a few cosmetic changes and a little bit of extra stuff in the middle. No more than a couple of weeks work.

It turns out our standard automated test tools can't just be readily fit onto the system we have.

But that's OK. It certainly doesn't mean we're not going test, and I'm damn sure that we're going automate a big chunk of it.

First of all we've separated all the functionality that can be delivered in a new module and that part WILL be fully unit and story tested. That leaves a pretty small amount of work in the legacy system. Small enough that we could probably accept the risk associated with not doing any automated testing.

But that would be defeatist.

So instead we've picked up Selenium.

Not the full blown selenium server and continuous integration hooks and whatever. Just the Firefox based IDE.

It's simple, and requires absolutely minimal set-up... it's an xpi that just drops straight into Firefox like any other extension. Having installed the IDE you get action record and playback, and nice context sensitive right click options on any page that allows you to 'assert "this ole text" appears on page' or 'check the value of this item is x'. Basically it's almost trivial to get a regression test up and running. Then you can use the IDE to run the test.

So having got that up an running, before we set about deleting huge swathes of functionality we create a regression test that ensures that the functionality we want to keep stays there. We've found that a decent sized test that covers a fair few screens and actions can take us as little as an hour to get together. To put it into perspective: we put together a test script today that took about 20 minutes to run manually. That test took us about 45 minutes to sort out in the Selenium IDE and then a minute or two to run it each time after that. So by the time we'd run it 3 times we'd saved ourselves 15 minutes!

Running it might involve executing a SQL script manually, running a Selenium script, then checking some e-mails arrive, then running another Selenium script. In short, it might involve a few tasks performed one after another. And yes we could automate the whole lot, but like I say... we just don't have the time right now. But using the tool to add what tests we can right now, to help us with our short (a few hours each) tasks means that we're building up a functional test suite without ever really thinking about it. We'll keep those scripts, and maybe in a couple of weeks we'll realise that we DO have the time to set up everything else we need for proper functional testing.

Yep, it could be a hell of a lot better (and on our other projects it is), but some informal testing using an automated runner is an order of magnitude better than no automated testing at all.

Technorati Tags: , , , , , , ,


Steven Feuerstein said...
This comment has been removed by a blog administrator.
Steven Feuerstein said...

Hi Robert,

I am always very excited about anyone working in the Oracle space and paying attention to things like Extreme Programming and automated testing.

I thought you might like to know about my latest effort at automating the unit testing of PL/SQL code (not sure if that is something you write...): Quest Code Tester for Oracle. It is the outgrowth of my first effort, utPLSQL (the JUnit analogue for PL/SQL). The big thing about Code Tester is that it generates test code from your specs, so you don't have to write the code yourself, by far the worst part of unit testing!

You can check it out at

Warm regards,
Steven Feuerstein

Rob Baillie said...

Cheers for the tip Steven.

We've been using utPLSQL for sometime now... it's a great tool. Can't thank you enough for it!

So Code Tester is the next big thing is it? Looking forward to taking a look. I hope it doesn't do too much generation though... actually stopping and thinking about the tests is half the point of doing unit testing isn't it ;-)

APC said...

>> And even if WE thought we
>> could, there's no way we'd get
>> it past the management team."

I've always taken strength from a line of Jim Highsmith's that Ron Jeffries quotes...

Questioner: How do I sell my executive team on doing this stuff?
Jim Highsmith: Don't. Just do it. They don't know what you're doing anyway.

Cheers, APC

Rob Baillie said...


Very very true APC,

I take great joy in the fact that the management team actaully does know what we do here... I have the ear of my boss's boss's boss, and yup, he does know how to program. But since he actaully understands what we do on a day to day basis we find that he trusts us (most of the time) to do it in the best way we can.


Not everyone has that.

As you say, some people have a boss that doesn't know what they do all day long... excellent. Pick up that automated testing and run with it. You'll not regret it.

But having a boss that doesn't know what you do isn't the only other situation...

If your boss DOES know what you do all day and doesn't understand why you would want to pick up automated testing even after having seen what it can do for a team... why would you let them continue to be your boss?