BOBABLOG: Oracle, PHP and Extreme Programming

Pleasing line

2010-05-17T08:47:00.003+01:00

Gotta admit, I'm quite pleased with this line from my new ORM object based database connection library...




$oFilter = Filter::attribute('player_id')->isEqualTo('1')->andAttribute('fixture_id')->isEqualTo('2');

Sainsburys reward carrier bag re-use!

2008-06-24T19:05:00.002+01:00

For every carrier bag you re-use they give you a whopping 1 (yep a whole ONE) Nectar point.

"QI: The Book of General Ignorance" is currently on offer at only 2,400 points, and using Amazon as a guide on its retail price (£4.76), that means each point is worth a staggerring 0.2p

Oh, and you have to actually use a carrier bag to qualify. Not using any carrier bags at all means you get nothing.

Way to go Sainsburys, that global warming's going to be stopped in its tracks now.

The Happiness Meter

2008-06-23T10:01:00.003+01:00

As part of any iteration review / planning meeting there should be a section where everybody involved talks about how they felt the last iteration went, what they thought stood in the way, what they though went particularly well and suchlike.

We find that as the project goes on, and the team gets more and more used to each other, this tends to pretty much always dissolve into everyone going "alright I suppose", "yeah fine".

Obviously, this isn't ideal and will tend to mean that you only uncover problems in the project when they've got pretty serious and nerves are pretty frayed.

This is where "The Happiness Meter" comes in.

Instead of asking the team if they think things are going OK and having most people respond non-committally, ask people to put a value against how happy they are with the last iteration's progress. Any range of values is fine, just as long as it has enough levels in it to track subtle movements. I'd go with 1-10.

You don't need strict definitions for each level, it's enough to say '1 is completely unacceptable, 5 is kinda OK, 10 is absolute perfection'.

At some point in the meeting, everyone in the team declares their level of happiness. When I say everyone, I mean everyone: developers, customers, XP coaches, infrastructure guys, project managers, technical authors, absolutely everyone who is valuable enough to have at the iteration review meeting should get a say.

In order to ensure that everyone gets to provide their own thought, each person writes down their number and everyone presents it at the same time. The numbers are then taken recorded and a graph is drawn.

From the graph we should be able to see:

The overall level of happiness at the progress of the project.

If there is any splits / factions in the interpretation of the progress.

If the level of happiness is low, this should be investigated; if there are any splits, this should be investigated; and just as importantly - if there are any highs, this should be investigated. It's good to know why things go well so you can duplicate it over the next iteration.

Factions tend to indicate that one part of the team has more power than the rest and the project is skewed into their interests rather than those of the team as a whole.

You may want to split the graph into different teams (customer / developer) if you felt that was important, but I like to think of us all as one team on the same side...

All said and done, the graph isn't the important bit - the discussion that comes after the ballot is the crucial aspect. This should be a mechanism for getting people to talk openly about the progress of the project.

UPDATE: Someone at work suggested a new name that I thought I should share: The Happy-O-Meter.

Ideas for improving innovation and creativity in an IS department

2008-06-21T10:49:00.003+01:00

At our work we've set up a few 'action teams' to try to improve particular aspects of our working environment.

The team that I'm a member of is responsible for 'Innovation and Creativity'.

We're tasked with answering the question "How do we improve innovation and creativity in IS?" - How we can foster an environment that encourages innovation rather than stifles it.

As a bit of a background, the company is a a medium sized (2,500 plus employees) based mainly in the UK, but recently spreading through the world, the vast majority of whom are not IS based. The IS department is about 100 strong and includes a development team of 25 people. It's an SME at the point where it's starting to break into the big-time and recognises that it needs to refine its working practices a little in order to keep up with the pace of expansion.

We met early last week and have put together a proposal to be taken to the senior management tier. I get a feeling it will be implemented since our team included the IS Director (you don't get any senior in our department), but you never know what'll happen.

I figured it might be interesting to record my understanding of the plan as it stands now, and then take another look in 6 months time to see what's happened to it...

We decided that in order to have an environment that fosters creativity and innovation you need:

Freedom:

Time for ideas for form, for you to explore them, and then to put them into practice.

Stimulus:

Outside influences that that can help to spark those ideas off - this may be from outside the organisation, or through cross-pollination within it.

Courage:

The conviction to try things, to allow them to fail or succeed on their own merit - both on the part of the individual and the organisation as a whole.

Natural Selection:

The need to recognise success when it happens, to take it into the normal operation of the business and make it work in practice. Also, the need to recognise failure when it happens, and stop that from going into (or continuing to exist within) the team.

Recognition:

When we have a good idea, the people involved need to be celebrated. When we have a bad idea, the people involved DO NOT need to be ridiculed.

Refinement:

The initial ideas aren't always the ones that are successful, it's the 4th, 5th or 125th refinement of that idea that forms the breakthrough. We need to understand what we've tried, and recognise how and why each idea has failed or succeeded so we can learn from that.

We put together some concrete ideas on how we're going to help put these in place - and bear in mind that this isn't just for the development team, this is for the whole of the IS department - development, project management, infrastructure, operations, service-desk, even the technology procurement...

Curriculum:

A position set up that will be responsible for defining / tracking a curriculum for each job role in the department.

Obviously this will be fed by those people that currently fulfil the roles, and will involve things ranging from ensuring the process documentation is up to scratch, through specifying reading lists (and organising the purchasing of the books for the staff) and suggesting / collecting / booking conferences, training courses and the like that might be of use.

This takes the burden of responsibility away from the staff and managers - all you need is the idea and someone else will organise it and ensure it's on the curriculum for everyone else to follow up.

IdeaSpace (TM ;-) ):

A forum for the discussion of ideas, and collection of any documentation produced on those ideas and their investigation. This will (hopefully) form a library of past investigations as well as a stimulus for future ones. Everyone in the department will be subscribed to it.

Lab days:

Every employee is entitled to 2 days a month outside of their normal job to explore some idea they might have. That time can be sandbagged to a point, although you can't take more than 4 days in one stint. Managers have to approve the time in the lab (so that it can be planned into existing projects) and can defer the time to some extent, but if requests are forthcoming they have to allow at least 5 days each rolling quarter so that the time can't be deferred indefinitely.

Whilst the exact format of the lab is yet to be decided, we're aiming to provide space away from the normal desks so that their is a clear separation from the day job and lab time. People will be encouraged to take time in the lab as a team as well as individually. Also, if we go into the lab for 3 days to find that an idea doesn't work, that idea should still be documented and the lab time regarded as a success (we learnt something)

Dragon's Den:

Gotta admit, I'm not sure about some of the connotations of this - but the basic idea is sound. Coming out of time in the Lab should be a discussion with peers about the conclusion of the investigation in a Dragon's Den format. This allows the wider community to discuss the suitability of the idea for future investigations, or even immediate applicability. One output of this meeting may be the formalisation of conclusions in the IdeaSpace.

Press Releases:

The company is already pretty good at this, but when something changes for the better we will ensure that we celebrate those changes and, even for a day, put some people up on pedestals.

None of the above should be seen as a replacement for just trying things in our day to day job - but the idea is that these things should help stress to the department that change and progress are important aspects of what we do, and that we value it enough to provide a structure in which big ideas can been allowed to gestate. Cross pollination and communication should just form part of our normal day job anyway, and we should ensure that our project teams are cohesive and communicate freely amongst and between themselves.

Also, an important factor in the success of the above has to be the format of the Dragon's Den - if it is in any way imposing or nerve-racking then the idea is doomed to failure. As soon as people feel under pressure to justify themselves then the freedom disappears.

I'm quite excited by the prospect of putting these ideas into practice, and I wonder exactly where we'll end up.

I'll keep you all posted.

An apology to IE users

2008-04-01T21:25:00.001+01:00

To all those visitors who've visited the site with IE over the last 6 months and had a borken layout.

All apologies... I had no idea that long titles in the Orablogs digest was breaking things. The offending feed has been removed.

Things I believe in

2008-03-29T12:35:00.003Z

It's easier to re-build a system from its tests than to re-build the tests from their system.

You can measure code complexity, adherence to standards and test coverage; you can't measure quality of design.

Formal and flexible are not mutually exclusive.

The tests should pass, first time, every time (unless you're changing them or the code).

Flexing your Right BICEP is a sure-fire way to quality tests.

Test code is production code and it deserves the same level of care.

Prototypes should always be thrown away.

Documentation is good, self documenting code is better, code that doesn't need documentation is best.

If you're getting bogged down in the process then the process is wrong.

Agility without structure is just hacking.

Pairing allows good practices to spread.

Pairing allows bad practices to spread.

Cycling the pairs every day is hard work.

Team leaders should be inside the team, not outside it.

Project Managers are there to facilitate the practice of developing software, not to control it.

Your customers are not idiots; they always know their business far better than you ever will.

A long list of referrals for a piece of software does not increase the chances of it being right for you, and shouldn't be considered when evaluating it.

You can't solve a problem until you know what the problem is. You can't answer a question until the question's been asked.

Software development is not complex by accident, it's complex by essence.

Always is never right, and never is always wrong.

Interesting is not the same as useful.

Clever is not the same as right.

The simplest thing that will work is not always the same as the easiest thing that will work.

It's easier to make readable code correct than it is to make clever code readable.

If you can't read your tests, then you can't read your documentation.

There's no better specification document than the customer's voice.

You can't make your brain bigger, so make your code simpler.

Sometimes multiple exit points are OK. The same is not true of multiple entry points.

Collective responsibility means that everyone involved is individually responsible for everything.

Sometimes it's complex because it needs to be; but you should never be afraid to check.

If every time you step forward you get shot down you're fighting for the wrong army.

If you're always learning you're never bored.

There are no such things as "Best Practices". Every practice can be improved upon.

Nothing is exempt from testing. Not even database upgrades.

It's not enough to collect data, you need to analyse, understand and act upon that data once you have it.

A long code freeze means a broken process.

A test hasn't passed until it has failed.

If you give someone a job, you can't guarantee they'll do it well; If you give someone two jobs you can guarantee they'll do both badly

Every meeting should start with a statement on its purpose and context, even if everyone in the meeting already knows.

A reading list for our developers

2008-03-25T17:54:00.006Z

An idea I'm thinking of trying to get implemented at our place is a required reading list for all our developers. A collection of books that will improve the way that developers think about their code, and they ways in which they solve problems. The company would buy the books as gifts to the employees, maybe one or two every three months.

Some questions though:

Is it fair for a company to expect its employees to read educational material out of hours?

Conversely:

Is it fair for an employee to expect to be moved forward in their career without a little bit of personal development outside the office?

If anyone has any books out there that they'd recommend - please let me know. Otherwise, here's my initial ideas - the first three would be in your welcome pack:

Update:Gary Myers came up with a good point, being that any book should really be readable on public transport. That probably rules out Code Complete (although I read it on the tube, I can see that it's a little tricky), but Design Patterns and Refactoring to Patterns are small enough I reckon.

Unfortunately, Code Complete is a really good book that gives a lot of great, simple, valuable advice. Does anyone out there have any other suggestions for similar books?

Update 2:Andy Beacock reminded me of Fowler's Refactoring, which really should also make the list.

Update 3:The development team have bought into the idea and the boss has been asked. In fact, I'm pretty pleased with the enthusiasm shown by the team for the idea. I can't see the boss turning it down. Interestingly though, someone suggested that Code Complete go onto the list...

In this order:

Extreme Programming Explained - Kent Beck

The Pragmatic Programmer - Andy Hunt & Dave Thomas

Pragmatic Unit Testing using JUnit - Andy Hunt & Dave Thomas

User Stories Applied - Mike Cohn

Software Configuration Management Patterns - Steve Berczuk

Refactoring: Improving the Design of Existing Code - Martin Fowler

Refactoring to Patterns - Joshua Kerievsky

Design patterns : Elements of Reusable Object-Oriented Software - Gamma, Helm, Johnson, and Vlissides

The Mythical Man Month and Other Essays on Software Engineering - Frederick P. Brooks

Ruled out because of their size:

Code Complete - Steve McConnell

Head First Design Patterns - Freemen(s), Bates and Sierra

Database Build Script "Greatest Hits"

2007-09-04T15:54:00.000+01:00

I know its been a quiet time on this blog for a while now, but I've noticed that I'm still getting visitors looking up old blog posts. It's especially true of the posts that relate to "The Patch Runner". Many of them come through a link from Wilfred van der Deijl, mainly his great post of "Version control of Database Objects".

The patch runner is my grand idea for a version controlled database build script that you can use to give your developers sandbox databases to play with as well as ensuring that your live database upgrades work first time, every time.

It's all still working perfectly here, and people still seem to be interested, so with that in mind I've decided to collate them a little bit. Basically provide an index of all the posts I've made over the years that directly relate to database build scripts, sandboxes and version control.

So, Rob's database build script 'Greatest Hits':

On the basic idea of building a database in an agile manner

On how you might start putting together a version controlled database build script

On how you can organise patches to reduce the pain of a rolling back changes

On the arguments between table-centric and patch-centric database build scripts

On what automated build scripts give you

On having the build script be able to tell when something goes wrong, and stop

On how you can use your automated builds to give you a sandbox

On why your DBAs shouldn't burst into tears when you tell them you need a database for each developer

On how you database build script can work with you version control tool

As well as: a presentation given to the UK OUG

All of the posts describe processes and patch runners that are very similar to those that I use in my work every day. I started playing with these theories over 3 years ago now and there is no way I'd go back to implement database upgrades the way I did before.

However, I'd LOVE to hear ideas on how things can be improved. I'd be amazed if my three year old thinking was still up to date!

Technorati Tags: oracle,, rdbms,, software,, development,, upgrades,, Robert+Baillie

Wow

2007-08-20T16:52:00.000+01:00

And you think software patents are bad...

China Regulates Buddhist Reincarnation

Problems with CVS removes?

2007-07-20T16:58:00.000+01:00

Accidently removed a file in CVS that you want to keep?

Sounds like a stupid question, because when you know the answer to this problem it just seems blindingly obvious,
but what if you've issued a 'remove' against a file in CVS and before you commit the remove you decided that you
made a mistake and still want to keep it?

I.E you issued (for example)


 > cvs remove -f sheep.php

But not issued


 > cvs commit -m removed sheep.php

I've heard work arounds such as:

Edit the "entries" file in the relevant CVS directory in your workspace, removing the reference to the file.
This makes the file appear unknown to CVS.
Perform an update in that directory. This gets the repository version of the file and updates the "entries"
file correctly

All you actually need to do is re-add the file:


 > cvs add sheep.php

 U sheep.php
 cvs server: sheep.php, version 1.6, resurrected

When used in this way, the add command will issue an update against the file and retrieve the repository version of the file.

A word of warning though, if you had uncommitted changes in that file before you issued a remove, CVS isn't going to recover that for you...

How about if you've removed a file, but your version of the file is out of date and so you can't commit it?

So you've issued the following:


 > cvs remove -f sheep.txt 

 cvs server: scheduling 'sheep.txt' for removal
 cvs server: use 'cvs sheep' to remove this file permanently

 > cvs commit -m removed sheep.txt 

 cvs server: Up-to-date check failed for 'sheep.txt'
 cvs server: correct above errors first!

You can't issue an update because you get the following:


 > cvs update sheep.txt

  cvs server: conflict: removed sheep.txt was modified by second party
  C rob_tmp.txt

Again, add the file.


 > cvs add sheep.php

 U sheep.php
 cvs server: sheep.php, version 1.6, resurrected

This gets you the most up to date version from the repository, that you can then check for changes (you wouldn't want to just remove it now that someone's added new content would you?)

Once you've convinced yourself that it's still a good idea to delete it, just issue the remove and commit.

Simple when you know how!

Can a change in execution plan change the results?

2007-07-12T14:15:00.000+01:00

We've been using Oracle Domain indexes for a while now in order to search documents to get back a ranked order of things that meet certain criteria.

The documents are releated to people, and we augment the basic text search with other filters and score metrics based on the 'people' side of things to get an overall 'suitability' score for the results in a search.

Without giving too much away about the business I work with I can't really tell you much more about the product than that, but it's probably enough of a background for this little gem.

We've known for a while that the domain index 'score' returned from a 'contains' clause is based not only on the document to which that score relates, but also on the rest of the set that is searched. An individual document score does not live in isolation, rather in lives in the context of the whole result set.

No problem. As I say, we've known this for a while and so have our customers. Quite a while ago they stopped asking what the numbers mean and learned to trust them.

However, today we realised something. Since the results are affected by the result set that is searched, this means that the results can be affected by the order in which the optimizer decides to execute a query.

I can't give you a full end to end example, but I can assure you that the following is most definately the case on one of our production domain indexes (names changed, obviously):

We have a two column table 'document_index', which contains 'id' and 'document_contents'. Both columns have an index. The ID being the primary key and the other being a domain index.

The following SQL gives the related execution path:


SELECT id, SCORE( 1 )
FROM   document_index
WHERE  CONTAINS( document_contents, :1, 1 ) > 0
AND    id = :2

SELECT STATEMENT
  TABLE ACCESS BY INDEX ROWID SCOTT.DOCUMENT_INDEX
    DOMAIN INDEX SCOTT.DOCUMENT_INDEX_IDX01

However, the alternative SQL gives this execution path:


SELECT id, SCORE( 1 )
FROM   document_index
WHERE  CONTAINS( document_contents, 'Some text', 1 ) > 0
AND    id = :2

SELECT STATEMENT
  TABLE ACCESS BY INDEX ROWID SCOTT.DOCUMENT_INDEX                       
    INDEX UNIQUE SCAN SCOTT.DOCUMENT_INDEX_PK

Normally, this kind of change in execution path wouldn't be a problem. But as stated earlier, the result of a score operation against a domain index is not just dependant on the individual records, but the context of the whole result set. The first execution provides you a score for the single document in the context of the all the documents in the table, the second gives you a score within the context of just that document. The scores are different.

Now obviously, this is an extreme example, but more subtle examples will almost certainly exist if you combine the domain index lookups with any other where clause criteria. This is especially true if you're using literal values instead of bind variables in which case you may find the execution path changing between calls to the 'same' piece of SQL.

My advice? Well, we're going to split our domain index look ups from all the rest of the filtering criteria, that way we can prepare the set of documents we want the search to be within and know that the scoring algorithm will be applied consistently.

Handy "Alert Debugging" tool

2007-07-04T19:29:00.000+01:00

One of the coolest things about OO Javascript is that methods can be written to as if they are variables. This means that you can re-write functions on the fly. Bad for writing maintainable code if you're not structured; Fantastic for things like MVC controllers (rather use the controller to forward calls on to the model, you use it to rewire the view so that it calls it directly, and all without the view even realising it!).

What I didn't realise was that the standard window object (and probably so many others out there) can have its methods overwritten like any other.

Probably the simplest example of that proves to be incredibly useful... changing the alert function so that the dialog becomes a confirm window. Clicking cancel means that no further alerts are shown to the user. Great for when you're writing Javascript without a debugger and have to resort to 'alert debugging'.


window.alert = function(s) {
 if( !confirm(s) ) window.alert = null;
}

In case you're wondering... I found it embedded in the comments on this post: http://www.joehewitt.com/blog/firebug_for_iph.php. Cheers Menno van Slooten

Tab Complete in Windows

2007-06-26T20:00:00.000+01:00

Another one of those things that I can never remember off the top of my head so find myself constantly looking it up whenever I get access to a new machine.

I figure it may as well be my own site that I get the info from :-)

To switch on 'Tab Complete' in Windows command line change the following registry keys to '09':

HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor\CompletionChar
HKEY_LOCAL_MACHINE\Software\Microsoft\Command Processor\PathCompletionChar

Haiku

2007-06-22T17:54:00.000+01:00

Saw a cracking Haiku on a t-shirt the other day:

Haiku are easy
But sometimes they don't make sense
Refridgerator.

Then my mind started dwelling on it:

Got the release out
But the testing's not finished
It's falling over

Or

Database is slow
Just can't see what's wrong with it
Set autotrace on

Or

A quick refactor
Turns into a bigger job
Should have unit tests

Records

2007-05-28T17:45:00.000+01:00

And to follow on from the last post... my current personal bests:

I figure if I keep them here, at least I'll always know where they are!

5km Run	23:44 (Battersea Park, 'Beat the Baton' 28/05/07)
10km Run	53:23 (Hyde Park, 'Run London' 08/10/06)
Half Marathon	2:17:49 (Redcar, 'Tees Valley Half Marathon' 12/03/06)
Rubik's cube	57 seconds

Targets

2007-05-28T17:12:00.001+01:00

You've gotta have targets.

The more I try to motivate myself to do things, the more I realise that if I don't have a target it's incredibly difficult.

When I realised this it came as a big surprise to me. I'm really not the sort of person to have a 5 year plan or career goals, but it seems that if I don't set myself an only just achievable goal I find it very difficult to motivate myself to do much.

I keep myself fit so that I get the most out of playing football. But just having that in mind isn't enough to get me out and running. If I didn't set myself a target time for a 5km or 10km run and then book a place at a running event, then I'd just sit on my fat arse every night watching TV. OK, so I may be exaggerating my self deprecation, but you get the idea.

I find that this affects me in many different aspects of my life.

To motivate myself to run I set a target (public) 5km or 10km time (this year it's 22:30 and 50:00 respectively).

To motivate myself to learn to do the Rubik's cube, I set myself a target completion time (1 minute - yup, managed it).

To motivate myself to save money I set a target amount to reach by a certain date (nope, not telling you how much).

A friend of mine decided that he'd set himself the target of taking a photo a day for a year and posting it on his site. I may have to steal that idea next year... but until then you can find his here: www.ysr23.com/blog. It really is damn good.

I do the job I do because I just flat out enjoy it. As soon as it becomes too much of a chore I'll move on. And I reckon I'm doing alright career wise in whatever way you choose to measure it. For me the only measure that truly counts is enjoyment, and in the main it's a damn fine job. Well, it is most of the time anyway ;-)

Someone at work once said to me: You know, every now and again Tom Cruise probably gets up in the morning, probably on set, in his trailer and thinks to himself "Damn, gotta do some of that acting shit again today". OK, so he gets paid more in a minute that I do in a year, but you get the point.

And the big thing that keeps my enjoying my job is that I'm still learning new things. I suppose I have a clear target in my career to always keep on learning and to surround myself in people who can teach me. It's probably one of the biggest reasons why I'm so pleased to be working with Extreme Programming. It makes it easy to fulfill that goal. And it works on a clear system of easy to understand targets.

A release to the business has a target set of functionality.
A single story has a clearly defined purpose.
A unit test gives you a goal that must be met, and a clear way of determining the success or failure.

Layers of targets.

And if you're doing XP properly you get to celebrate when you meet those targets.

A brief whoop when the unit test passes.
A handful of jelly beans when the story's complete.
A damn big meal and a piss up when a release hits the business.

OK, so real life targets don't have quite the same level of celebration, but it's the same deal.

Set yourself a clear target and you get clarity of purpose in aiming for it, and the celebration when you pass it.

Roll-out

2007-04-11T08:12:00.000+01:00

I'm pretty sure that most people that read this blog will also read The Daily WTF.

But just in case you don't, there's a nice entry on 'soft-coding'.

Overall the article makes sound sense, but there's a line right at the end that resonates with me, especially since I read it the day after someone told me that they needed a developer for a whole day (9 hours) to roll out their system...

With the myriad of tools available today, there is no reason that your deployment process need be any more complicated than a simple, automated script that retrieves the code from source control, compiles it, copies/installs the executables, and then runs the relevant database scripts.

It makes me feel like I'm not alone.

Question

2007-03-23T07:40:00.000Z

I've had this conversation a couple of time with people, and I've realised that I can't get to a satisfactory answer without some research. And I'm lazy. So I'm going to pose a question... and if I don't get a satisfactory answer here I might well send it to The New Scientist in the hope that they'll answer it.

Assuming that the cost of setting up and maintaining the infrastructure is already taken care of, which is more energy efficient: an electric kettle or a stove top kettle?

I am still here...

2007-02-17T12:03:00.000Z

Sorry people, I promise I'm still here and I WILL get round to finishing my text on estimating and answering the request for more info on the database patch runner. I will, I will, I will!

The problem is, I've started reading again, and I've started playing on-line poker. Damn it :-)

But I'm enjoying it, especially a Cohn book on Agile Estimation and Planning. It is an absolute MUST read. It takes off where the estimation chapter from User Stories Applied left off, and it really doesn't dissappoint.

Unfortunately it seems to say an awful lot that I agree with, and was going to form the bulk of my next couple of posts. So if you like what I have to say on the topic, then Mike Cohn is definately worth a read... he goes into a lot more detail than I ever will here!

Obviously I'm reading an awful lot on Texas Hold 'em as well... but I'm not going to tell you what 'cause that might take away my advantage ;-)

Producing Estimates

2007-01-07T13:52:00.000Z

OK, so it's about time I got back into writing about software development, rather than software (or running, or travelling, or feeds) and the hot topic for me at the moment is the estimation process.

This topic's probably a bit big to tackle in a single post, so it's post series time. Once all the posts are up I'll slap another up with all the text combined.

So – Producing good medium term estimates...

I'm not going to talk about the process for deriving a short term estimate for a small piece of work, that's already covered beautifully by Mike Cohn in User Stories Applied, and I blogged on that topic some time ago. Rather I'm going to talk about producing an overall estimate for a release iteration or module.

I've read an awful lot on this topic over the last couple of years, so I'm sorry if all I'm doing is plagiarising things said by Kent Beck, Mike Cohn or Martin Fowler (OK, the book's Kent as well, but you get the point), or any of those many people out there that blog, and that I read. Truly, I'm sorry.

I'm not intending to infringe copyright or take credit for other people's work, it's just that my thinking has been heavily guided by these people. This writing is how I feel about estimating, having soaked up those texts and put many of their ideas into practice.

Many people think that XP is against producing longer term estimates, but it isn't. It's just that it's not about tying people down to a particular date 2 and a half years in the future and beating them with a 400 page functional design document at the end of it; it's about being able to say with some degree of certainty when some useful software will arrive, the general scope of that software, and ultimately to allow those people in power to be able to predict some kind of cost for the project.

Any business that allows the development team to go off and do a job without providing some kind of prediction back to the upper management team being irresponsible at best, grossly negligent at worst.

Providing good medium term estimates is a skill, and a highly desirable one. By giving good estimates and delivering to them you are effectively managing the expectations of upper management and allowing them to plan the larger scale future of the software in their business. When they feel they can trust the numbers coming out of the development teams it means they are less likely to throw seemingly arbitrary deadlines at their IT departments and then get angry when they're not met. Taking responsibility for, and producing good estimates will ultimately allow you to take control of the time-scales within your department. If you continually provide bad estimates or suggest a level of accuracy that simply isn't there by saying "it'll take 1,203 ½ man days to complete this project", then you've only got yourself to blame when that prediction is used to beat you down when your project isn't complete in exactly 1,203 ½ man days.

Whilst being written within the scope of XP, there's no reason why these practices won't work when you're not using an agile process. All you need to do is to split your work into small enough chunks so that each one can be estimated in the region of a few hours up to 5 days work before you start.

In summary, the process is this:

Produce a rough estimate for the whole of the release iteration.
Iteratively produce more detailed estimates for the work most likely to be done over the next couple of weeks.
Feed the detailed estimates into the overall cost.
Do some development.
Feed the actual time taken back into the estimates and revise the overall cost.
Don't bury your head in the sand if things aren't going to plan.

1 – Produce a rough estimate for the whole of the release iteration.

Before you can start work developing a new release you should have some idea of what functionality is going into that release. You should have a fairly large collection of loosely defined pieces of functionality. You can't absolutely guarantee that these stories will completely form the release that will go out in 3 months time, but what you can say is: "Today we believe that it would be most profitable for us to work on these areas of functionality in order to deliver a meaningful release to the business within a reasonable time-frame".

Things may change over the course of the next few months, and the later steps in this process will help you to respond to those changes and provide the relevant feedback to the business.

Kent Beck states (I paraphrase): You don't drive a car by pointing it at the end of the road and driving in a straight line with your eyes closed. You stay alert and make corrections moment by moment.
I state: When you get in a car and start to drive you should at least have some idea where you're going.

The two statements don't oppose each other.

The basic idea is this:

In broad strokes, group the stories in terms of cost in relation to each other. If X is medium and Y is huge then Z is tiny.
Assign a numbered cost to each group of stories.
Inform that decision with past experience.
Produce a total.

So, get together the stories that you're putting into this release iteration, grab a room with a big desk, some of your most knowledgeable customers and respected developers and off you go.

Ideally you want to have around 5 to 8 people in the room, more than this and you'll find you end up arguing over details, less and you may find you don't have enough buy in from the customer and development teams.

As a golden rule: developers that are not working on the project should NOT be allowed to estimate. There's nothing worse for developer buy-in than having an estimate or deadline over which they feel that have no control.

Try to keep the number of stories down, I find around 50 stories is good. If you have more you may be able to group similar ones together into a bigger story, or you may find you have to cull a few. That's OK, you can go through this process several times. You can try to go through the process with more, but I find that you can't keep more than 50 stories in your head at one time. You're less likely to get the relative costs right. And it'll get boring.

Quickly estimate each story:

The customer explains the functionality required without worrying too much about the about the detail;
The developers place each story into one of 4 piles: small, medium, large and wooooaaaah man that's big.

People shouldn't be afraid to discuss their thinking, though you should be worried if it's taking 10 minutes to produce an estimate for each story.

You may find that developers will want to split stories down into smaller ones and drive out the details. Don't, if you can avoid it. At this point we want a general idea of the relative size of each story. We don't need to understand the exact nature of each individual story, we don't need enough to be able to sit down and start developing. All we need is a good understanding of the general functionality needed and a good idea of the relative cost. Stories WILL be re-estimated before they're implemented and many of them will be split into several smaller ones to make their development simpler. But we'll worry about that later.

As the stories are being added to piles be critical of past decisions. Allow stories to be moved between piles. Allow piles to be completely rebuilt. You may start off with 10 stories that spread across all 4 piles, then the 11th 12th and 13th stories come along and have nowhere to go – they're all an order of magnitude bigger than the ones that have gone before. That's OK, look at the stories you've already grouped in light of this new knowledge and recalibrate the piles.

Once all the stories are done, spread the piles out so you can see every story in each one. Appraise your groupings. Move stories around. It's important that you're comfortable that you've got them in the right pile relative to each other.

If the developers involved in this round of estimating recently worked on another project or a previous release of this project then add some recently completed stories from that work. Have the developers assign those stories into the piles. You know how long those stories took and you can use that to help guide your new estimates. There's nothing more useful in estimating than past experience... don't be afraid to use that knowledge formally.

Once you're that the piles are accurate, put a number of days against each pile. The idea is to get 'on average a single story in this pile will take x days'.

If you have to err, then err on the side of bigger. Bear in mind that people always estimate in a perfect world where nothing ever goes wrong. The stories you added from the last project can be used to guide the number. You know exactly how long it took to develop those stories and you'd expect the new numbers to be similar.

The resulting base estimate is then just the sum of (number of stories in each pile * cost of that pile). This number will give you a general estimate on the overall cost.

You'll probably want to add a contingency, and past experience on other projects will help you guide that. If you've gone through this process before, pull out the numbers from the last release... when you last performed this process and compare the starting estimate with the actual amount of time taken. Use that comparison for the contingency.

The gut feeling is to say, "Hang on, the last time we did this we estimated these 50 stories, but only built these 20 and added these other 30. That means that the starting estimate was nothing like what we actually did". But the point is that if every release has the same kind of people working on it, working for the same business with the same kinds of pressures then this release is likely to be the same as the last. Odds are that THIS release will only consist of 20 of these stories plus 30 other unknown ones... and the effect on the estimate is likely to be similar. Each release is NOT unique, each release is likely to go ahead in exactly the same way as the last one. Use that knowledge, and use the numbers you so painstakingly recorded last time.

Aside: Project managers tend to think that I have a problem with them collecting numbers (as they love to do). They're wrong... I have a problem with numbers being collected and never being used. The estimation process is the place where this is most apparent. Learn from what happened yesterday to inform your estimate for tomorrow.

Once the job's done you should have enough information to allow you to have the planning game. For those not familiar with XP, that is the point at which the customer team (with some advice from the development team) prioritises and orders the stories. It gives you a very good overall view of the direction of the project and a clear goal for the next few weeks of work.

At this point you should have a good idea of cost and scope and have a good degree of confidence in both. Now's the time to tell your boss...

Next up – using the shorter term estimates to inform the longer term one...

Technorati Tags: XP, Extreme+Programming, agile, software, development, programming, estimating, user+stories, software+development, Robert+Baillie

New Comments Feed

2007-01-05T16:49:00.000Z

I've just found out that the new version of Blogger comes with a comments RSS feed.

I've added an auto discovery link to the page, and once I get round to looking for a nice RSS Comments image I'll put a link on the top bar.

If you can't wait that long, and can't auto discover it, you can find the feed here: http://robertbaillie.blogspot.com/feeds/comments/default

Last P-dd post on BobaBlog

2007-01-05T16:07:00.000Z

OK, so I'm going to stop going on about P-dd now for a bit on this blog.

It's just to let everyone know that the P-dd blog is properly up and running now, and I'm going to try tit-bits of information on there, as well as the release announcements as code becomes available.

There's finally an easy to use link to a zip file for the latest version as I've managed to come to terms with the Google Code downloads tab (rather it was case of "oh, there's a 'download' tab, that's handy").

So, unless something pretty drastic happens in the P-dd world, this'll be the last note on the topic on BobaBlog.

Promise.

:-)

Technorati Tags: software, development, database, diagram, dot, dotty, neato, er, open+source, Oracle, Postgres, MySql, Robert+Baillie, documentation

P-dd 0.1 finally released

2006-12-31T17:20:00.000Z

So, with just 6 1/2 hours to go before the year ends I've finally managed to get P-dd - The PHP Database Documentor up to version 0.1 standard.

The blog's up and running, a slab of source code sits on Google code and at last I feel like I can stand beside the library and say "yeah, well it's good enough for now".

You can find the code here: http://code.google.com/p/p-dd/

And the blog here: http://p-dd.blogspot.com/

So what does it do?

Put simply, it's a library of PHP classes that allow for the easy generation of documentation from a set of database sources.

The idea is that, over time, database sources will be added that will allow for the collection of meta-data from all the major database players (Oracle / MySql / Postgres / etc) and produce documentation in most of the popular forms (HTML / XML / RTF / PDF / etc) including ER diagrams.

The aim is to make the library simple to use to produce either applications that output documentation for static publication or applications that allow for navigation through the database structure. Note that it is not the aim of the project to produce either of these applications, merely to allow for their creation.

It is also recognised that in the future it is desirable to take the library into a more analysis role. For example - inferring foreign keys that are not explicitly stated, either by examining the table structures or the data within those tables.

The library is very much in its early stages though, and for now we've got the following:

A database model that consists of:
- Tables
- Columns
- Primary Keys
- Foreign Keys

The database model can be created from the following sources:
- Oracle
- XML File

The model can be rendered into the following formats:
- HTML
- XML
- Graphviz Neato Diagram (producing an ER diagram)

There are also lots of other little goodies in there such as datasource in dependant filters, a datasource caching system that limits the round trips to the database and a plethora of examples showing how components can be used as well as a simple Oracle database viewer application to show off what can be possible with just a small amount of work.

I hope the code is of use, and I'm fully committed to getting more and more functionality into the code as soon as possible in the new year.

Note: The eagle eyed of you may notice that I've added a new sidebar to this blog which will list the blog posts from the P-dd blog...

Technorati Tags: software, development, database, diagram, dot, dotty, neato, er, open+source, Oracle, Postgres, MySql, Robert+Baillie, documentation

Repeating a point

2006-12-06T09:07:00.000Z

The other day I mentioned the principle "Don't repeat yourself". I think it may have inspired Andy Clarke to write this up, and he's quite right. It comes from the Pragmatic Programmers.

APC's spot on in his description as it relates to writing code, but he doesn't go far enough.

DRY relates to every part of software development, not just the bit where you're knocking out code.

If, in any part of the process, you find you have a duplication of knowledge then you have a potential problem.

Anyone ever read that comment at the top of a procedure and found its description doesn't match the code that follows?

Watched that demonstration video and found that it's showing you an utterly different version of the system to that which you've just installed?

Looked at that definitive ER diagram and found it's missing half the tables?

Well, don't put a comment at the top of the procedure, instead document the behaviour by writing an easy to read unit test for it. Whilst the knowledge might be duplicated (the test duplicates the knowledge inside the procedure), at least those pieces of knowledge are validated against each other (if you have to repeat, put in some validation)

Don't have a team writing automated functional tests and another producing videos, write your video scripts as automated tests and have them generated with every build.

Instead of manually creating a set of ER diagrams and documentation on what the system will be like, write some documentation generation software and have it generated from the current state of the database instead.

You might notice that there's a running theme here... generation. Well yup. One of the best ways of reducing the chances of discrepancies between sources of knowledge is by ensuring there is only one representation of that knowledge and generating the others.

It's one of the reasons why I've been working on the new Open Source library 'P-dd' (Php Database Documentor). It's intended be a simple library for the production of database documentation from a number of different sources - the ultimate aim is to be able to read from any of the major RDBMS systems, Wikis, XML files and suchlike and be able to simply output many different forms, HTML, GIF, PDF, XML, Open Office Doc. Over the next week I intend on letting people know where they can find it, in its early form...

Worth repeating...

2006-11-29T09:16:00.000Z

A mantra for all elements of software development...

Repeat after me:

Don't Repeat Yourself

Don't Repeat Yourself

Don't Repeat Yourself.

Testing doesn't have to be formal to be automated

2006-11-06T21:45:00.000Z

Something I hear quite a bit from people who don't 'do' automated testing is the set of excuses that goes something like this:

"Look, we know it's a really good idea and everything but we just can't afford the start-up time to bring in all these automated test tools, set up a continuous integration server and write all the regression tests we'd need in order to get it up and running. And even if WE thought we could, there's no way we'd get it past the management team."

Now let's for a second assume that the standard arguments haven't worked in response: How can you afford not to; It'll save you time in the long run; Yada yada yada. When people are in that mind set there's not much you can do...

For reasons that I'm going to explain to you right now, we have a project on the go that isn't covered by automated tests. It's an inherited system that can't have tests retro-fitted in the kind of time we have. In reality, most of the work we're doing is actually removing functionality, with a few cosmetic changes and a little bit of extra stuff in the middle. No more than a couple of weeks work.

It turns out our standard automated test tools can't just be readily fit onto the system we have.

But that's OK. It certainly doesn't mean we're not going test, and I'm damn sure that we're going automate a big chunk of it.

First of all we've separated all the functionality that can be delivered in a new module and that part WILL be fully unit and story tested. That leaves a pretty small amount of work in the legacy system. Small enough that we could probably accept the risk associated with not doing any automated testing.

But that would be defeatist.

So instead we've picked up Selenium.

Not the full blown selenium server and continuous integration hooks and whatever. Just the Firefox based IDE.

It's simple, and requires absolutely minimal set-up... it's an xpi that just drops straight into Firefox like any other extension. Having installed the IDE you get action record and playback, and nice context sensitive right click options on any page that allows you to 'assert "this ole text" appears on page' or 'check the value of this item is x'. Basically it's almost trivial to get a regression test up and running. Then you can use the IDE to run the test.

So having got that up an running, before we set about deleting huge swathes of functionality we create a regression test that ensures that the functionality we want to keep stays there. We've found that a decent sized test that covers a fair few screens and actions can take us as little as an hour to get together. To put it into perspective: we put together a test script today that took about 20 minutes to run manually. That test took us about 45 minutes to sort out in the Selenium IDE and then a minute or two to run it each time after that. So by the time we'd run it 3 times we'd saved ourselves 15 minutes!

Running it might involve executing a SQL script manually, running a Selenium script, then checking some e-mails arrive, then running another Selenium script. In short, it might involve a few tasks performed one after another. And yes we could automate the whole lot, but like I say... we just don't have the time right now. But using the tool to add what tests we can right now, to help us with our short (a few hours each) tasks means that we're building up a functional test suite without ever really thinking about it. We'll keep those scripts, and maybe in a couple of weeks we'll realise that we DO have the time to set up everything else we need for proper functional testing.

Yep, it could be a hell of a lot better (and on our other projects it is), but some informal testing using an automated runner is an order of magnitude better than no automated testing at all.

Technorati Tags: software, development, testing, test, test+automation, selenium, extreme+programming, Robert+Baillie

Going Dotty

2006-11-01T17:49:00.000Z

There's a new big thing in my sphere of interest: Dotty.

For the uninitiated: Dotty, Neato and Lefty are a family of products from Graphviz that take pretty simple text files and generate directed or un-directed graphs.

For the initiated: No, I can't believe I've never found it before either!

It was name checked during the Google LTAC by at least one presenter, and they reminded me that I'd heard its name quite some time ago and meant to look it up. When we were looking at a diagramming problem just a few weeks ago and I figured I should track it down. I was by no means disappointed.

Basically we wanted something that would graph our MVC workflow configurations to make them more readable.

That is, our MVC structure allows us to string arbitrary tasks together: perform x, if result is y, go to task z, if result is h go to task i.

The idea is to keep these configurations as simple as possible; they're only really receiving user input and then prodding objects, but still, there are some complexities. This is especially true when branches split and rejoin. For some reason, XML files or a PHP arrays can be difficult to read ;-)

Quite a long time ago we wrote a small application that would graph them in HTML, but we never liked its results. When paths split and rejoin, the HTML representation wouldn’t show the rejoin.

So, as I say, we picked up Dotty.

Simply genius.

For directional graphs the Dot output is stunning. We can pass it a file in (the trivially simple) Dot language and it'll produce great looking diagrams.
For example, the file:


digraph finite_state_machine {
 node [  fontsize="12",  fontname="arial"]
 edge [ fontsize="8", fontname="arial"  ]
 EntryPoint [ label="EntryPoint (BuildSheepFromInput)", shape="diamond" ];
 EntryPoint->SaveEditedSheep [ label="DEFAULT" ];
 SaveEditedSheep [ label="SaveEditedSheep (SaveEditedSheepTask)" ];
 SaveEditedSheep->SaveCheese [ label="DEFAULT" ];
 EditSheep__EntryPoint [ shape="box" ];
 SaveEditedSheep->EditSheep__EntryPoint [ label="ERRORS" ];
 SaveCheese [ label="SaveCheese (SaveCheeseTask)" ];
 SaveCheese->GetCheeseType [ label="DEFAULT" ];
 GetCheeseType [ label="GetCheeseType (GetCheeseTypeTask)" ];
 DisplayWensleydale__EntryPoint [ shape="box" ];
 GetCheeseType->DisplayWensleydale__EntryPoint [ label="WENSLEYDALE" ];
 GetCheeseType->DisplayCheddar__ComposeMessage [ label="CHEDDAR" ];
}

Would produce:

Stunning!

It's not difficult to write code to generate the DOT files, and the output from neato (the same as dot, but for undirected graphs) is just as high quality.

Of course, as soon as I saw the output the cogs started moving in my mind... I'm now on a bit of a brainstorm on what can come next: How about ER diagrams generated from the database schema and published on an internal site. Generated documentation is never out of date, and it's a damn site easier having it generated of the fly than it is to load up Visio and get THAT monstrosity to do the job for you.

Anyway, the ER diagramming library will be open source, and it IS on its way... I promise.

(Note: if you want more info on dotty, take a look here)

Technorati Tags: software, development, database, diagram, dot, dotty, neato, er, Robert+Baillie, documentation

Run Done, and other news

2006-10-31T13:58:00.000Z

Well, it's done. Finally the running season's finished for me after completing the Rainforest Foundation's 10km run.

My target for the year was to beat 55 minutes, and I managed it twice. Sunday's run was finished in 54:01, but I managed to beat that 3 weeks ago in the Nike Run London event... 53:23. So say I'm pleased is an understatement!

Aside: To those people protesting against Nike at the event: You have my sympathies, you really do, but it was a Nike 10km 2 years ago that go me running. I'd not long before given up smoking and the Nike run gave me a real incentive to get myself fit again (I say again, but really I'm not sure I've ever been that fit).

Now that doesn't mean that Nike is a great company that gave me my health, or anything like that. But it does illustrate a point... the vast majority of the people at the event were runners, and Nike took the time to organise a great run, probably not an audience that's going to be swayed by your argument. As I said to the protesters on the day, and I really mean this: You organise a 10km run, and I'll run it. An anti-globalisation 10km run in the middle of the most multi-cultural city in the world...

Anyway, next year I'll be stepping it up a touch more and the aim is to get under 50 minutes. The first run's booked already.

And the other news? Well, over the last few weeks I've been trying to get some coding done on a small open source project. It's a PHP library that will generate documentation on arbitrary relational databases. It's in its early stages right now, but I thought it was time to mention it and see if there's any interest out there.

It'll produce bog standard HTML documentation as well as dotty files that you can use to generate diagrams like this:

It'll be available soon...

Doing things I'm not very good at

2006-10-05T07:50:00.000+01:00

Well, last night was my last training run before my next 10Km run. And this time I've got a target...

When it's come to official timing, I've managed a 10Km in just over 58 minutes. This time I want to get it to under 55. But what's the point in me doing a run if that's the kind of time I'm going to manage? Odds on bet the winner will do it in something like 35 minutes; I'm not unfit, it's just that I'm not a very good runner; I eat fairly well and am generally pretty healthy, so I don't need to do this in order to keep fit.

I do it because I'm not very good at it. Because running reminds me that I'm not the best at things that I do. It's blindingly obvious that there are people out there that are vastly superior to me on the track. In fact, there are 60 year olds out there that are faster than me. When I did my one and only half marathon I was beaten by a veteran speed walker.

But I'm getting better. I'm learning more about running and training, and I will get faster. One day I may manage under 45 minutes... and then I'll have to pick something else.

The last run of the year will be the Rainforest Foundation 10Km, and if you know me (and want to), you can sponsor me.

Countries I've visited

2006-09-20T07:38:00.000+01:00

A friend pointed me in the direction of this little beauty: A site that generates a map of the world showing the countries you've visited.

I've got to admit that the comments on the site are more actual fun than the map, but still...

So here's mine:

Not that impressive.

Fingers crossed, if all goes to plan, in 18 months it'll look more like this:

Though still, it'll look nothing like as impressive as Tom Kyte's "countries I've in which I've given presentations" ;-)

Make your own mind up

2006-09-15T16:22:00.000+01:00

Earlier today Bob Binder made a comment on my test conference entry that included a (not too complimentary) summary of his talk. In response to his comment I made a statement that I want to make absolutely clear to everyone else...

I thought that every single one of the talks at the conference was worth attending.
Every one had value, and I think you should watch every one on Google video.

If any comment on an individual talk makes it sound uninviting it doesn't mean it wasn't a good talk. The conference quality was far above the norm, and every talk is worth seeing. And you can, they're on Google Video

Other than that, I invite comments from the speakers, other attendees and anyone that might disagree with me. Just because I have an opinion, doesn't mean that I don't want to hear yours! But don't expect me to agree with you ;-)

The Google Test Automation Videos are here...

2006-09-15T09:12:00.000+01:00

Well, not here, but here

I'll update the earlier posts with each relevant video link when I can...

XHTML Transitional

2006-09-13T21:56:00.000+01:00

OK, so it's taken a bit of time, though very little effort (if you know what I mean), and I think it's probably one of the sadest exercises I've ever gone through (pretty difficult to believe, I know)... BUT the home page of this site is now XHTML Transitional compliant.

It's been validated in two seperate sources.

First up, the obvious: W3C 's online validator

Secondly (because one is never enough), the much handier HTML Validator Firefox extension by Marc Gueury, which is based on the Tiny validator.

The plan is to keep any new posts to the site XHTML transitional compliant. Though I've got no plans to go back and fix earlier posts. Frankly I've got better things to do with my time than remove the <br /> tags that Blogger sticks between <li> tags and change all the &s to &s in the hrefs. :)

At least the site finally looks the same in Firefox and IE6 now!

Update: Is it just me, or is the commented out CDATA tag inside the script tag telling the browser that there's unstructured data a bit of a hack. I know my javascript should be in a different file, but I don't have any hosting for the files, and I'm not going to set it up for 2 or 3 1K files!. Anyway, you'd think the spec would cover this problem in a more natural way. Maybe an attribute on the tag

unstructured="true"

Maybe

type="CDATA"

Maybe it can be implied by the script tag itself?

I'm no XML expert, but the solution seems more than a bit nasty to me.

Google LTAC - A more personal note

2006-09-13T13:54:00.000+01:00

Alright, so I've been going on about the Google LTAC quite a bit recently, but I wanted to mention a few more personal observations...

Is it coincidence that the comedy presenters were called Adam and Joe?
Google might talk about Work / Life balance, but you're always at the conference even when you're on the toilet thanks to the highly informative 'testing on the toilet' hints and tips sheets ;) (see below)
Even the user interface 'simplicity innovators' can't help themselves when it comes to conference freebies... I never realised that I needed a coloured light on my pen until now. (Shame the light makes the pen a bit too heavyweight, the ink keeps clogging meaning a slow startup time and the blue LED keeps on cutting out)
Also, for a company that's very much 'of the now', a mouse mat is just soooo last millennium.
I've never been at a conference where there were so many laptops. Although I'm a little surprised that no-one brought any internet enabled CI lava lamps with them.
Google may not be Evil, but they still gave us plastic cutlery and polystyrene plates (boooo)

And a couple of awards

Phrase of the conference: "That's a big bucket of suck"
Agile Pimp: Dan North, a man with an eye for spotting the delegate that's ripe for a bit of lean process
Free snack food of the conference: Innocent Smoothies.
Information Download Award: Adam Porter, watch the video (when it's out), you'll understand.
Demo of the conference: Jason Huggins, the cutting edge can cut you deeply when you've got an audience.

Spare a moment for the little people

2006-09-12T13:54:00.000+01:00

They're here, and they may need your help.

Also, it's good to see the new world order we were promised has finally arrived, though if might appear that Disney World has taken the political situation a bit far over here

A language for discussing performance testing

2006-09-10T18:13:00.000+01:00

OK, so all I'm doing here is repeating the text that I previously had in a post about the Google London Automation Testing Conference, discussing the talk by Goranka Bjedov on Performance testing.

I figured that it was sitting in the middle of a fairly large post, and I wanted it to be seen and reviewed by more people than would be bothered to plough through the other stuff.

It's a suggested series of terms by which different types of performance tests can be described:

Performance Test: Given load X, how fast will the system perform function Y?
Stress Test: Under what load will the system fail, and in what way will it fail?
Load Test: Given a certain load, how will the system behave?
Benchmark Test: Given this simplified / repeatable / measurable test, if I run it many times during the system development, how does the behaviour of the system change?
Scalability Test: If I change characteristic X, (e.g. Double the server's memory) how does the performance of the system change?
Reliability Test: Under a particular load, how long will the system stay operational?
Availability Test: When the system fails, how long will it take for the system to recover automatically?

In addition to the above, there is then another term, which I would suggest is not a type of test in its own right, rather it is a term denoting the depth of analysis being performed.

Profiling: Performing a deep analysis on the behaviour of the system, such as stack traces, function call counts, etc.

Any thoughts?

Google Automated Testing Conference (London) - Day 2 (part 2)

2006-09-09T12:24:00.000+01:00

Selenium: The in-browser acceptance testing tool (Google video)

Another talk I was looking forward to was Jason Huggins, the creator of Selenium talking about his tool. I was hoping for a little on the basics of the tool, but really the idea is simple enough to be easy to describe.

It is a test framework that allows you to build Fit like (HTML table), or code based UI tests and then drive your web application through multiple browsers. The result is a solution to the web version of the mobile problem... How do you test the against the diversity of target platforms.

The product ships with an IDE that sits in Firefox and allows you to record actions against the application, then edit the resulting test scripts. These scripts can be in any of many languages such as Java, C# and Ruby. I did notice the lack of PHP support though. Shame, but does it really matter?

The bulk of his talk was then one some of the possibilities you can get from this. In particular, his demo was to check a change into subversion, this would kick off cruise control and run the unit tests for his app. The successful build message would then get picked up by a number of listeners running in different OSs running on virtual machines. These listeners would kick off Selenium for different browsers, all run the same set of tests and record the resulting actions as movies.

His ultimate aim was then to add some voice over and these screencasts could be used as marketing videos: in fact, he suggested that the Apple voice synthesis would be good enough for that job (especially the new version coming out soon), and the spoken text could be in the test itself.

Very nice idea. it's a shame the demo didn't actually work ;)

One point made was that the movies are going to get big, and maybe you wouldn't want to keep them for all versions. Maybe, but then maybe you don't need movie files... Maybe just the html files will do. You can then wrap it up in a player that will move you between pages and kick off the right bit of audio at the right time. A selenium driven app won't show you the mouse pointer, so you don't loose that... You could highlight clicks and focus with a bit of nifty css. Alas I didn't get a chance to pass the idea on to Jason as he was (unsurprisingly) surrounded for most the rest of the day.

Main message: When you're innovating, your demos might not be as smooth as you want them to be ;)

Or maybe

Main message: The DRY principle (don't repeat yourself) crosses so many boundaries it's crazy. Be aware that there are many ways you can re-use the same data / process / tools, and automation can be applied to many many things.

Testing Metro WiFi (No Google video yet :( )

Karl Garcia of Google was next up, talking about testing the open wireless network in Mountain View, California.
This new network covers about 12 square miles of reasonably densely populated urban area. The network consists of just under 400 wireless nodes sitting on the top of lampposts spread no more than about 150 yard apart. This mesh is then connected to 3 base stations via a number of gateways spread more sparsely.

The question is, how do you automate the testing of such a beast.

The testing was covered in two main stages. First the coverage test... Simply taking a device that will poll the network and driving it down every street. Hook that up to GPS and get it to record the network strength and you've got stage 1 covered.

The second is the throughput testing. For this the team grabbed cheap routers and PDAs, installed Linux and iPerf (network testing tool), setup their start-up so that they'd report in to the main server and get ready to run test, make sure that if they fail they go into a restart mode and then gave them cheap solar panels. Then it was just a case of picking a spot to test, taking the clients out and leaving them.

Back at the office the test server polls for the clients, gets them to run the suites and reports the results. OK, so the machines have to be taken out and placed, but a small number of inexpensive components makes the kit cheap. If the client machines go down then you just need to wait for them to reboot and they'll get back in touch. No need to go out and reset them.

When you need to change the test area you just go out, pick the clients up and move them. Karl admitted that they could have had more clients and that they considered having more expensive bits, but in the end the kit they had was stupidly cheap and more than fit for purpose.

Main message: Not every part of the process can be automated, but you can minimise those bits that can't and still get great benefits. The most expensive way of doing that isn't always the best.

Distributed Continuous Quality Assurance: (Google video)

And the last full talk of the day was Adam Porter. With an incredible bio (in terms of academia), it seemed as though Adam thought the lightning talks had started early and he powered through 45minutes of information dispensing, at an incredible speed.

His discussion as very much at an academic level, but the system he talked about sounds like it may be one of the next big things in testing.

The starting premise is this: traditional testing strategies are failing large scale development because the variations possible in the configuration of most new systems are enormous. It simply isn't possible to test the full variation of configuration options, deployment operating systems, setups of those operating systems and so on.

His system (Skoll) attempts to address that.

I'm going to go into no depth on this topic whatsoever because the amount of information was overwhelming, but I'll try to give an overview.

By splitting QA tasks into small chunks of work, defining the valid combinations of deployment, providing a means of producing those variations and using a grid of machines to test on it is possible to cover an enormous amount of combinations in your tests.
If you then get smart about which combinations should be tested then you can statistically cover the whole set without having to actually test them all.

You can test disparate configurations until you find a failure, then take that configuration, change it in a single dimension and then test again... Feel out the extent of the failure scape.

You can then datamine those results to try to provide clues on the underlying problem.
He gave compelling arguments that certain strategies for choosing particular configurations work and followed every one with a case study that tested it empirically.

He also showed how the approach can be used for tracking performance problems with a selective method of benchmarking.

It was a very solid, very info laden, well thought out talk n a great approach. When the video comes out, I urge you to watch it. At half speed.

And if you're really interested in reading the papers, there's one here and you can find many more (or copies of the same) by searching on Google for Skoll testing.

Main message: It's not just about the volume of QA you perform, it's about the QUALITY of the QA. By getting smarter in your testing you can cover more of the application in less time.

Lightning Talks: (No Google video yet :( )

Finally it was the turn of the lightning talks. 10 speakers, 10 subjects, 5 minutes per speaker, no exceptions. I'm not going to cover much of the ground here because it was, of course, a lot of info in a small amount of time. It's worth picking up the video when it's available and watching it. The quality's a bit hit and miss, but you can always skip the 5 minutes of the person you don't like.

Highlights were Dan North on 'Getting Lean' – what it means and how automation helps you get there, Steve Freman and Nat Price giving us a glimpse of jMock, particularly of jMock 2 (looking tasty). James Lyndsay reminding us that "automation good" does not equal "manual bad" and Jordan Dea-Mattson reminding us that our QA processes need to have "defence in depth", or "overlapping fields of fire".

All in all, an excellent 2 days!

Google Automated Testing Conference (London) - Day 2 (part 1)

2006-09-09T11:42:00.000+01:00

So, day 2 has been and gone, and it seemed slightly quieter than day 1.
It seems like the free bar took its toll. Despite the slight drop in numbers, the conversations seem a little more free flowing. Damn it! Lesson to earn from this conference: 'No matter how bad you feel, go to the free bar'.

Anyway, this time round I'm going to tackle the blog entry in two chunks... Thursday's entry was just too damn big!

Objects - They just work: (Google video)

I may be biased here, but I was really disappointed with this one. It looked like it might be a talk on the decomposition of objects into the simplest form so that they just work.

It wasn't.

The title was a reference to the NeXTSTEP presentation given by Steve Jobs way back in 92 when he said the same. The point being that they didn't just work. There was a lot of pain and hardship to get them to work, and that's a recurring theme in software development.

So the talk was given by Bob Binder, CEO and founder of mVerify. He discussed a few of the difficulties and resulting concepts behind their mobile testing framework.

We've already had a discussion on the difficulties involved (permutations), so that didn't tell us anything we didn't already know.

He moved on to mention TTCN (Testing and Test Control Notation) an international standard for generic test definition. It's probably worth a look.

He also mentioned the fact that their framework added pre and post conditions to their tests - require and ensure. I may be a die hard stick in the mud here, but the simplicity of 'setup – prod – check – throw away' seems like a pretty flawless workflow for testing to me. Though I admit, I could well be missing something: If anyone can enlighten me on their use I'd be reasonably grateful. Though thinking about it, I wasn't interested enough to ask for an example then, so maybe you shouldn't bother ;)

One good thing to come out was the link to that NeXTSTEP demo.

I really want to say more and get some enthusiasm going, but sorry Bob, I just can't.

Main message: Try as they might, CEOs can't do anything without pimping their product and glossing over the details.

Goranka Bjedov - Using Open Source tools for performance testing: (Google video)

After the disappointment of the first talk, this one was definitely a welcome breath of fresh air.

Like the first time I read Pragmatic Programmer, this talk was packed full of 'yes, Yes, YES' moments. If you took out all the bits I had previously thought about and agreed with you'd be left with a lot of things I hadn't thought about, but agreed with.

When the videos hit the web you MUST watch this talk.

Goranka proposed a vocabulary for talking about performance tests, each test type with a clear purpose. Having this kind of clear distinction allows people to more clearly define what they're testing for, decide what tests to run, and ultimately work out what the test results are telling them.

Performance Test – Given load X, how fast will the system perform function Y?
Stress Test – Under what load will the system fail, and in what way will it fail?
Load Test – Given a certain load, how will the system behave?
Benchmark Test– Given this simplified / repeatable / measurable test, if I run it many times during the system development, how does the behaviour of the system change?
Scalability Test – If I change characteristic X, (e.g. Double the server's memory) how does the performance of the system change?
Profiling – Performing a deep analysis on the behaviour of the system, such as stack traces, function call counts, etc.
Reliability Test – Under a particular load, how long will the system stay operational?
Availability Test – When the system fails, how long will it take for the system to recover automatically?

I would probably split the profiling from the list and say that you could profile during any of the above tests, that's really about the depth of information you're collecting. Other than that I'd say the list is perfect and we should adopt this language now.

She then put forward that infrastructure you need in order to do the job.

I don't want to be smug about it, but the description was scarily similar to that which we've put together.

Alas, the smugness didn't last long because she then went on to tell us the reasons why we shouldn't bother trying to write this stuff ourselves... That the open source community is already rich for doing these jobs, directing us to look at Jmeter, OpenSTA and Grinder. A helpful bystander also directed us to opensourcetesting.org - there are a lot of test tools on there.

Fair enough... I admit we didn't look when we put together our test rig, but you live and learn. And I'll definitely be taking a look for some DB test tools.

A big idea I'll be taking away is the thought that we could put together a benchmarking system for our products. This isn't a new thought but rather an old one presented in a new way. Why shouldn't we put together a run that kicks off every night and warns us when we've just killed the performance of the system. It's just about running a smoke test and getting easy to read latency numbers back. Why not? Oh, I'll tell you why not... We need production hardware ;)

She then gave us a simple approach to start performance testing with, a series of steps we can follow to start grabbing some useful numbers quickly:

Set up a realistic environment
Stress test
- Check the overload behaviour
- Find the 80% load point
Build a performance test based on the 80%
- Make it run long enough for a steady state to appear
- Give it time to warm up at the start
- Collect the throughput and latency numbers for the app and the machine performance stats.

If I wasn't already married, I might have fallen in love :)

Main message: You CAN performance test with off the (open source) shelf software, it just takes clarity of purpose, infrastructure, a production like deployment and time.

Oh, and you're always happiest in a conference when someone tells you something you already know ;)

Testing Mobile Frameworks with FitNesse: (Google video)

As the last one before lunch Uffe Koch took the floor with a pretty straightforward talk. By this time I was sick of hearing about mobile testing ;)
The thing is, the manual testing problem is so big with mobiles that it's prime for automation.

It turned out that he gave a pretty good talk on the fundamentals of story (or functional) testing practice. For a different audience, this would have been a fantastic talk, but unfortunately, I think most people here are already doing many of the things.

A lot of the early part of the discussion crossed over between the Fit and Literate Testing talks from the day before, though the ideas weren't presented in the same kind of depth. The suggestion that the test definition language of 'click 1 button' was a domain was pushing it, but the point is reasonably valid. The structure of test definition languages need to be very different to the programming languages we're used to. This is one of the winning points of Fit, since its presentation is very different to Java, C# or whatever it's approached by the developers in a very different way. Kudos to Uffe for realising this explicitly and producing a language for driving the app framework.

His team have put together a UI object set that can be driven through the serial or USB port of a phone and can report the UI state back to the tester at any time, passing it to the tester as an XML document.

It's very similar to our method. We do it so that the test don't need to know about page structure, just the individual components; so we don't need to worry with things like XPath when we want to extract data from the page; so our story tests aren't as brittle as they could be. They're doing this to solve the problem of screen scraping from a phone.

It's an elegant solution to testing these phones and whilst Uffe admits that it means you're not testing the real front end, or the real screen display, it allows them to hook up a phone to the test rig and run the full suite. I'm sure those tests must take and age though... doing a UI test of a web page is bad enough, but some of those phones can take some time to respond! I'd like to see the continuous integration environment. I've got an image of 500 Dell machines hooked up to different phones through masses of cables. That'd be cool!

The common FitNesse question did come up: How did you address the version control of the FitNesse scripts. Like everyone else (it seems), the archiving was switched off, local copies of the wikis were created and they got checked into the same version control as the code when they were changed. I really feel I've got to ask the question: If this is the way everyone does it, why isn't there an extension to the suite to allow this out of the box?

Main message: With a bit of thought and design even the most difficult to test targets can be tested. You just might need a tiny touch of emulation in there.

And that led us on to lunch...

Google Automated Testing Conference (London) - Day 1

2006-09-07T19:42:00.000+01:00

Google Automated Testing Conference (London) - Day 1

(Get ready, it's a long one)

So, finally the Google testing conference has come around, and it's pretty good at making a man feel like a small fish in a big pond. It's pretty clear that the place is populated by developers who are all working in some form of test driven way. A large number contribute to open source software, and many of those are working on test frameworks... 3 of the 4 main contributors to jMock are in attendance. I don't mind admitting that I feel like a bit of an interloper.

But before I start, I've got to point out that (of course) all the words here are my own interpretations of the presenters words... and I could very easily have got it all very very wrong. But then that would be their fault for presenting badly ;)

Also, Google assured us that the talks will be available on Google video, and that many supporting links will be sent out to attendees. Once I get those things I'll update this entry to keep you updated.

Anyway, the first day didn't disappoint:

Distributed Testing with SmartFrog: (Google video)

Steve Lougran and Julio Guijarro talked about their HP Labs research project on testing distributed systems: SmartFrog. They've got a good looking framework together that allows you to define a system deployment as a class hierarchy and then describe their relationships. That means that you can use it to state when and where components need to be deployed, services need to start and so on. A nice tool for rolling out complex distributed systems.

But their interesting points came when they talked about using it for testing. By wrapping a few testing frameworks (JUnit for one) and then describing the test suites as components to install, they're producing a framework that allows you to test each component of the system in a place similar to where it would run when in production. Not only that, but it allows you to install emulated components such as switches, routers and flaky proxy servers allowing you to test on complex and failure prone infrastructure. Not bad.

Their main problems at the moment seem to come from trying to then collect all the test results, logs and suchlike and compiling that into a reasonable test result summary. It's fine when things pass, but as soon as you get a lot of failures it starts to stutter. But that problem's surely surmountable (if dull to solve ;) ). A few people out there must be looking at the same thing under a different guise.

One great thing to come out of it was the call to arms to those producing test frameworks... Where is the common reporting standard? A guy working on SimpleTest (sorry, didn't catch his name) seemed to be up for it...

Main message: Test using real deployments, not just idealised or local versions.

Literate Functional Testing: (Google video)

Robert Chatley and Tom White of Kizoom talked about their functional testing framework that extends the Knuth idea of 'Literate Programming'. It's most succinctly described in Knuth's own words: "Let us change our traditional attitude to the construction of programs. Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do."

The aim is to end up with a language that can be used by both developers and customers; that the resulting test code can be read by non developers and therefore can by used by the customer to validate the developers' interpretation of the system requirements.

They have produced a language that takes few jMock ideas (like constraints) and uses them to produce truly elegant code.

So, the test cases become along the lines of


assertThat( currentPage, has( 3, selectBoxes.named('Region', 'Time', 'Method' ) ) );

The idea is lofty, looks pretty damn good and definitely reverberates with my own ideas on producing story tests. They drive through the user interface and are easy to read. Implementing the same tests in a more traditional Java approach leads to a very difficult to read lump of code.

It's a shame they haven't taken it a little further so that the code is a fully readable English script rather than a halfway house between English and Java, but I love it none the less.

Oh, and like any good tool should, it's going open source...

Main message: It's possible to write functional tests that can be read by your customer, it just takes a different approach to the rest of your code.

Testing using Real Objects: (Google video)

Massimo and Massimo (Arnoldi and Milan) talked about their approach to generating test data for their (sounds) highly successful company Lifeware. By no means particular to their market (life insurance), they find that bugs don't occur in their system with short lived data. Rather it's the contracts that have lived in the system for years, and have a huge amount of complex events in their lifetime are always the ones that fail. Those contracts are basically atypical of the ones that are usually used for testing.

They found that there was always a great deal of difficulty in producing test cases for these large data sets and so have produced a method of exporting data from live systems an importing it into the test suite.

They admitted that it was born of a data migration tool, and you can see how. Having identified an object that needs to be extracted they generate a set of events that will re-create it. Those events are simply any data changes that you would need to create the primary and related objects in the state that they exist now. If a contract has a number of payments against it, then you'll see a number of distinct payment events in the list.
This list of events is then translated into a series of method calls that can be used to recreate the object in another environment. Having created the data set it's merely a case of describing whatever assertions you need in your test.

It sounds like there's some clever Smalltalk code in the background going on, and much of the talk was on that, but it's the idea that's the important component.

As a means of extracting or generating test data it sounds great. The events list is a neat solution. However, as a means of describing a data migration it sounds phenomenal! And that's where I really see the benefits. Being of a DB background that's no real surprise ;-)

If you can always generate data in your system from a set of precise events then when you need to migrate data from an external system you don't need to create a data-mapping, you need to create an event mapping. Customers are notoriously bad at data mapping because the data often not in a form they recognise. But these events sound to me like a domain language just waiting to jump out.

Main message: Test using real data (objects), not just idealised versions.

Doubling the value of Automated tests: (Google video)

Next up was Rick Mugridge, the man behind FitLibrary (an extension to Fit a means of specifying and running tests), and co-author of 'Fit for Developing Software'. He's a big believer in story tests and much of his work seems to be around getting the process of producing them as slick as possible.

His big idea is 'Story-test Driven Development' (SDD). That is, taking the idea of Test Driven Development (TDD) a step further and getting system requirements specified in tests as early as possible in the process. The 'story' to which the TLA relates is the Extreme Programming idea of a story. A single action that a user wishes to perform with a system.

He proposed that writing story tests in a language that describes user interface interaction is like programming in assembler. It has its uses, but is far too low a level for many purposes; that using a low level language can hide the underlying purpose of the test, being to test the business rules of the application with respect to the story.

In producing a story test you should be talking in the domain language not a UI language. By doing so the business rules become apparent and the vocabulary becomes clear enough to be used by business analysts, product managers, the customers. He advocates the use of story tests as a means of the customers defining the requirements of the system. That these can then be further refined by the developers and the test teams, but that the customers ultimately own them.

Also, if the purpose of the story test is to convey information between disparate parties (in both geography and time), then concise, concrete examples are the way forward.

I whole heartedly agree with the basic premise that there is a different approach to testing that can be forgotten about, namely testing the integration of objects in the domain language. I'm just not sure it replaces the UI testing. I'm not 100%, but I don't think that Rick was suggesting this.

Also, I'm really not sure about the Fit method of showing tests in tables (example here). I like the idea of a non text based representation, but I'm just not sure the tables really work for anything other than trivial examples. Still, I was very impressed with his notion that tests could be described in diagrams.

Main message: Test in the domain language, not in a UI language. If you do that you can always generate a UI test, and your tests will express the essential business rules rather than workflow.

Auto-test, Push button testing using contracts: (Google video)

Next it was Andreas Leitner's turn, A PHD student working with ETH Zurich.

In typical PHD style, his talk centred around Design by Contract. For those that aren't familiar with the concept he described the ideas of the pre and post conditions and invariants that are apparent in such languages (Eiffel being his language of choice). The idea is that for a given method there will be defined:
Pre-conditions - The conditions that must be true before that method is called.
Post-conditions – The conditions that the method guarantee will be true once the method is complete.
Invariants – The conditions that can never be broken.

The framework Andreas has put together can be used to test Eiffel classes to ensure that the post and invariant conditions are never broken. The innovative approach that this framework takes is that it does not require the developer to produce any test code. Instead test code is generated based on a 'strategy', for which there are already a number created.

The most basic of those strategies is the purely random: Create a bunch of objects, call methods on them with parameters that pass the pre-conditions, make sure the post conditions are true. He also offers what he calls an 'Adaptive Random' strategy, where each object tested is aimed to be as different in structure from the last as possible, and AI based strategies influenced by the well understood maze solving technique of World /State / Goal definition.

These tests are then intended to run for a large amount of time, unlike the traditional unit test idea of 'run as fast as you can through the interesting cases'. This then becomes a brute force attack on the objects, attempting to break them pretty much by chance.
On finding a failure case, the framework will then try to extract the essential nature of the test and provide you with a script that can be added as a standard unit test to your suite. The obvious (but clever) way of checking the generated test script is valid... it runs the script again and checks it fails.

The big win is the fact that this can be used to test third party libraries without having to define the tests yourself. OK, so the intent of each method isn't really tested, merely the accuracy of the post-conditions and invariants, but that's definitely better than just taking everything on trust.

He also asserted that you don't need explicit pre and post conditions in your language in order to use this technique. Java has extensions that provide the capability, and SPEC# does something similar in the C# world. Also, it was pointed out that there are other tools doing similar things, Agitar being one that takes Java classes and trying to work out how to break them.

Main message: It's possible to produce auto generated test cases for classes as long as you have (or can infer) design by contract components and enough time.

Does my button look big in this? Building Testable AJAX Applications: (Google video)

Finally some Google guys got in on the act and Adam Connors and Joe Walnes gave us a presentation on how to test AJAX apps.

In reality this was a pretty generic talk illustrating the fact the any type of application in any language can be decomposed into testable components. It quite rightly put forward the idea that the industry (or more accurately, the people in it) is still naive when it comes to writing Javascript. Good practice goes out the window as soon as the manipulation of a Document Object Model and an XMLHTTPRequest object comes into play.

But, as they demonstrated, it's not that hard to design Javascript code in a way that can be tested, it just takes a little bit of thought and a lot of discipline.

Still, they did suffer the wrath of the audience when they suggested that the DOM interaction and the View code doesn't need to be unit tested. They proposed that it's OK not to unit test some components just as long as those components are as simple as possible in what they do. Contentious, but it does have merit. I tried to suggest that the story tests can take care of that, but Joe didn't seem to want to bite!

Main message: AJAX apps are like any other type of app. There are easy to test bits, hard to test bits and seemingly impossible to test bits. Separate the bits and you make your life easier. The fact that it's Javascript is no excuse for bad design.

And that was day one. I could have gone to the pub, but in the end it was far too exhausting for me and I took my free t-shirt, fancy LED laden pen and Google notepad and skulked off home to prepare for tomorrow (and write this entry, of course).

If tomorrow's as good as today, I'll be exhausted, happy and a lot richer for the experience!

Update: As per the multiple requests by Google, this post is tagged: Google LATC

Configuration is the new code

2006-09-07T19:29:00.000+01:00

Fairly recently I was thinking about the development processes for the configuration of a large, off the shelf system. you know the type; CRM, ERP, TLA ;), that kind of thing. All things to all people, completely generic, no need to do any development to get it just right for you business, just a bit of configuration needed.

Only it's not just a bit of configuration, it's a lot of configuration. And with the business world the way it is, it's ongoing configuration much the same as it's ongoing development for every other bit of software we have.

So, if we're going to have a team of people continually working on configuring this system, configuring the system is basically changing the behaviour of the system, then what differentiates it from source code?

As far as I'm concerned, nothing.

When the configuration of the system goes as far as it does on the particular system (and it's not alone), then the configuration of the system has to be dealt with as if it's the source code of that system. It has to undergo the same quality checks, regression tests, audited rollout processes, version control.

The particular product I was looking at has had some functionality added to support these kinds of ideas. It has a clear migration method to get from development to test to staging to live. It supports that kind of structured, scripted rollout. But the config (development) tool can be attached straight to the live environment and be used to 'just make a quick change'. And there's nothing you can do to lock it down.

The configuration all lives in a database, so you can't just simply check the configuration in and out of version control. The development tool does has some version control integration, but it doesn't allow you to branch, tag or, most importantly, revert. Not only that, but the dev tool can be used to change any number of configuration sets, but when you flick between them the version control module you're using doesn't change. So you can check a config from one environment into the version control module of another!

So I find I have to ask the question... What's the point in having the option if it's so hopelessly crippled?

My only conclusion is that there is none!

Anyway, the process isn't completely doomed, there is a process that will allow us to make sure our release versions are version controlled and tagged, and therefore audited.

Unfortunately, since the solution means putting a single binary (rather than multiple files) into version control we loose many of the day to day benefits of version control, like granular logs of changes and the ability to diff. But hey, at least our process is auditable.

The whole way through the examination I was told by consultants that "most people don't do this" and "I've never worked on a project where people thought version control was necessary". Probably very true... But that's because a lot of the industry doesn't know what it's doing when it comes to software development.
It's a big shame, because the inclusion of the migration tools and the lip service towards integrated version control points to the fact that they've started to think about it. It's just that it's not very well thought out yet.

On day soon, the big players will wake up, provide the proper tools for version controlling their configurations and maybe then the rest of the industry will learn to use it.

Hopefully, the Google Test conference I'm attending this week will give me some ideas on how to add automated regression tesing, and plug another gap in their toolset...

Well I Never - Followup 1

2006-08-31T14:02:00.000+01:00

OK, so I've managed to grab some time during the day to experiment, and I've got things to post. For now I've just got the time for this...

Turns out that William Robertson was quite right, the TO_CHAR 'too many declarations' issue has gone away (certainly by the time it reached 9.2), and I never even noticed!


SQL*Plus: Release 9.2.0.1.0 - Production on Thu Aug 31 13:53:25 2006

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.


Connected to:
Oracle9i Release 9.2.0.6.0 - Production
JServer Release 9.2.0.6.0 - Production

SQL> SELECT TO_CHAR( 'CHARACTER' ) FROM DUAL
  2  /

TO_CHAR('
---------
CHARACTER

SQL> SELECT TO_CHAR( NULL ) FROM DUAL
  2  /

T
-


SQL>

Second up (also in 9.2) the first suspicion I had was quite right... the following doesn't work.


SQL*Plus: Release 9.2.0.1.0 - Production on Thu Aug 31 13:42:49 2006

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.


Connected to:
Oracle9i Release 9.2.0.6.0 - Production
JServer Release 9.2.0.6.0 - Production

SQL> CREATE OR REPLACE PACKAGE test_pkg IS
  2    --
  3    FUNCTION cannot_be_overloaded RETURN NUMBER;
  4    FUNCTION cannot_be_overloaded RETURN VARCHAR2;
  5    --
  6  END test_pkg;
  7  /

Package created.

SQL>
SQL> CREATE OR REPLACE PACKAGE BODY test_pkg IS
  2    --
  3    FUNCTION cannot_be_overloaded RETURN NUMBER IS
  4    BEGIN
  5      RETURN 0;
  6    END cannot_be_overloaded;
  7    --
  8    FUNCTION cannot_be_overloaded RETURN VARCHAR2 IS
  9    BEGIN
 10      RETURN 'Character';
 11    END cannot_be_overloaded;
 12    --
 13  END test_pkg;
 14  /

Package body created.

SQL> SELECT test_pkg.cannot_be_overloaded FROM DUAL
  2  /
SELECT test_pkg.cannot_be_overloaded FROM DUAL
       *
ERROR at line 1:
ORA-06553: PLS-307: too many declarations of 'CANNOT_BE_OVERLOADED' match this
call

SQL> DECLARE
  2    vn_number  NUMBER;
  3    vc_character  VARCHAR2(100);
  4  BEGIN
  5    vn_number := test_pkg.cannot_be_overloaded;
  6    vc_character := test_pkg.cannot_be_overloaded;
  7  END;
  8  /
  vn_number := test_pkg.cannot_be_overloaded;
                        *
ERROR at line 5:
ORA-06550: line 5, column 25:
PLS-00307: too many declarations of 'CANNOT_BE_OVERLOADED' match this call
ORA-06550: line 5, column 3:
PL/SQL: Statement ignored
ORA-06550: line 6, column 28:
PLS-00307: too many declarations of 'CANNOT_BE_OVERLOADED' match this call
ORA-06550: line 6, column 3:
PL/SQL: Statement ignored

SQL>

However, my second suspicion was off the mark (at least in 9.2). Almost certainly this is related to the change in behaviour to TO_CHAR described above,


SQL*Plus: Release 9.2.0.1.0 - Production on Thu Aug 31 13:45:47 2006

Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.


Connected to:
Oracle9i Release 9.2.0.6.0 - Production
JServer Release 9.2.0.6.0 - Production

SQL> CREATE OR REPLACE PACKAGE test_pkg IS
  2    --
  3    FUNCTION can_be_overloaded ( pn_number   NUMBER   ) RETURN NUMBER;
  4    FUNCTION can_be_overloaded ( pc_varchar  VARCHAR2 ) RETURN VARCHAR2;
  5    --
  6  END test_pkg;
  7  /

Package created.

SQL>
SQL> CREATE OR REPLACE PACKAGE BODY test_pkg IS
  2    --
  3    FUNCTION can_be_overloaded ( pn_number   NUMBER   ) RETURN NUMBER IS
  4    BEGIN
  5      RETURN pn_number;
  6    END can_be_overloaded;
  7    --
  8    FUNCTION can_be_overloaded ( pc_varchar  VARCHAR2 ) RETURN VARCHAR2 IS
  9    BEGIN
 10      RETURN pc_varchar;
 11    END can_be_overloaded;
 12    --
 13  END test_pkg;
 14  /

Package body created.

SQL> SELECT test_pkg.can_be_overloaded( 0 ) FROM DUAL
  2  /

TEST_PKG.CAN_BE_OVERLOADED(0)
-----------------------------
                            0

SQL> SELECT test_pkg.can_be_overloaded( 'WORD' ) FROM DUAL
  2  /

TEST_PKG.CAN_BE_OVERLOADED('WORD')
--------------------------------------------------------------------------------
WORD

SQL> SELECT test_pkg.can_be_overloaded( '100' ) FROM DUAL
  2  /

TEST_PKG.CAN_BE_OVERLOADED('100')
--------------------------------------------------------------------------------
100

SQL> DECLARE
  2    vn_number     NUMBER;
  3    vc_character  VARCHAR2(100);
  4  BEGIN
  5    vn_number     := test_pkg.can_be_overloaded( 0 );
  6    vc_character  := test_pkg.can_be_overloaded( 'WORD' );
  7    vc_character  := test_pkg.can_be_overloaded( '0' );
  8    vn_number     := test_pkg.can_be_overloaded( TO_NUMBER( '0' ) );
  9    vn_number     := test_pkg.can_be_overloaded( TO_CHAR( 0 ) );
 10  END;
 11  /

PL/SQL procedure successfully completed.

SQL>

Cheers to everyone who commented on the last post... it's led me to check out a few things that I might not have bothered with and I reckon I'll be looking a little deeper in the next few days. Contrived examples of where named parameter notation could go wrong are called for I think ;-)

Well I never

2006-08-29T22:27:00.000+01:00

Good to be reminded that there's always something that you don't already know. And that's especially true of Oracle. I'd never suspected that some of those things are pretty fundamental, like the fact that package functions and procedures can be overloaded! I'd always assumed that since standalone functions and procedures can't, that the same was true of packages. Turns out that assumption was all wrong...

I.E.
This doesn't work:


CREATE FUNCTION cannot_be_overloaded RETURN NUMBER IS
BEGIN
  RETURN 0;
END cannot_be_overloaded;
/

CREATE FUNCTION cannot_be_overloaded ( pc_varchar  VARCHAR2 ) RETURN VARCHAR2 IS
BEGIN
  RETURN pc_varchar;
END cannot_be_overloaded;
/

But this does!


CREATE OR REPLACE PACKAGE test_pkg IS
  --
  FUNCTION can_be_overloaded RETURN NUMBER;
  FUNCTION can_be_overloaded ( pc_varchar  VARCHAR2 ) RETURN VARCHAR2;
  --
END test_pkg;
/

CREATE OR REPLACE PACKAGE BODY test_pkg IS
  --
  FUNCTION can_be_overloaded RETURN NUMBER IS
  BEGIN
    RETURN 0;
  END can_be_overloaded;
  --
  FUNCTION can_be_overloaded ( pc_varchar  VARCHAR2 ) RETURN VARCHAR2 IS
  BEGIN
    RETURN pc_varchar;
  END can_be_overloaded;
  --
END test_pkg;
/

I'm sure there are gotchas in there, and I'm not really sure it's actually that useful (I've gone 8 years without it ;-) ), but still... how did I miss it? What else have I missed?

Update - aside: Why does Oracle allow overloading in the package, but not with standalone. I'm guessing, but probably because not allowing the standalone overload makes things like the 'DROP PROCEDURE' command a lot simpler to use (care to specify which procedure with that name to drop?)
Probably because allowing the package procedures to be overloaded seemed like a god idea to someone in Oracle ;-o

Update: Just re-reading the bulk of the post... and now I'm blogging something that I've not tested (no Oracle at home). But I reckon that the following won't work:


CREATE OR REPLACE PACKAGE test_pkg IS
  --
  FUNCTION cannot_be_overloaded RETURN NUMBER;
  FUNCTION cannot_be_overloaded RETURN VARCHAR2;
  --
END test_pkg;
/

CREATE OR REPLACE PACKAGE BODY test_pkg IS
  --
  FUNCTION cannot_be_overloaded RETURN NUMBER IS
  BEGIN
    RETURN 0;
  END cannot_be_overloaded;
  --
  FUNCTION cannot_be_overloaded RETURN VARCHAR2 IS
  BEGIN
    RETURN 'Character';
  END cannot_be_overloaded;
  --
END test_pkg;
/

Any calls to the functions would be ambiguous. Surely Oracle can't choose which one to use based on the type of the variable you're going to hold the value in... that would be a nightmare bit of compiler to implement. No no no!

Also, I reckon you'd have to take care with this:


CREATE OR REPLACE PACKAGE test_pkg IS
  --
  FUNCTION can_be_overloaded ( pn_number   NUMBER   ) RETURN NUMBER;
  FUNCTION can_be_overloaded ( pc_varchar  VARCHAR2 ) RETURN VARCHAR2;
  --
END test_pkg;
/

CREATE OR REPLACE PACKAGE BODY test_pkg IS
  --
  FUNCTION can_be_overloaded ( pn_number   NUMBER   ) RETURN NUMBER IS
  BEGIN
    RETURN pn_number;
  END can_be_overloaded;
  --
  FUNCTION can_be_overloaded ( pc_varchar  VARCHAR2 ) RETURN VARCHAR2 IS
  BEGIN
    RETURN pc_varchar;
  END can_be_overloaded;
  --
END test_pkg;
/

Even if the above would compile (I don't know if it would), then if you were to call the above with:


  test_pkg.can_be_overloaded ( '100' );

I suspect that Oracle will throw a wobbler. The parameter passed could be treated as a number or a varchar2, meaning either function could be valid for passing.
The only reason I suspect the package would compile is that the call could become non-ambiguous with:


  test_pkg.can_be_overloaded ( pc_varchar => '100' );

I can see how this could get very knotty, positional parameters could make all kinds of function calls ambiguous.
Reckon I might try some experiments tomorrow...

Version Control and the Patch Runner

2006-07-25T13:46:00.000+01:00

Commenting on my first post on upgrading databases, someone with the moniker 'gonen' asked a question:

"How do you handle the situation where one developer is checking in code that doesn't have to be released in current release?"

It's a good question, but it's a question mainly of version control rather than the patch runner itself. In fact, whilst the question was asked against the database patch runner post, it's interesting that the question doesn't mention the word 'patch', it uses the word 'code'. This is a general question that relates to any code, documentation, or other product of software development.

But, before I answer the question I'm going to turn it round a little and ask:

'Why do you have code that you want in version control that you don't want in the next release?'.

I can see three main answers to this one:

1 - The code is useless and so there's no reason to ever release it.

This is probably the most unlikely answer, but hey, it happens.
If this is the case, why are you keeping it? Commonly such code is kept because "it might be useful in the future and I don't want to throw it away".
Well, is it really likely to be useful? If it is historic code that is already is version control, then with most VC you can remove it and then still get it back at a later date if you need to. Proper version control software will record the fact that the file was deleted at a particular point in time but make sure the version exists in the historic versions. Otherwise you wouldn't be able to re-release only versions of your software that requires that code.
If it's new code that's not used anywhere, isn't yet in version control and isn't currently useful... why on earth would you want that in your system? It'll likely never get used, and merely hang around for years to come by which time people will be confused by existence but too scared to delete it.
The answer: If you don't need it, don't check it in. If it's already checked in, remove it. In either case, delete it, you don't need it.

2 - You're trying to fix a bug in the live system, but since you've made the original live release you've done a load more development. You're not ready to release the new stuff to live yet.

Or to put it another way. There was a live version of the system was released and since then Alex has been working on bug fixes. At the same time Ben's working on the new release and he new funky functionality. A few weeks in and Ben's been checking in his new stuff, but hasn't quite finished it. Alex gets given a critical bug to fix that must ship to the customer as soon as it's fixed. She can't commit and release the head of the trunk because it contains Ben's half finished stuff.

Ok. So when you made the live release you tagged up the release version in your version control didn't you. If not, you can always go back and do it, you just need to know the date you cut the release. And from now on you'll tag up everything that you release right?

The tagged version is the version you want to make your live bug fixes against, not the version at the head of the trunk. When you release a bug fix version you only want the bug fixes to be released, never the new functionality

To do this, you create a branch from the tag and do your bug fixing there. On that branch you now have your live version plus your bug fix. This branch doesn't contain any of the changes that you've made in the trunk since the live version was released. Of course, whenever you make a bug fix in the branch you make sure that the bug fixed is made in the trunk version as well, otherwise when your customer comes off the branch the bug will reappear.

3 - You have several people working on code at the same time, some on short term work, some on longer term stuff. Therefore have incomplete work you don't want to release yet.

For example, Alice has a long running bit of work and has been checking code into VC as she goes along. She hasn't really finished the job yet. Ben has been doing the same, but has managed to complete his. The trunk now contains a mixture of complete and incomplete work.

If you have several streams of concurrent long term development and often get to points where you need to release one stream without releasing another then branches can help again.
Rather than allow Alice and Ben to work directly on the trunk, give them a branch each. Known as 'task branches', each branch exists purely for the length of that task. Once the task is complete the developer merges their branch into the trunk and then discards the branch. When the next long term task arrives another branch is created and the work for that task done there

Going back to the example above:
Alice and Ben work on their own branches. As Ben finishes his work and commits into his branch we have the following situation: Ben's work is complete on his branch. Alice's incomplete work is on her branch. The trunk doesn't contain either Alice or Ben's work.
Now Ben's finished his work on the branch he merges it onto the trunk, effectively promoting it into the general release. Ben's task branch can now be discarded. The trunk can now be released to the customer.

Note that if Alice had finished her work first, then her work hits the trunk and the customer release can contain that bit of functionality instead of Ben's. It is not important which bit of work is finished first, which branch is discarded first. There is flexibility inherent in the system.

However, in this situation there may be another underlying problem. Ask yourself the question, would it be possible and be more efficient to have a single stream of work that gets completed quickly rather than multiple streams that each take time? That could engender a team attitude as well as focus the developers, testers, project managers, and business as a whole on a single task at a time. Yep, I know, that's not always possible...

So what about the patch runner then?

The patch runner can work very nicely in this branching structure, just as long as it's coded to deal with the fact that patches should only ever be installed once.

To quote myself, from that earlier post:

So, in summary, you check out a given tagged version of the application to run against an arbitrary database and run the patch runner. It loads the list of patches that are applicable for that tagged version. It runs over each patch in turn and checks if it has previously ran. If it has not, then it runs the patch.
By the time it reaches the end of the list it has ran all the patches, none of them twice. You can even run the patch runner twice in succession and guarantee that the second run will not change the state of the database.

Also, if the answer is in version control then the patch runner, the patch list and all the patches themselves need to be in version control. If they're not, then the fact that version control can answer your original question is itself a great reason for putting version control in place. Did you know that CVS and Subversion are both free... as is Tortoise (CVS and SVN). Download and install one of them, it'll take you all of 15 minutes to get a local version up and running and experimenting with.

Finally, if you want more information on version control then there are three books out there that you really should read. Well, you should read two out of three of them anyway...

Software Configuration Patterns - Steve Berczuk and Brad Appleton
Pragmatic Version Control using Subversion - Mike Mason
or
Pragmatic Version Control using CVS - Dave Thomas

And if you want more blog posts on version control then your man is Mike Mason... The go to guy on using Subversion in the real world.

Technorati Tags: Robert+Baillie, CVS, SVN, version+control, Oracle, upgrading, software, development, programming, database, DB, SQL, tortoise, patch

Merge triggers

2006-07-04T13:57:00.000+01:00

Had a quick puzzler at work today that I thought I'd share... and the link to Ask Tom that solved it (before I could be bothered to delve into the test case myself).

Q: Which statement level triggers fires on a MERGE statement?

A: Since it is possible for either inserts or updates to occur during the statement, and since statement level triggers always fire even when no rows are processed, both the INSERT and UPDATE triggers fire every time, regardless of whether any inserts or updates occur.

Obvious when you think about it. Trouble is, how many our YOUR statement level triggers would work properly if both sets fired at the same time? There's no reason why they shouldn't, if you code for the possibility....

Tom (and Mikito and Kevin and others) explains here.

World Cup Trivia

2006-06-29T13:55:00.000+01:00

Well, I'm finally back from my honeymoon, and with the wedding out of the way I may finally get some time to write blog entries again. But wait, hang on, it seems like there's a World Cup (footy that is) to take up almost all of my evenings. Ah well, no blogging just yet then!

Still, there's no matches on at the moment (not until Friday anyway) and so to quench my need for football I decided to go on a trivia hunt. And thus was born Honest Bob's World Cup Trivia facts... I can't guarantee the accuracy of any of these facts as all were researched on the internet ;-)

Of the 17 events (prior to Germany 2006), the hosts have won 6 times, with only Sweden being a losing host finalist (1958, losing to Brazil).

Argentina and Brazil are the only countries to win outside of their own continent. Brazil have managed to win on every continent the competition has been played:

Asia: South Korea / Japan (2002)

North America: USA (1994), Mexico (1970)

South America: Chile (1962)

Europe: Sweden (1958)

Not only that, but Brazil have managed to qualify for every world cup tournament.

However, they are the only team that have won the competition to have NOT won it as hosts.

Although a German team have won the competition 3 times, they've never managed it as a combined Germany, only ever as West Germany.

In total only 7 teams have won the competition out of 207 countries who have competed for qualification, and 78 who have made it to the world cup proper. The 7 winners are:

Brazil (5 times)

Italy (3 times)

West Germany (3 times)

Uruguay (2 times)

Argentina (2 times)

England (once)

France (once)

The competition has been utterly dominated by European and South American teams, with only two teams outside of the two continents having made it to the semi-final stages. USA (1994) and South Korea (2002) both being hosts when they managed it.
The fastest goal in a world cup match was scored by Hakan Sukur (Turkey), 11 seconds after kick off against South Korea in the 2002 tournament.

However, that time is beaten when you also take into account the qualifying matches. David Gualtieri (San Marino) scored after 8 seconds against England in their ill-fated qualification attempt for the 1994 Finals. England needed to win by 7 clear goals and have Holland lose their match against Poland. England only managed to win 7-1, though it mattered little as Holland eventually won their game 3-1.

In the whole of that qualification group (10 games) San Marino managed to score only one other goal and conceded 46.

The 1950 World Cup was not decided by a final. Rather it was a league contested by 4 teams, with the match between Uruguay and Brazil (2-1) being the decisive match and therefore generally regarded as 'the final'.

The only person to have played both World Cup Football and World Cup Cricket is Viv Richards - West Indies at cricket (obviously) and for Antigua in their 1974 World Cup Qualifying matches, which ultimately ended in failure.

Best Practice?

2006-05-25T12:32:00.000+01:00

Yeah yeah, it's been very quiet on the Bob front recently... I have an excuse though, it's only a week and a half to my wedding, so I've been a busy busy boy. Plus I've got the task of completely rewriting our development manual, so that's taking all of my creative juices. I'm running on empty.

But, when you get sent a link like this, you just have to share:

If you're anything like me, you'll absolutely hate the phrase 'Best Practice'. I cringe every time I hear it. It suggests to me that the speaker has stopped learning... "This is the best practice there could possibly be, so there's no point in trying to improve it". In addition it has the connotation that "This is the best practice for all situations, whatever it may be". Sorry, but that just doesn't work.

So anyway, a site that calls itself Fairly Good Practices was always going to pique my interest...

In my spare time I contribute to an OS CMS...

2006-04-24T19:55:00.000+01:00

Whenever I see a CV cross my desk containing the words 'Open Source Content Management System', I shiver. Visibly.

So what was I supposed to do with the news that my manager had decided to write one?

Well, it turns out that I needed to take a look at the home page, then take a look at then code, and finally decide that it's not actually that bad.

Obviously, the last thing the world needs is yet another Wiki, but as it's actually very very lightweight, looks ludicrously easy to set up, and is very well structured, then why not make space for just one more?

Obviously, he needs a slap for not writing unit tests, but the plug in authentication and storage classes are spot on.

In fact, I'd like to present it as an example of how PHP code can look when its not produced by an idiot. In addition, the CSS driven layout is a great example of content and presentation separation. All in all it's starting from a very nice position. Hopfeully it'll continue in the same way.

There's the odd little bit of refactoring to do, but once it's on sourceforge or freshmeat I'll help him with that...

Rest assured, it won't appear on my CV...

P.S. Momentous occasion: This my 100th blog entry
P.S.S Momentous occasion 2: It's also my boss's 30th birthday. Don't you hate it when your boss is younger than you!

Auto-increment in Oracle

2006-04-20T13:46:00.000+01:00

I'm sure that 100s of references exist all over the web for this, but someone asked me today how to do this, and so it's trivial for me to add it onto this blog...

In MySql you have an 'auto-increment' column. Does Oracle have one, and if not, how do you implement one in Oracle?

Well, we tend to say it's best to wrap up your INSERT statements in procedures and then manually grab the sequence number yourself... but if not:

You might notice the rather useful RETURNING clause in the insert statements. This bit seems to be missing from most internet examples I found...


CREATE SEQUENCE rob_tmp_seq
/

CREATE TABLE rob_tmp_tab ( id NUMBER, descr VARCHAR2(200) )
/

ALTER TABLE rob_tmp_tab ADD CONSTRAINT rob_tmp_tab_pk PRIMARY KEY ( id )
/

CREATE TRIGGER rob_tmp_tab_trig BEFORE INSERT ON rob_tmp_tab FOR EACH ROW
DECLARE
BEGIN
  --
  IF :new.id IS NULL THEN
    SELECT rob_tmp_seq.NEXTVAL
    INTO   :new.id
    FROM   DUAL;
  END IF;    
  --
END;
/

SET SERVEROUTPUT ON SIZE 1000000

DECLARE
  --
  vn_number   NUMBER;
  --
BEGIN
  --
  INSERT INTO rob_tmp_tab( descr )
  VALUES                 ( 'This is a description' )
  RETURNING id INTO vn_number;
  --
  DBMS_OUTPUT.PUT_LINE( 'Created a record with the automatically assigned ID: ' || vn_number );
  --
  INSERT INTO rob_tmp_tab( id, descr )
  VALUES                 ( rob_tmp_seq.NEXTVAL, 'This is a description' )
  RETURNING id INTO vn_number;
  --
  DBMS_OUTPUT.PUT_LINE( 'Created a record with the ID grabbed from sequence manually ID: ' || vn_number );
  --
  INSERT INTO rob_tmp_tab( id, descr )
  VALUES                 ( 150, 'This is a description' )
  RETURNING id INTO vn_number;
  --
  DBMS_OUTPUT.PUT_LINE( 'Created a record with the ID specified manually ID: ' || vn_number );
  --
END;
/

Measuring Performance

2006-04-16T07:28:00.000+01:00

A couple of months ago we started to get reports into our service desk that the larger of our databases were suffering from performance problems. Nothing catastrophic, just that the speed of the applications accessing those databases were starting to slow down.

As we'd added new versions of our system into the live environment fairly recently, the development team was asked to join into the discussion of how to solve the problem. This was a little unusual, as we don't normally get involved in the support of the live systems... be that firefighting or strategic support. And it wasn't that our applications were under suspicion, it was just that it made political sense to get us involved.

Rather than jump straight into the database, we took a holistic look at the performance of the applications, taking in as many components as we could: from client machines, through network infrastructure down to individual datafiles on the database server. In the end we came to the usual, almost inevitable conclusion... the database was the bottleneck, and it wasn't any single aspect of the database.

One thing we did notice however, was that the memory settings for the database server appeared to be unusual. So we advised to the DBA that these be looked at straight away. The DBA's suggested approach to this was
"Change the settings one at a time on live, if it doesn't break the system then we're probably on the right lines. Because it's dangerous doing this, we can only make one change per day and then wait for any calls on the service desk before we make the next change"

The rest of us didn't think that sounded like a safe way to go, so I suggested a new approach.

I put forward the idea of taking a production quality server that we could run in isolation, and importing the database owned by the brand that had the most urgent problems. We would then write an application that would stress that database in line with the behaviour of the live system, we would monitor the system's peroformance during the test and record our findings. I argued that once we had that in place we could produce a benchmark, make a change and rerun the test. This would then tell us how the system's performance characteristics had changed. We could then make the next proposed change, run the test again. Then make another change...

Having this tool in place would mean that we could thoroughly test the performance of the system in relation to our changes without ever having to rollout a change into a live environment. Once we were sure that we had made a step change in the performance of the system we could roll that change onto the live system reasonably safe in the knowledge that we would always have a positive impact on the system.

The rest of the team agreed and we set about producing our performance test rig.

Our requirements were:

The test should be completely repeatable. Meaning that, as far as is reasonably possible, the test should perform the same queries, at the same time, with the same parameters, every time the test runs.

When the test is running, the database being tested should appear to the DBA like the live version of the system under a particularly high level of stress.

The connections to the database should be made in the same style as the live system. E.g. emulations of connections from a web application should appear, make their requests and then disappear. Connections from a client machine application should appear at the start of the test and then remain connected for the duration of the test.

It should be easy to increase / decrease / change the characteristics of the test load.

It should be possible to run the test by entering a single command into a suitably set-up machine. It should then be possible to re-run the test by entering the same single command.

Running the test should include reporting on the performance characteristics.

I hope to go into the design of the solution in another blog entry soon...

We've since put the performance test rig together, and using it we managed to very closely replicate what was hapenning on live. Each test takes an hour to run and emulates a load equal to around double to the normal live load. We ran several test runs, each time changing a single memory setting until such a time as we had a decent change in performance.

We measured the performance differences by checking the average, minimum and maximum run times of each of our sets of emulation queries (we had a set for each of our applications, sometimes a set for each module in the application). The test rig also produces two stats pack snashots for each test, one at the start and another at the end. All the test results are then copied into a seperate database for long term storage and reporting.

Once we had the test rig up and running it took a day to tune the memory settings for our biggest database. We were able to run 5 tests in that day and end up with a very clear picture of the effects our changes would make on the live system.

The changes involved slashing the shared pool to less than a quarter its original size, taking the PGA target up by a factor of four and doubling the buffer cache. We then rolled the complete set of changes out onto the live database in one go.

It more than doubled the overall performance of the database. Exactly as predicted.

Since then the performance test rig has been earmarked for a few other jobs:

Balancing our read : write ratio on the local disk controllers.

Testing and tuning a Network Attached Storage device as a replacement for local disks.

Checking the impact of adding new indexes on heavily used areas of the system.

Investigation into possible designs for a new realtime "MIS light" application module.

Testing if regular defragmentation of our DBs give us an increase in performance.

Examination of how block sizes affect the performance of the system.

Basically, because the test rig properly emulates what happens on the live system, it can be used to investigate any change that would impact the performance of the system, be that hardware or software. And it allows us to make that investigation quickly and without any impact on any live system.

New Agile DB Blog on the block

2006-04-14T18:50:00.000+01:00

I've just spotted a few new links into this site, and (as always) I followed them back... to here. Warren Mayocchi's blog looks like its been around for a while, but the author's just on a new journey to solve the Agile DB release problem. He's definitely thinking along some very sound lines and I for one am very interested to find out where the journey takes him.

Good luck fellow pilgrim!

PS. Honestly, he's not getting a plug JUST because he linked to me ;-)
PSS. I like the 100 word fiction. Warren, just us some more!

Pretty Print

2006-04-14T18:36:00.000+01:00

In my current pilgrimage towards blog template beauty, I've now added CSS to format the page nicely for printing. Try the print preview and you'll see what I mean.

It turns out it's nice and easy... just add a media="screen" to your on screen CSS definition and add a media="print" to your paper one. Et Viola!

As with the javascript additions, all the formatting and suchlike that I've added has been added inline. That way you guys can take a gander if you want.

At some point soon the whole template is going to get a refactor to simplify it, maybe make it a bit more readable... just hang on in there for now!

Oh, and if anyone has any good idea for tracking sites, I'm getting a bit sick of going to three different places for my full set of tracking needs!

Brief question: When will Blogger.com's dictionary include the word blog?

It's a rhetorical question, AND I DEMAND AN ANSWER!

Attack of the Javascript

2006-04-12T14:33:00.000+01:00

Recently I've been wondering just what kind of lightweight javascript stuff can be added to the blog in order to make it "look kinda cool".

I think I've decided to do it because we spend all day here building systems that users actually want rather than just following our own flights of fancy and doing what we think might be cool.

Anyway, from now on you're likely to see little new additions to the site that I like. Let me know if you like it, or don't like it, or don't care.

I'll try to make sure things fail gracefully when they do... but I'm almost certainly going to mess up someone's web experience. If I do, then let me know and I'll fix it. It's the only way I'll learn, and the only way it'll get fixed.

Anyway, for now... have folding sidebar panels.

Woooohoooo

On Cloud Nine...

2006-04-06T22:09:00.000+01:00

I'm a Boro fan, I don't know how many times I've been asked what it is about football that gets me and so many people excited.

If you'd have been a Boro fan watching the Boro v Basel match tonight you'd completely understand why.

Every now and again you get to watch YOUR team play like they're playing for their lives. The whole of their existence being entirely for that one night. When they're playing like nothing will ever matter as much as this does in this moment. Every single one of them is on the same wavelength as every other player and they're plaing as if though they're a single being.

And you know that they're doing it all just for you.

Absolutely phenominal

Dixons in "good service" shocker

2006-04-05T07:34:00.000+01:00

I hate Dixons.

Every time I go in there I have to wait an absolute age for anyone to help me. They have the same staffing level problems that Debenhams and Maplins suffer from. They simply don't have any people on the shop floor.

However, when I went snowboarding last month I decided that I wanted to take a cheap video camera with me, one that I wouldn't mind breaking. And Dixons had exactly what I was looking for, an end of line Sony Handycam for 200 quid. Nice one.

As it was end of line I decided to haggle and managed to get 3 years of their Mastercare thrown into the deal. It covers accidental damage.

When we were out boarding I managed to stack it big time and break the flip out screen on the camera. Really annoying, because as soon as you flip out the screen the viewfinder switches off, and the menu buttons are under the screen. AAAaaargh.

Anyway, we soldiered on through the holiday with the broken camera (still recorded fine) and when we got back I phoned up their Mastercare phone line expecting a nice big argument about whether or not it's covered.

They just asked what the problem was, not once did they ask how it happened. They sent out a courier bag that I got about 2 days later. I put the camera in and called DHL to pick it up; they came the next day. That was Thursday. On Monday this week I got a DHL package back; I assumed that they'd messed up and just sent my camera to me instead of Mastercare. It WAS the camera, but it was fixed!

I couldn't believe it. I reckon it must have taken no more than a week from start to finish to get the camera fixed, the best bit being the TWO WORKING DAYS it took for them to get delivery of the camera, fix it and send it back.

Quite simply... WOW!

At 65 quid, the Mastercare service is a bit pricy, no matter how good it is... but it's well worth trying to get it thrown in for free!

Net Send from an Oracle DB

2006-04-01T11:18:00.000+01:00

Now I'm no DBA, but this (XUTL_NET) looks useful...

It's a small footprint product that adds windows style NET SEND capability to Oracle 8i and above, running on any operating system... just as long as you've got NETBIOS over TCP/IP on your network and Windows clients to receive the message.

It just seems like a neat little way of sending youself notification messages when those long running jobs finish, so you don't have to sit there with a window to your server open.

Chris also supplies an API for UDP (XUTL_UDP of course)... it seems he needed one to write XUTL_NET, and if you've written it, why not make it available?

UDP is available for free, as is a trial version of NET. I admit that I've not used either... but they both sound worth checking out if you need that kind of thing!

UK OUG Feedback and letting it slide

2006-03-25T11:35:00.000Z

I got back some feedback on my presentation from the UK OUG. I thought it was a nice touch. You get to hear what people thought of your presentation in terms of content / topic / presentation skills / quality of slides and the overal presentation.

All in all I was pretty pleased with the response, I was obviously a little disappointed as I can't handle being told that I'm anything other than exemplary at anything I do ;-)

Still, I couldn't be that disheartened as everyone seemed to agree that the presentation overall was well done, and of value. Wooo hoo. I'd agree that my presentation skills need a little bit of work... I was a little 'rabbit in headlights', and I'm likely to be like that for the next couple of presentations I do (assuming there are more to come).

However, I was very disappointed at the response on my slides. Apparently nearly 70% of people thought my slides were poor. Damn it!

I think this probably comes from a difference of opinion on the usefulness of slides as much as anything else.

I wasn't presenting anything technical, there were no difficult concepts to comprehend and there was no structures that needed describing visually. I was talking on what was a very human subject. A natural one. One I could happily discuss in the pub, without the need to resort to drawing on napkins.

I wasn't presenting as a representative of the company I work for. I develop in-house systems and we don't do consultancy. There was no need to advertise my firm to the audience.

With those two points in mind I figured that there was no great need for a lot of slides, or to make the ones I did use flashy.

I personally don't like a lot of slides with bullet point representations of the detailed discussion. I sometimes can't help but read the slide and then ignore the presenter. I also don't go for graphics, points sliding in from the side, fading in, fading out. They don't add anything to the discussion.

Anyway... I ended up with 5 slides, every one was black text on a grey background. They weren't sexy by any measure!

Each slide contained a single sentence that was a salient point of that part of the discussion. I wanted to convey the fact that when you walked away from the presentation I wanted you to walk away with these 5 points in your mind. Those 5 points were the main ingredients of the talk, the rest was just spice to fill out the flavour.
I didn't think I needed to sugar coat them, or spoon feed them, or shove them down your throat. I thought the words had enough impact on their own.

If that's what poor slides are... then I'm more than happy to get that feedback!

Tool envy and SQL Developer

2006-03-21T13:18:00.000Z

I've finally started evaluating SQL Developer as a general tool for our team. Like most people I've been speaking to, I'm getting a bit sick of TOAD's increasingly bloated footprint and array of meaningless icons. (And I liked Sue Harper 's presentation on it...)

Anyway, the early signs are reasonably good. As version 1.0 tools go this is definitely one of the most stable and feature rich applications I've used and it's got some really strong points.

Things I like:

The ability to name database connections

Autocompletion for object names

Pretty much everything about the package / procedure editor

The simplicity and general lack of clutter in the UI

Things I don't like:

The procedure editor doesn't yet save to files

There's no TOADlike session browser

For now, the lack of transparently saving packages and procedures as files is a bit of a block to us picking it up as a full blown IDE tool, but I'm sure this is coming and we'll be watching SQL Developer very closely for when this appears.

The session browser I can live without as there are other tools out there that'll do this job.

But the thing that's starting to get me REALLY excited is the prospect of Eclipse IDE integration...

So how far are Oracle likely to take this? Here's what I'm hoping:

In your Eclipse Project you get a new set of properties that allow you to assign database connections to your workspace.
Whenever you want to edit a package / procedure / etc it launches the SQL Developer procedure editor.
Saving one of the said files issues a save against the file system and a compile against the database.
The result is that I get a feature rich editor in Eclipse plus the fact that I can completely ignore my database when I'm writing my code. No more accidentally forgetting to install the latest version of the package before I test it.

Beyond that? I'm kinda hoping that Oracle take the lead from the Java community and start offering refactor tools. Imagine a 'Rename Package' option that drops your old version, renames the files, creates the new version, and then updates all references to that package in other packages / procedures / functions / etc / etc.

Similarly with a 'Rename Table'. Change the table name and then change every reference to that table that exists in the system. Generate me a script that does the rename table operation and I just need to put it into my patch runner.

Sorry, I'm drooling now.

Then we start to get a development environment for Oracle that properly supports you working in an agile manner.

Oh, and while we're at it, integration with UT-PLSQL would be nice too...

UK-OUG and running

2006-03-15T17:46:00.000Z

Well, that busiest of weeks is finally over, and I'm getting back into working life.
After a week of flying down powder runs in the French Alps, I managed to complete my first (and last) half marathon in a phenominal 2 hours, 17 minutes and 49 seconds.
If that wasn't enough, I then followed it up with my first public presentation, to UK OUG, as part of their combined SIG day.

It was a great experience, and I can't thank Andrew Clarke enough for inviting me to speak, as well as Penny for putting up with me chopping and changing the presentation at the last minute.
The full script for the presentation should make it on to the UK OUG site in the next day or two (pdf hopefully), but for those that can't wait (honestly... can you REALLY not wait), the full text is repeated below.

The highlight was most definately the demonstration of SQL Developer (AKA Raptor) by Sue Harper, a product manager who has most definately got her head screwed on right. It's good to hear Oracle making the right noises, and SQL Developer seems to be a clear tool developed in an environment of customer collaboration. Quest should really be watching their backs...

Also, Ivan Pellegrin's talk on service oriented architecture was quite enlightening, demystifying the principles of SOA, clearly cutting through the hype and revealing the important concepts... reusability, transparency and simplicity.

For those that wanted to catch me at the bar, please accept my most humble of apologies. I had somewhere to which I needed to run, and couldn't avoid. If you have any questions... any at all... please mail me and I'll happily open up some correspondance. If it's appropriate I'll post the responses on this site.

Also, Any feedback, even negative, is more than appreciated!

UK OUG presentation
March 14th 2006 – 12:45pm

Lessons from an Agile project
Introduction

For the last 18 months I've been team leading a project to replace a legacy Oracle and Powerbuilder system with an Oracle and PHP system. The requirements were that over the coming 3 years or so we would completely replace the existing functionality of the legacy system, and enhance the system where-ever the business thought it could gain competitive advantage.

We were told that we should expect to be able to release components of the new system with very short notice, basically whenever the business thought we had something it could use, that the direction of the project was likely to change frequently during its course and that at all times our new system would have to access the same data the legacy system does, that the data must be completely in step at all times.

Basically, the project is the same as every other project I've ever worked on. It's just that the organisation was honest about it up front.

To cut a long story short, we've had a pretty successful 18 months. In that time we've replaced about half the legacy system, and we've added a whole load of bells and whistles that the business feels has had a tangible effect on practices, turnover and, most importantly, profit.

We've had more good feedback from the 500 or so end users than I've ever heard from any other set of end users in my life, and I've got to say that all in the new system is more than liked by those that use it.

We get a constant feed of new ideas from the shop floor, and we have a dedicated customer team who are great at filtering those requests and combining them with the board of director's strategic view of the project. The result is a very clear short term development direction and a feeling that we know where we're generally headed.

We've made 4 releases of the system to the business now, averaging one every 4 or 5 months, and every time we've made those releases we've made them to 12 independent installations of the system. That is, in the last 18 months our system has been released into a production environment nearly 50 times.

We've never had a release fail.

We made a major release of the system about two weeks ago, and since it's been released the users have reported 2 bugs. Which we've fixed and released, which makes our current outstanding live bug count zero.

As we've been working on the system we've made a few demo releases. In fact, every time we finish a small piece of functionality we release it to the demo environment.

That equates to an average of one release every two days... a total of about 300.

Often, the customer team will come up with an idea for a new piece of functionality, and if they think it's important enough, it can be specced, coded, tested and released into the demo environment within 3 or 4 days.

Basically, we're working in line with the principles of Extreme Programming, and I can honestly say that I've never worked on a project that has been any where near as successful as this.

Now, I firmly believe that internal system development is perfectly suited to Extreme Programming and vice versa, and I could very easily spend all day up here talking about how XP works and the advantages it gives you. I can happily evangelise about the individual principles and practices that XP embodies and try to convince you that XP is the way forward.

But XP isn't for everyone. And XP probably isn't universal.

Instead I'd like to talk about what working on an XP project has taught me in a more general sense.

I'd like to talk about some core practices that we work with, that are complimentary to XP, and that I think are fundamental foundations upon which you can build a development environment.

That is, Extreme Programming encourages you to think about why the process and your practices are the way they are, to question their worth and in turn improve them.

Working on an XP project has taught me to take a new look at the practices I'd taken for granted. To examine the problems I thought were essential, rather than accidental, re-evaluate them, and hopefully solve them.

We've come up with some good practices that I think are are good whether you choose to pair program and unit test or not. That these are practices from which any development process will benefit, and for which I believe there are NO excuses for not following. Certainly none that I've heard.

These are things that are complimentary to any development process and that I think are probably unspoken assumptions of processes like XP.
These are things that I am worried may be 'no brainers' in the Java, .NET and Ruby, communities, but that we, as Oracle developers and DBAs need to learn.

My biggest hope is that as I'm describing these lessons, many or most of you will just think 'well yeah, of course, jeez, didn't you already know that', that many of you will go away in a 40 minutes time thinking, 'I learnt nothing from that, he just said a lot of stuff that's obvious'.

I hope that it's just me that's taken this long to learn these lessons.

I hope that my own experiences of Oracle development prior to this project are not the same as everyone else's.

I hope that I was just unlucky.

Understand version control

On the face of it, version control tools are simple beasts. Technically, they do very few things:

• Record changes that have occurred to files - Commit.
• Allow you to extract the files as they appeared at any particular point in time - Checkout.
• Label particular points in time to be of interest - Tag.
• Manage multiple changes made to single files, at the same time, by multiple people – Lock or Merge.
• Segregate sets of changes from each other - Branch.
• Perform actions when commits occur - Trigger.
• Group changes together in a set and put comments against it.

All in all, the tools provide nothing that a manual process couldn't.
In fact, I've worked at a place that implemented a pessimistic locking version control system purely through manual processes and a directory structure on a network drive. We'd copy files, change their extensions to lock them, change their extensions to release them, copy folders to tag, copy them again to branch.

The result was a lot of time spent following the manual process, and fixing problems that were caused by people not following them. Every developer hated it. It sucked time out of your day and it was a dull, repetitive process that you could never get quite right.

But that wasn't because there was anything wrong with the processes they were trying to implement.
The problem was solely with the methods by which the processes were carried out.

That is, there are two parts to version control:
• Underlying processes that allow teams to manage changes to files over a period of time.
• Tools that help to make those processes easy to follow.

The process without the tool is laborious and cumbersome.
The tool without the process is, well, useless.

So often you see organisations that pick up the tool, but don't understand the principles behind it, or who understand the processes but don't get how the tool can help you perform them easily.

My old friends with the manual processes DID eventually pick up CVS, after I'd left, but then completely failed to marry up their old processes with the new tool and ended up alienating the developers in a different way. It was a bit of a shame, especially considering that the original processes were actually built on sound principles.

In order to really take advantage of version control you need to understand both those aspects. How the processes work, and how the tools can help you perform those processes so that the development team don't see them as a barrier to getting the job done.

In order to understand how to put a version control process together that works we need to understand why we need version control, and what it does for us.

For me, the most important aspect of version control is that when it's used properly, it becomes an absolute truth.

In order for the developers to work, they need an absolute truth that they can rely on. A place from which any piece of development can start. A good version control repository provides that. It allows a developer to take any source code file they need and say with a great deal of certainty,

"This is the file against which I must develop. This specification
relates to this set of files, and this is the truth from which I must
work".

In teams where the version control repository either doesn't exist, or is an unreliable source of the truth, developers will find an alternative truth. Most often this is either the live system (which is a particular, narrow truth), or their own world (which is almost always a lie).

So in order to make version control work for us, we need to make sure that it is reliable.

What do I mean by reliable?

First of all, we need to make sure that the team agree that from our version control we can build the live systems.

Regularly ask of your version control:

"If the live version of our system was to suddenly disappear,
along with all our backups, can I completely rebuild the structure
of our system from the version control repositories, and be
certain it's correct?"

If the answer is no then your repository is not good enough. Its content is unreliable in the sense that either:
• It does not hold all the information that it needs to, or
• The information it holds is inaccurate.

There is reliability of completeness.

And

There is reliability of accuracy.

Secondly, we need to make sure that the software in the repository is of a standard that all parties agree to.

It may be that all code in the repository must always be in a ready to release state and that the build never breaks, it may be that the code must be regarded as broken except during short periods of well defined time.

Whatever the decision is for the level of quality of code, it must be known throughout the team what that level is.

If not, a developer will avoid updating their workspace to the latest version of the code... they'd never know if doing so would create a whole swathe of bugs to appear.

There is reliability of quality.

Finally, we need to make sure that whenever a developer wants a particular version of a particular piece of code that they can reach it, easily.

Also, that whenever they need to check code into the repository that there is a way of them to do so, and that doing so takes a trivial amount of time.

The version control system and its standards should not stand in the way of the developers from reaching their goals, it should aid them. Its crucial that whatever standards and tools are put in place that they match the working practices of the team involved.

There is reliability of usefulness.

Without those things, the version control processes will never be perceived as anything more than a pointless chore that gets in the way. And when people think it gets in the way, there will always be people who will find ways around it. As soon as one developer works round it, it ceases to be the truth. As soon as the repository is no longer the truth, the whole idea of version control collapses.

The good news is that ensuring that the aspects of reliability are reached is really just about recognising that they're there and they're attainable. It's really just about understanding the relationship between the process and the tool and the impact their use has.

It's about developing a process that matches your aims and your team and then finding a tool that matches the process.

Thankfully there's plenty of good thinking out there that can help us put together appropriate processes, and there's one book in particular that really stands out:

Software Configuration Management Patterns by Steve Berczuk and Brad Appleton.

It's brilliant, and a must read for anyone that is looking to put version control processes in place, or wants to understand how they can improve the processes they have. Quite simply – if you haven't already, you should read it.

All in all, I can't understate how important I think the implementation of a good version control policy and tool is. I found that understanding how version control works on a conceptual level led us to put together a set of processes that really don't get in the way of developing code but ensure that the repository is a reliable truth.

And doing so led me very quickly onto the next lesson. Something that I didn't realise was possible until I'd seen the capabilities of a version control tool used inside a clear and robust process...

Implement a build script that an idiot could run

The most dangerous part of any database development is the bit where you roll it out onto a live server.

Because it's risky, many organisations think that they need highly skilled database developers or DBAs to perform the upgrade. That way, if anything goes wrong then you're guaranteed to have someone there to fix it.

It doesn't sound like a bad idea.

But because these people are highly skilled and able to keep track of the things that need doing during the upgrade, then you find that the upgrade becomes a series of disparate tasks. A long mundane series of steps. If you're really lucky then they may have a check-list beside them as they're doing it, but often they won't.

The problem is that when you have a developer or DBA producing and performing the upgrade, you end up with a build script that a developer or DBA can run. And that means it's a build script that needs a developer or DBA for it to be run, and that means its basically time consuming and difficult to run.

When anything is time consuming and difficult to do, then you do anything you can to avoid doing it. You do it as few times as you possibly can, and that leads to problems...

I've worked for far too many people who will run an upgrade on a live system that they have never ran anywhere else. That means that the most dangerous part of the system life cycle is effectively done without any testing.

Not only that, but I've worked for software houses that will do so and then have the cheek to charge the customer extra for support during the upgrade.

Looking back I can't understand how I accepted, and how everyone around me accepted that when you decide you're going to upgrade a database you need to spend a long time putting together a collection of scripts and then run them in an informal manner.

I've since interviewed a lot of developers and every single one has more or less said the same thing:
Upgrades are time consuming, untested, informal processes.

Now that sounds like a bad idea.

So, I've got an alternative.

Assume that you won't have highly skilled people running your builds on live systems.

Assume that the person performing the upgrade is an idiot.

If you've got an idiot performing the upgrade then in order to make the upgrade safe you need to make the build scripts safe. You can no longer rely on highly skilled DBA to nurse it through.

Basically, the build script has to be of as high a quality as any other piece of code in the system.

The build script needs to work first time, every time, and needs to do so with the minimum of input from the user. It needs to tell the person who's running it that it's working, that every component is doing what it should be doing, and as soon as it stops working it needs to stop and flash red and tell the idiot that the world's going to end... and then maybe stop and roll-back to a time when things were still working.

The code that forms the build need to be covered by proper testing strategies, fall under version control, and be held in a structure that everyone on the team understands and can support. As well as being trivial to run the build, it needs to be trivial to add to the build. You want all of your developers to be able to actively change the build themselves, not ask someone else to do it for
them. You want every developer to take personal responsibility for the quality of the build scripts in the same way that they take responsibility for the system.

When the developers have to ask a third party to produce the scripts for them there is the possiblity of misinterpretation. If the developers add the scripts themselves there is clarity and no risk of ambiguity.

I argue that if you need to, then you can put together a build script that an idiot can run. That doing so doesn't actually require much more work to do than producing a build script a DBA could run. And that doing so makes the build script easier to run, safer to release to live and far quicker to produce.

When you've got that it means that you have a build script that anyone can run, and can run easily.

We've got our build scripts to such a point that we can roll out any version of our system to any earlier version of our system by switching off the web server, running a single command with 3 parameters, waiting for the script to finish, and then switching the web server back on again.

If we need to create a brand new replica of the system, we take a configured web server and a fresh Oracle instance, and run the same script.

We use the same script to roll out to the demo environment as we do for the live environment and every developer working on the project will roll out the system to the demo environment every few days... that is, when they finish each piece of work.

This means that in a 3 month release cycle the upgrade scripts will have been ran, on average, 20 times by every developer on the team.

Having the developers run the scripts that often really helps in getting the build scripts to run smoothly, and with the minimum of effort. Ask a developer to do the same task every two days and if they're worth employing they'll do everything they can to make sure that task is trivial to perform.

We've done this by having a very clear structure for our build scripts. We've put a structure together that consciously removes the need for a build manager. That is, at every previous place I've worked there has been a person employed who's responsibility it is to construct a large chunk of the build script.

When a customer wanted a release, or a version was to be sealed, the build manager would go through the changes made between version x and version y. In some places this would mean checking through their e-mails, in others it would mean checking differences in the version control repository, in yet others it would be checking the differences between reference databases.

In all cases, that person's job could have been replaced by a script. It's just that no-one had the imagination to realise it.

I'm sure that there are many ways in which the solution can be implemented, but for me they are all covered by this process:

• Divide your database changes into:
• Things that can be ran many times without causing damage.
• Things that can only be ran once.
• Make sure that you have a structure that ensures that the things that can only be run once, only run once, and all the rest run every time.

When it comes to making a build:
• Find out which things need running
• Run them, in the right order
• Stop when you get an error
For our Oracle build, this basically translates to a build script that incorporates 2 parts:

First we have a patch runner. A generic script runner that will take an ordered list of scripts and ensure that every one of those scripts runs against the database once, and no more than once. If any of the scripts cause an exception then the whole thing stops.

These scripts basically consist of table and data changes: the only components of an Oracle build script that we need to ensure only run once.

The patches and the patch list are held in version control, along with the rest of the source code, so if we want to take a database up to version 2.4.5, we just check out version 2.4.5 and tell the patch runner to run. The patch list only contains those patches that were part of 2.4.5, and the patch runner makes sure it only installs the patches that have not already been run.

Is it happens, we add another level of complexity on top of that. For every patch we have an additional 3 scripts. We have two that check the pre and post conditions for a patch. The pre conditions embody the assumptions that are made about the state of the database before the patch can be ran. The pre conditions are tested, and if they fail the patch doesn't run. That way, instead of picking apart data that has been mangled by a patch that should not have ran, we can look at a database to determine why the conditions required for the patch weren't met. A much easier proposition. When the patch is completed the post conditions are checked. These state the success
conditions of the patch and ensure that when a patch is ran it does what was intended. Again, if the conditions aren't met then the upgrade stops.

The third script is the reversal script. This can be used in a disaster recovery situation, when a patch has completed but must be removed. All four scripts are produced by a single developer... the one that requires the database to change.

Second in the build, we have a script that re-installs absolutely everything else.
Every view, trigger, package, procedure, function in the system is re-created, every time. You don't need someone to work out which packages have changed and ensure they get added to the build script if you always install everything. And what harm is done by always installing everything? It might take extra time to run the script, but the advantage is that the only thing the build script cares about is the target
version... not the source version.

Of course, the detail of the build script is more complicated than that, but that's the basic structure described. And with that structure we can take any version of the system and upgrade it to any other version. We can take an empty database and build the system in less than 5 minutes.

And we use that fact to give ourselves the best present a developer can have...
A Sandbox.

Make yourself a sandbox to play in

Every decent developer I've worked with understands that you don't just dive in and make changes on a live server, and they all buy into the basic idea of version control for files.

People instinctively get the idea:

"If I'm working on this particular file then I want to be sure that I
am the only person updating this file.
When I save it and close it I need to know that when I open it
again it'll be the same file.
I need to know that I'm working in isolation."

When you're developing, you need to have your own version of the source code you're changing, and have complete control over it.

Version control tools like CVS and Subversion take this a step further and give you a private copy of the whole application. That way you can change any source code file and know that the changes you've made are private to you.

Sometimes this idea is called 'the workspace', other times it's 'a sandbox'. Both are basically the same thing, but I like the metaphor of the sandbox... it's simple:

I'm in my sandbox.
I'm safe from the rest of the world and the rest of the world is safe from me.
I can play.

The idea is phenomenally successful when applied to source code. Probably because it's simply an extension of what any decent developer would instinctively do. That's why I'm so surprised that I so rarely see the idea extended to databases.

I've worked at a few places where I got my own set of source code to play with, but invariably I had to install it into a database that was shared between the development team.

Sometimes that DB would be specific to a branch on the development tree, other times it would just be a single conglomeration of all the different versions in an unstructured mass.

It's like giving yourself a sandbox... and then putting it too close to the swings.

You can be happily playing away, safe in the knowledge that you're completely unaffected by what other people are doing and BANG, another developer:

• Updates the database, wiping out your test data
• Overwrites your piece of code
• Or even worse, introduces a subtle bug into some code that you use but haven't changed... you spend hours nursing your bad head, trying to track it down, and then find that the bug mysteriously disappears again.

You think you're in isolation, but you're not.

Once you get used to the fact that your world can change around you without warning, then you get more and more likely to assume that the bugs you see in your development are not your fault.

The natural reaction is to blame the weakest link, and you'd never dream of accepting that you might be that link. Working in real isolation means that there's no ambiguity... you are the only link.

Sandboxes have to be complete, or they're use is seriously compromised, and they're only complete if you have your own database.

If you have your own database you can treat it fully as your own, and know that any changes that occur to it occur because you made them.

When you need to integrate with development trunk you follow the simple pattern.. update your sandbox files, run the database upgrade and test.

When you update your sandbox you're updating both the application source code and the build script, and since an idiot can run your build script then you can upgrade your database with no fuss.

Not only are you using your sandbox to separate yourself from the rest of world, but you're also using it test your build script. If you have 10 developers working in 10 sandboxes then you immediately have 10 build testers as well.

Traditionally, as the number of developers increases, then the fragility of the build also increases: since the build is changing more frequently it's more likely to break. But if every one of those developers is testing the build as they go along, then you can maintain the quality.

Obviously, this kind of informal testing is no replacement for full blown testing, but any testing increases quality.

So my argument is that I've if you've got 10 developers (or 10 pairs if you're doing it the XP way), then you need 10 databases...

Now I can hear DBAs screaming: for most DBA's I've probably at least doubled the number of databases they manage. But it's not as bad as all that once you take a look at what a development database should actually be.

The worry comes from the thought that a development database is the same species of beast as the live database. It isn't, not by a long way.

The major difference between a live and development database is how their role defines the use of data. In a live database the single most important thing in that database is the data it holds. It is the sole purpose for the existence of that database. The live database without the live data is useless.

In the development environment this is not the case; the most important thing in a development database is its structure.

Its purpose is to work as a validation tool for the live database, to ensure that the application we're producing works against the live database and to allow us to develop the structure of that database in order to allow to us to develop the applications.

It's important that we hold data that has a texture similar to that which exists in live, but it doesn't matter which data we hold. The data itself has no intrinsic value. We can generate our own.

In fact, when we run unit tests on our PL/SQL code, this is exactly what we do; we generate our test data so we know exactly what form it is in. That way we can make clear statements about the form we expect it to be in when our PL/SQL has completed.

We then use this generated test data as the normal data for our development. By clearing down and recreating test data regularly we are sure that any bugs we see in our development environment are caused by our code, not by some out of date and twisted set of data that was created by someone else half way though some development.

Also, we don't need to hold the same volume of data. In live, every single piece of data is of importance and so you can't just remove it. In development we can limit ourselves to create only the data that is of importance to us at this point in time.

If our development databases contain a very small amount of data, and the means to re-generate the data it needs, then you remove a large amount of the DBA's work.

That is, most the work a DBA will do in maintaining a live database is purely down to the fact that it is a system that contains a large volume of critical data. It must always be available, it must serve data within certain performance requirements, and if the machine it's running on suddenly blows up then we must be able to recover the data. Quickly.

Take away all that away and you've taken away most of the DBA's work in maintaining that database.

So the approach we've taken is to get the DBA to create a single development database instance, and then within that we've created several independent schemas. Each schema is owned by a particular developer and they're responsible for the state of that schema.

Each schema contains a mirror of the structure of the live system, with any changes that the latest bits of development have created. Each schema is a completely self contained working version of the system.

OK, so the structure of the development workspace is slightly different to that of the live system; the live system has only the one application schema on each database instance, but we hide all that in the build script. The build script itself understands what type of database it's installing and changes its behaviour when we're upgrading a development database. As it happens, the only real change is replacing public synonyms with private synonyms for some objects, and then installing
the test suite.

I admit that it took a bit of work to get the structures right, and in making sure that differences between live and development were as small as possible but now that we have that structure in place we can create a new sandbox for a new developer in about 5 minutes... the amount of time it takes to run our build script. And every developer in the team can go into a sandbox and play, safe in the knowledge that NOTHING they do in that sandbox will affect anyone else.

I don't think there's a single developer in the team that would give it up.

Conclusion

Obviously, in the last 18 months there's been a lot of other lessons that I'd take onto another project, but the combination of good version control policy, a simple build script and individual developer sandboxes have had a fundamental impact on our day-to-day life.

The implementation of those foundations has allowed our developers to focus on the job at hand... the development of new a system. And it gives our DBA the confidence that the installation of a new version of the system won't cause him a weekend of headaches and pain. Rollouts of the system aren't trivial, and never will be. There are always going to be other project management issues to deal with... first line support and user training, for example. But we've got to a point where the actual upgrade of the database is not an issue.

The development team love the pair programming and unit testing aspects of XP, but without the sandboxes we'd still have a team working with the usual frustrations caused by not working in isolation. Morale definitely improves when everyone feels as though they're in charge of their own space.

And the solid version control policy allows us to manage the whole process of concurrent development without ever really thinking about it. We support live systems and develop new functionality side by side with very little pain.
All in all, these three things just make our jobs a hell of a lot easier. And let us focus on what we do best, and ultimately, what we're paid for. Developing new systems... and drinking coffee.

Technorati Tags: Oracle, development, software, XP, version+control, sandbox, workspace, UKOUG, OUG, Andrew+Clarke, Robert+Baillie, Sue+Harper, SQL+Developer

Busy times and UK OUG

2006-02-22T07:21:00.000Z

Once again I've been going through a quiet time; that the problem with something that's habit forming, sometimes you can fall out of the habit.

There are three main reasons for not posting recently, and all translate to me being very busy...

1) I'm getting married: And along with getting married comes organising a wedding. I wish someone would have told me that before I agreed!

2) I'm training for a half marathon: After managing my first 10km about 2 years ago I've been thinking about trying one... and a family member challenged me to one in my family's home town, so I thought 'what the hey'.

3) I'm presenting for the first time at the UK OUG in Slough on March 14th.

When Andrew Clarke surprised me by inviting me to participate, I couldn't really refuse, especially since he shares his name with my brother-in-law. So I'll be talking about the things that I've learnt about Oracle development over the last 18 months, a time I've spent on an Extreme Programming driven project. It'll be covering things that are outside of the core XP principles. Rather it'll be those things that I would now say were essential in a project no matter what methodology was being followed.
That doesn't mean to say that I don't think unit testing, pair programming, short iterations and close customer involvement aren't essential... it's more that I don't think I can add more to the discussion by focusing on other things.

Anyway, for those who read this blog, the topics will probably be quite familiar. Here's hoping it also makes sense.

I look forward to seeing some of you there!

Developing an environment

2006-01-21T16:08:00.000Z

Previously I've talked about how important I think that development databases are, and that I think that they should be proper sandboxes, protect from the outside world and each other. I admit, I've kind of glossed over how that might be done, and so I wanted to correct that...

In our current project we're building an application that will eventually replace an existing one. Until it does so we need to ensure that our new application can live together with the legacy one, and will share the same data. The two systems must exist symbiotically.

The existing application, let's call it Legacy is a classic Forms or Powerbuilder and Oracle structure. All it's objects exist in the Oracle user LEGACY, and are wrapped in public synonyms.
Each users of the system is given an Oracle user that then accesses the LEGACY objects through the public synonyms. Let's say have the users USER1, USER2 and USER3.
So we have a structure that looks like this:

The new application is to live within the same database. We'll call that application NewApp, and install it into the new Oracle user NEWAPP. As an Intranet delivered system (over HTTP) we decided that all user connections to the Oracle database would occur through the user NEWAPP.

We decided that in order to insulate ourselves from Legacy, we would ensure that all access to the LEGACY objects would be done through a new set of private synonyms. As an aside, this allowed us to develop a kind of 'name space'. I.E. The private synonyms for the LEGACY objects are all prefixed with LE_. That way we can tell which are LEGACY objects and which are NEWAPP and reduce the chances of clashing object names.

Anyway, that then gives us the following structure in our live environment:

We then state that we do not add new functionality into the LEGACY user. I.E. All development for NewApp is done in the NEWAPP user. We are not the owners of LEGACY and therefore do not have the ability to change LEGACY at will. We are the owners of NEWAPP and therefore the contents of NEWAPP are entirely under our control.

So how do we deal with this structure in our workspaces?

One option is to build a full database per developer / pair. Starting with an empty database we could either import a copy of LEGACY or build one from scripts. Then by running our NewApp build script we can build the NEWAPP user as it would exist in the live environment.

This may be appropriate for a small team of highly Oracle skilled developers, but I'm not too sure.
With the large number of databases you then have an overhead in terms of maintaining those databases. You either need a development DBA on hand to troubleshoot problems on each database, or you need each developer to be capable of dealing with those problems such as the listener not starting up, upgrading the DB to 9.2.0.7 to match the live environment, etc. In addition, if each database is on a client machine, you run the risk of each database being configured in a slightly different way... being fiddled with by developers that don't necessarily understand the impact of settings being different in development as they are in live.

In short, I'm not too sure. I know it CAN work, and that it's far superior to not having individual workspaces... but we followed a different route.

We decided we wanted a single development database, in which many workspaces could exist. However, in order to have individual workspaces, we don't just need individual NEWAPP users, we also need individual LEGACY users. OK, so we may not be changing the structure or functionality of the LEGACY user, but much of our data exists in that user. If we're to be insulated from the changes made in other workspaces, then we need to be insulated from data changes as well as structural ones.

So how do we implement the individual workspaces in a single database?

The answer is actually surprisingly simple; we have a dual mode build script, to which we can state that we're building a development environment.

If we're building into live / demo / any other non development environment then we build our private synonyms and issue our grants to LEGACY objects, then build / upgrade the NEWAPP user.

If we're building into a development environment then we start off by creating our development version of the LEGACY user (lets call it DEVLEG1), then create our private synonyms and grants to DEVLEG1 instead of LEGACY.

So if we were to build the development environments DEVNEW1/DEVLEG1, DEVNEW2/DEVLEG2 and DEVNEW3/DEVLEG3 then we'd have the following structures:

If we then found that we needed to be able run the front end for Legacy in our development environments then we could add the creation of relevant users along with private synonyms to the appropriate DEVLEGx user, giving us the following structures:

The result is that we have fully independant development workspaces whilst allowing the database to be centrally managed. We can use our DBA team to keep the databases up and running and control the set-up whilst allowing us to develop the code freely without treading on each other's toes.
The eagle eyed of you may notice the fact that our development databases may be a little deviod of data, not a great scenario. However, we have noticed that ourselves and put a solution in place. More on that later...

Technorati Tags: Oracle, software, development, database, upgrade, blog, agile, Robert+Baillie, patch, db, dbms, workspaces, sandboxes

We need space to develop

2006-01-15T17:56:00.000Z

I've talked previously about how important I think it is that individual developers (or pairs) get to work in their own sandbox (not least here). This means that developers can experiment; can try avenues that would otherwise break the workspace for other developers; can refactor the database itself.

Obviously, in order to do this, you need to actually have development databases for those developers to own. This tends to lead to the DBA team announcing that you'd need far too many databases, that you simply can't be serious, that we've no idea how overworked they already are and that we can't expect them to take on another x databases, where x equals the number of developers. Oh, and by the way... who's going to build them? I can fit the first 3 in some time the middle of the year after next, god knows when the other 7 will get built.

I can understand the reaction. I'd be exactly the same if I thought that someone was suggesting that I doubled my workload. Luckily for DBAs, that's most definitely not what I'm suggesting.

The misunderstanding comes from the thought that a development database is the same species of beast as the live database. It isn't, not by a long way.

The major difference between a live and development database is how their role defines the use of data. In a live database the single most important thing in that database is the data it holds. It is the sole purpose for the existence of that database. Fine, we reshape that database so that we can access the database in new and interesting ways, but all said and done, if that data wasn't there then the database would be useless. In the development environment this is not the case.

The most important thing in a development database is its structure. Its purpose is to work as a validation tool for the live database, to ensure that the application we're producing works against the live database and to allow us to develop the structure of that database in order to allow to us to provide our application. It's important that we hold data that has a texture similar to that which exists in live, but it doesn't matter which data we hold. The data itself has no intrinsic value. We can generate our own.
In fact, when we run unit tests on our PL/SQL code, this is exactly what we do; we generate our test data so we know exactly what form it is in. That way we can make clear statements about the form we expect it to be in when our PL/SQL has completed.

Also, we don't need to hold the same volume of data. In live, every single piece of data is of importance and so you can't just remove it*. In development we can limit ourselves to create only the data that is of importance to us at this point in time. If our development databases contain a very small amount of data, and the means to generate the data it needs, then you remove a large amount of the DBA's work.

That is, most the work a DBA will do in maintaining a live database is purely down to the fact that it is a system that contains a large volume of critical data. It must always be available, it must serve data within certain performance requirements, and if the machine it's running on suddenly blows up then we must be able to recover the data. Quickly.
Take away everything other than the 'it must always be available', and you've taken away most of the DBA's work.

When we started development, we didn't tell our DBA that we needed 'x' new development databases. We told our DBA that we needed 1. We said we didn't need it backed up, just that we needed it to be up during working hours. We said, make it about the same size as live and it'll always be big enough.
We then ran our build scripts multiple times on that database, creating a schema for each of our pairs of developers. This is the same build script we would use to upgrade live, it just has a development mode that allows us to say which schema name we want it to install into.
The DBA has needed to do next to nothing with the database since it was created.

Now I admit, we have only around 10 development databases at the moment. Maybe the problem gets a lot harder to manage when you have 100 development databases. But then again, if you have 100 developers then I'm damn sure you've got more than 1 DBA. You've got a DBA team. And if you've got a team then surely someone in that team's got the time to build you enough databases to support your development team. If not, then you've probably got more serious problems to deal with...

Of course, none of this takes into account the fact that your customer needs to test the application, that you'll need to performance tune and test along side all the other systems running before you release and you need a test environment for duplicating issues once the system goes live. But NONE of these things should be happening in your development environment... a topic for another post. Later.

* Actually, you often find that most the data in a live database is useless and only a small proportion is ever used. But try telling that to your customer and getting them to let you archive it...

Technorati Tags: Oracle, software, development, database, upgrade, blog, agile, Robert+Baillie, patch,db, dbms, workspaces, sandboxes

Database Upgrade Links

2006-01-10T13:46:00.000Z

Around August last year I started to put together some notes on writing database upgrade scripts. It was kicked off after a discussion with Wilfred van der Deijl, based on experiences on my current project where I honestly we've got it about as right as it ever needs to be.

For a while now I've been fiddling with FeedDigest to try to get it to categorise my entries so that I can produce a summary page, and I've finally given up. So instead, I've put this entry together. It's just a link page to each of my database upgrade entries... I hope it's of use!

Introduction

Rollbacks

Why all the effort?

Dealing with code

Table Centric Views

Design By Contract

Google Pack

2006-01-08T20:32:00.000Z

It's been a while since I've been 'in the blogging scene', and I suppose the only way to get back is to start posting again! So, whilst this post may be small, I'm hoping it's beautiful.

Google have decided to address that age old problem... getting your new PC from the point it's at when you first buy it to actually being useful and on the way to being safe to use on the web.

They've put together the 'Google Pack'.

OK, so I like Google Earth, but I'm not sure it's quite essential, but the rest:

Google Desktop
Picasa
Google Toolbar for IE
Norton Antivirus
Ad-Aware
Adobe Reader

And THE most important one:
Mozilla Firefox

It's a stunning idea, though I'd probably add to that:
Open Office
AVG (Ok, so you've already got Norton, but AVG is ACTUALLY free)
Skype
Hitman Pro (Ad-aware is only one part of the anti spyware suite)

Valued estimation

2005-12-07T17:53:00.000Z

In "User Stories Applied", Mike Coen puts forward a method for estimating the cost of developing stories (or any other piece of work).

In summary:
* The development team get together with the customer.
* The customer describes a story.
* The developers quiz the customer until they are happy they understand the story.
* Each developer that wants to participate writes an estimate down.
* All developers show their estimates.
* The developer with the highest estimate states why they think it's so big.
* The developer with the lowest estimate states why they thing it's so small.
* The developers discuss the problem further until they're all happy they understand what's involved, and they estimate again.

The loop keeps going round until the team all have the same estimate, or they're happy to accept a compromise on one.

Once the stories are all estimated, they're put in a line of ascending cost and examined to make sure there are no discrepancies. If there are, then that story is re-stimated.

It works a treat, and the customers like it.

In fact they like it so much that they've come up with a variation...

Excluding the project manager, our customer team consists of two full time members and a number of people that work on our project only part-time. One slight issue is that over the last 18 months the full time members are becoming more a part of the Information Systems team and less in touch with 'The Shop Floor'. The part-time members help to ground them in the reality of the system.

On the other hand, the part time members aren't available to instantly evaluate new stories, and prioritise them accordingly.

So, once a week the customer team now get together in a 'Value and Take-up estimation'. It works in the same way as the developer estimates except the customer team are evaluating the value that a given story will add to the system, and then the take-up level expected for that part of the system in a scale of one to ten,

Obviously, the priority of a story is based on the combination of all three variables, Value / Take-up and Cost. Still, as a general rule, a story that scores high on Value and high on Take-up is going to be a story that will be prioritised very highly unless the cost is extortionate!

In fact the particular values may not be that important; there is no point discussing if a Value 8, Take-up 7 story is more important that a Value 7, Take-up 8 story on the basis of the raw values alone.

Rather it's the forum for the discussion of the values that provides the real insight, and allows everyone to have their input.

It's working to ensure that the team is working on what the whole customer team decide is the most important task, and not just the one with the loudest voice...

Technorati Tags: XP, Extreme+Programming, agile, software, development, programming, estimating, user+stories, software+development, Robert+Baillie

Firefox past 10 percent

2005-11-11T16:09:00.000Z

According to this week's New Scientist, Firefox has managed to get more than 10% of the web browser market.

When I read that I was surprised.

It seems that readers of this blog are much more discerning than your average internet user. For some time now Firefox has been the browser used in more than HALF the requests to this site!

	New Scientist report	Bobablog readers
MSIE	85.45%	44.42%
Firefox / Mozilla	11.51%	51.44%
Safari	1.75%	1.56%
Netscape	0.77%	0.32%
Opera	0.26%	1.56%

I'm sure that says more about bloggers and their readers than it does about this site in-particular...

Non-techie Tip of the Month

2005-11-11T15:59:00.000Z

Sorry to still be way off topic, but I was given a beautiful gem of a tip today.

Have you ever accidentally written on a whiteboard with a permanent marker?

If so, you'll know how painful it is trying to remove it again with solvents. Far too much elbow grease required for my liking.

Well, we had one of those moments today and someone piped up with a glorious bit of help.

Simply write over the existing text with a proper whiteboard marker, then rub it off with a normal whiteboard rubber.

Zero effort and a clean whiteboard!

I promise that normal Oracle and PHP based services will be resumed soon...

Subversion de-perversion

2005-11-08T09:20:00.000Z

Mr Mason has done it again.

For those that use Subversion for their version control, Mike Mason's blog is a must subscribe. He doesn't post often, but when he does it's invariably useful. His latest post covers splitting and merging Subversion repositories. As usual, reading it leads to an "Of course that's how you it!" moment.

If you haven't already picked it up, his book, "Pragmatic Version Control with Subversion" is a great introduction to Subversion.

Technorati Tags: Subversion, version+control, CVS, VCS, SCM, Mike+Mason, Robert+Baillie

Questionable Search...

2005-11-07T10:06:00.000Z

I know I've been quiet for a while, but I'm still here. There's things I'm going to get round to saying sooner or later, but a recent move has meant a distinct lack of internet access at home.

For now though... my referrer tracking uncovered a hit to the site from an unexpected search, one that I'm not sure how to deal with...

What does smug mean?

Things I have learned this week

2005-10-14T17:25:00.000+01:00

A couple of friends and I were joking over the weekend that if we all learned one thing each, every day for a year, then at the end of the year we'd be able to release the book 'Over 1001 things that 3 people learned in a year'

Here's what I learned this week:

08/10/2005: Cannon Street London Underground station used to have an ornate glass roof that was removed at the start of WW2. It was moved to the countryside to avoid bomb damage. The barn it was kept in was bombed a week later, completely destroying the glass roof. Cannon Street station remained virtually untouched throughout the whole of the war.

09/10/2005: Archduke Franz Ferdinand was assassinated in Sarajevo.

10/10/2005: The ship the Queen Elizabeth 2 is always written down with the number 2 rather than the roman numerals II. This is done in order to differentiate it from the Queen with the same name.

11/10/2005: The most abundant metal in the human body is calcium

12/10/2005: The most abundant bird in the world is the domesticated chicken. In 2003 the US ate 8.2 billion chickens, and if a chicken is grown for 7 weeks before being killed for meat, then in order for the 65 million people in the UK to be able to eat their average of 0.75 chickens a week then a total of 341.5 million chicken must exist at any given time in order to support the market.

13/10/2005: It's not enough to learn from your mistakes, you have to actually remember that you learnt from them and put those lessons into action.

14/10/2005: Whenever you find a gap in a test and you think "No, I won't put that test in, it'll never happen", it will happen, and it'll probably happen on the live system and give you a big headache.

I cannot make any guarantee as to the accuracy or relevancy of any of the facts, nor will I compensate anyone for the loss of time spent reading them.

Casting a table: variations on a theme

2005-10-05T17:12:00.000+01:00

A heads up from William Robertson and I've taken yet another look at the TABLE command (which allows you to issue SQL against a PL/SQL table as if it was a database table). As he points out here, it seems that the TABLE cast has trouble working out the data type when you're doing a SELECT *, but can often work out the data type for the same statement with the column list specified.

Obviously, if I hung around on Ask Tom or Metalink I'd find it much easier to find these nuances out... and I'd do it in a far less public way. But hey, I'd probably forget it then, wouldn't I! ;-)

Anyway, I've put together 3 scripts that some illustrate slight variations that are available. Note I've included DROP statements to get rid of any nasty residue, not to imply that you should drop these types in any real application.

So, first up is a variation that uses a SELECT * and a TABLE + CAST selecting from a PL/SQL table of a primitive datatype


SET SERVEROUTPUT ON SIZE 100000

CREATE TYPE char_table_type AS TABLE OF VARCHAR2(1);
/

DECLARE
  --
  char_table           char_table_type;
  --
  CURSOR cur_tab IS
    SELECT *
    FROM   TABLE( CAST( char_table AS char_table_type ) );
  --
BEGIN
  --
  char_table  :=   char_table_type('A','B','C','D','E','F','G','H','I');
  --
  FOR tab_rec IN cur_tab LOOP
    DBMS_OUTPUT.PUT_LINE( tab_rec.column_value );
  END LOOP;
  --
END;
/

DROP TYPE char_table_type;
/

The second variation uses a SELECT column list, and a TABLE selecting from a PL/SQL table of a primitive datatype. Note that the column name is inferred.


SET SERVEROUTPUT ON SIZE 100000

CREATE OR REPLACE TYPE char_table_type AS TABLE OF VARCHAR2(1);
/

DECLARE
  --
  char_table           char_table_type;
  --
  CURSOR cur_tab IS
    SELECT column_value
    FROM   TABLE( char_table );
  --
BEGIN
  --
  char_table  :=   char_table_type('A','B','C','D','E','F','G','H','I');
  --
  FOR tab_rec IN cur_tab LOOP
    DBMS_OUTPUT.PUT_LINE( tab_rec.column_value );
  END LOOP;
  --
END;
/

DROP TYPE char_table_type;
/

The last variation uses a SELECT column list, and a TABLE selecting from a PL/SQL table of objects. Note that the column name is no longer implied, but there is a requirement to construct objects in order to use this variation.


SET SERVEROUTPUT ON SIZE 100000

CREATE TYPE char_table_row AS OBJECT ( letter VARCHAR2(1) );
/

CREATE TYPE char_table_type AS TABLE OF char_table_row;
/

DECLARE
  --
  char_table           char_table_type;
  --
  CURSOR cur_tab IS
    SELECT letter
    FROM   TABLE( char_table );
  --
BEGIN
  --
  char_table  :=   char_table_type( char_table_row ('A')
     ,char_table_row ('B')
     ,char_table_row ('C')
     ,char_table_row ('D')
     ,char_table_row ('E')
     ,char_table_row ('F')
     ,char_table_row ('G')
     ,char_table_row ('H')
     ,char_table_row ('I') );
  --
  FOR tab_rec IN cur_tab LOOP
    DBMS_OUTPUT.PUT_LINE( tab_rec.letter );
  END LOOP;
  --
END;
/

DROP TYPE char_table_type;
/

DROP TYPE char_table_row;
/

More investigation will take place, and I'm sure this won't be the last post on the topic!

Technorati Tags: , Robert+Baillie, oracle, table, cast, software+development, software, database, 9i, ask+tom, metalink

If I had a hammer...

2005-09-30T12:55:00.000+01:00

I'm sure that pretty much every experienced developer out there has heard the proverb:

If the only tool you have is a hammer, then everything's a nail.

Well, I've heard a few variations on the theme and so I've decided to start collecting them...

Sometimes it's just a nail, and all you need is a hammer.

Don't ignore your hammer just because you heard someone else hit their thumb with theirs.

When you're in the market for a new tool, make sure you're not buying yet another hammer.

You can enhance the usefulness of your hammer by adding a chisel.

A Rotary Electro-pneumatic hammer delivering 4600 hammer blows a minute is still just a hammer.

Anyone got any more?

The In Thing - Simplified

2005-09-28T13:59:00.000+01:00

I've had a few people visit the site looking for some help on the Oracle 9i TABLE command. With this, you can issue SELECT statements against PL/SQL tables.

Now, previously I'd posted about the self same command, in a series of posts here:

http://robertbaillie.blogspot.com/2005/07/in-thing.html
http://robertbaillie.blogspot.com/2005/07/in-thing-example.html
http://robertbaillie.blogspot.com/2005/07/in-thing-simpler-example.html

Soon, I plan to put a few more notes together, starting from a simpler example. But for now I'll just thank Amis for the post here, which has pointed out a complexity in my own example that I didn't realise...

For some reason, the TABLE command cannot work out the data type of variable passed to it, and so will tend to throw an exception:
ORA-22905: cannot access rows from a non-nested table item

As I'd only ever done this when working with package, I previously worked around this by coding a function that returns the content of the relevent package variable. TABLE has no problem working out the datatype returned from a function.

It turns out I could have got away with just a simple CAST.
E.G. SELECT * FROM TABLE( CAST( table_variable AS table_type ) )

So, with that in mind, the first 'In Thing' example becomes this:

The Example:

In order to have something to access, we need the tables and data:


CREATE TABLE product
  ( id     NUMBER
  , descr  VARCHAR2( 100 ) )
/

INSERT INTO product ( id, descr ) VALUES ( 1, 'one' );
INSERT INTO product ( id, descr ) VALUES ( 2, 'two' );
INSERT INTO product ( id, descr ) VALUES ( 3, 'three' );
INSERT INTO product ( id, descr ) VALUES ( 4, 'four' );
INSERT INTO product ( id, descr ) VALUES ( 5, 'five' );
INSERT INTO product ( id, descr ) VALUES ( 6, 'six' );

COMMIT;

In order to perform the cast we need to declare a type for the row in the table, and then the table type itself:


CREATE TYPE gt_id_type AS OBJECT ( id NUMBER )
/

CREATE TYPE gt_id_list_type AS TABLE OF gt_id_type
/

Then we define the package that will perform the cast.
We have the following functions

GET_PRODUCTS, which is passed a table of IDs, and returns a ref cursor containing the products requested.

GET_PRODS_BY_NAME, which is passed a string, and returns all the products that have a description containing that string. This function uses GET_PRODUCTS to return the product details


CREATE OR REPLACE PACKAGE product_pkg AS
  TYPE gt_product_cur IS REF CURSOR;
  FUNCTION get_products ( pt_id_list  gt_id_list_type )
                                    RETURN gt_product_cur;
  FUNCTION get_prods_by_name ( pc_product_name   VARCHAR2 )
                                    RETURN gt_product_cur;
END;
/

CREATE OR REPLACE PACKAGE BODY product_pkg AS
  --
  gt_id_list     gt_id_list_type;
  --
  FUNCTION get_products
    ( pt_id_list  gt_id_list_type )
    RETURN gt_product_cur IS
    --
    vt_product_cur gt_product_cur;
    --
  BEGIN
    --
    gt_id_list := pt_id_list;
    --
    OPEN vt_product_cur FOR
      SELECT *
      FROM   product
      WHERE  id IN ( SELECT id
                     FROM   TABLE( CAST( gt_id_list AS gt_id_list_type ) ) );
    --
    RETURN vt_product_cur;
    --
  END;
  --
  FUNCTION get_prods_by_name
    ( pc_product_name   VARCHAR2 )
    RETURN gt_product_cur IS
    --
    vt_product_ids   gt_id_list_type;
    --
  BEGIN
    --
    SELECT  gt_id_type( id )
    BULK COLLECT
    INTO    vt_product_ids
    FROM    product
    WHERE   descr LIKE '%'|| pc_product_name|| '%';
    --
    RETURN get_products( vt_product_ids );
    --
  END;
  --
END;
/

Finally, we have some code to run the GET_PRODS_BY_NAME function and return its set of values.


SET SERVEROUTPUT ON SIZE 1000000

DECLARE
  --
  vt_cur          product_pkg.gt_product_cur;
  vr_product_rec  product%ROWTYPE;
  --
BEGIN
  --
  vt_cur := product_pkg.get_prods_by_name( 't' );
  --
  LOOP
    --
    FETCH vt_cur INTO vr_product_rec;
    EXIT WHEN vt_cur%NOTFOUND;
    DBMS_OUTPUT.PUT_LINE( vr_product_rec.descr );
    --
  END LOOP;
  --
END;
/

And the output:


two
three

Disclaimer: I've not yet managed to test this out with access from a PHP layer...

Technorati Tags: Robert+Baillie, oracle, table, cast, software+development, software, database, 9i, Amis

Friday afternoon...

2005-09-23T17:36:00.000+01:00

Friday afternoons always seem to lead to pointless conversations, which sometimes lead to classic moments of comedy...

During an E-mail conversation about being vegetarian:

"In fact humans aren't carnivores at all. We meat eaters are technically eating carrion, which is meat that is dead for several days or weeks. That's why they hang cows and stuff up to enhance their flavour. We eat carrion, I.E. like vultures and hyeenas. This means we shouldn't kill at all, just eat what we find. I just happen to find it in the supermarket. It's not my fault there's a load of people who go around murdering animals and leave them in shops for me!"

Google maps? PAH!

2005-09-19T14:07:00.000+01:00

Ever since Google released the API for Google Maps it seems the whole of the internet has gone crazy for it. Cool things have appeared like an index of the current BBC news stories on a map of the UK. Add to that the fact that Google Earth is pretty tasty too and you'd be forgiven for thinking that no other mapping tools existed.

But... there is, and one of them does something cool and useful that Google Maps doesn't.

I run. Not much, but enough to manage a 10km run without passing out. And when I train I like to make sure that I know exactly how far I'm travelling. I like to plan my training schedule so I know that on Monday I'll do 5km, on Wednesday I'll do 2km fast + 1km jog + 2km fast + 1km jog. I know, I know, I can't help it.

Anyway. Google Maps doesn't help me there. But Map 24...

Map 24 has a really handy little ruler tool. Point and click a rubber band round the route you run and BOSH, the distance travelled rounded to the nearest 100th of a mile (or thereabouts). Very nice for working out 1km / 3km / 5km laps round your local streets.
Oh, and it does a really cool wooshy zoom thing when you put an address in!

Technorati Tags: Google, Google+maps, Google+earth, Map+24, Robert+Baillie

Feeling Smug

2005-09-16T13:37:00.000+01:00

We've just gone live with a 40 user pilot of the latest version of the product we're developing, and once again I've been reminded why we work in an extreme programming kind of way, and why sometimes I just plain love this job!

Feedback, from the coal face...

"This is really, really good. You've done a great job."

"It's very straightforward, isn't it?"

"I'm loving this! You can tell that a lot of thought has gone into this"

And my personal favourite:

"I thought I was going to hate it - but I love it!"

Technorati Tags: extreme+programming, XP, agile, Robert+Baillie, software, development, rollout

Ch-ch-ch-ch-Chaaaanges

2005-09-15T13:47:00.000+01:00

Well, after a couple of weeks of fiddling with HTML and CSS I've finally managed to replace the blog's bog standard Blogger template with a home grown one...
It may not be perfect, but at least it's mine!

Comments are more than welcome.

Easy RSS syndication with FeedDigest

2005-09-11T12:44:00.000+01:00

Thanks to a heads up from Andrew Beacock, I've taken a bit of a look at FeedDigest.

It's a very simple syndication site that allows you to easily put together RSS feeds and then produce HTML versions of them for placing on your site.

I can see how this sort of tool can really start to push the use of RSS. For me all it's meant that Bobablog now has the last 5 OraBlogs posts and the last 3 BobaPhotoBlog posts. But there's no reason why it should stop there.

You could put up a feed of your del.icio.us bookmarks, the latest news from the BBC, a central collection of your task lists from BackPack, or (using the search facility) a short list of your own posts on a particular topic.

The reason I think this is cool isn't because it does anything new... it doesn't. The reason is that it does things in a way that means that non developers can do it.

Technorati Tags: feed, digest, feeddigest, rss, syndication, feed+digest, Andrew+Beacock, Robert+Baillie

Database Patch Runner: Design by Contract

2005-09-10T12:41:00.000+01:00

So let's say that you've managed to put together a build script that will install the latest version of your database with the minimum of work and you've got your developers using the build to upgrade their own workspaces.

How can you push the patch runner? Actually test the upgrade and get easy to analyse information back from a nightly or continuous integration build? How can you protect your live databases and ensure that patches are only ever applied to databases when those databases are in the right state to receive those patches?

Bertrand Meyer developed the concept of Design by Contract. He suggested a technique whereby for each method there will be a contract, stated in terms of:

Preconditions: These conditions should be true before the function executes.

Postconditions: Given the preconditions are upheld, the function guarantees that on completion the postconditions will be true.

In some languages (e.g. Eiffel) this contract is enforced, and if either side of the contract is broken the method throws an exception.

Our patch runner works in a similar way. Whenever a patch is written, pre and postcondition scripts are written. The precondition script embodies the assumptions about / requirements for the state of the
database immediately before the patch runs. The postconditions state the shape the database should be in once the patch has completed.

Immediately before running a given patch the patch runner will run the corresponding precondition script. The patch will not then be run unless the precondition script both exists and runs to completion. Only if and when the precondition script runs and reports no errors will the patch be applied.
Once the patch is complete the postcondition script is executed. If the postcondition script is missing, or reports a failure, then the build stops and the error is clearly reported.
Only when the pre, patch and post scripts have completed is the patch log updated to state that the patch has been applied. If any point of the process reports an error then the patch log is updated to state that the patch failed, and for what reason.

For the majority of patches, the pre and postconditions are fairly trivial, and can appear to be overkill. For example, if you're adding an unpopulated column to a table then the precondition is that the column does not exist, the postcondition is that it does.

The real advantage comes when you write data migrations. For example, a patch is needed that will move column X from table A to table B. A requirement may be that every value that was in B must exist in at least one column in table A, that no data is lost during the migration.
A precondition could be that there is at least one destination row in table A for each value. By checking this condition before the patch starts we can stop the patch from destroying data.

Another example may be the reforming of some tables that contain monetary values. It may be required that whilst data is being moved between rows or columns, that the sum total of all values in a particular set of columns be the same on completion of the patch as it was at the start. The precondition script can store the original sum totals, the postcondition can then check the resulting totals after the patch is complete.

Producing the pre and postcondition scripts focus the developer on two things:

What assumptions am I making about the structure of the data I am about to transform, and which of those assumptions are critical to my patch's success?

How can I measure the success of my patch, and what requirements do I have of the resulting data?

Both of these things are very different to the usual focus of: What processes do I need to apply to the data? However, the result of focusing on the first two will usually result in a clear path to the third.

We've found that using this structure greatly increases the feedback we get from our builds.

If you can get your DBA to buy into the idea, you can produce an overnight test build using the most recent backup from your live system. Overnight, automatically copy live, upgrade it and have your build produce a report on the success of the current build against the current candidate for upgrade. This report will warn you if the data in the live system is moving away from the assumptions you originally had when producing your patches. It is much easier to deal with such issues when you find out during your development cycle, rather than during or, even worse, days after the upgrade has been applied to the production environment. An upgrade patch becomes a very reliable thing when it's been ran 50 times against 50 different sets of data, each time reporting its success of failure.

In addition, the postcondition scripts act as clear documentation as to the aim of the patch. This is of great use if and when you find a performance problem with a particular patch. Rewriting a patch 2 months after it was first put together is so much easier when you have a postcondition script to act as both documentation and sanity check.

As a possible variation, on finding failures it should be possible for the patch runner to automatically rollback the failed change, taking the database back to the state immediately before the patch started. In some environments this may be critical, however we've not yet found a pressing need and therefore in our version we do not automatically rollback failures.

Of course, the mechanism is only as good as the pre and postcondition scripts that are produced, but then you ensure quality by always pair programming don't you ;-)

Technorati Tags: Oracle, software, development, database, upgrade, blog, agile, Robert+Baillie, patch, DbC, Design+by+Contract, db, dbms

The Database Patch Runner: Dealing with code

2005-09-07T20:22:00.000+01:00

Previously I've talked about the database patch runner as a way of easily and consistently getting database changes into version control, onto databases and under automated frameworks. But one topic I think I glossed over was: Which database objects fall under the patch runner, and how do you deal with those that don't.

A big advantage of the patch runner is that it groups functional changes together and will ensure that those changes are applied to a given database once and only once. A disadvantage is that it removes the object centric view of changes from version control and puts it into the database. For tables, I really don't see this as a problem. For packages, procedure and functions this is simply unacceptable; we need to keep source code grouped in the more traditional object centric manner.

We could have developers make their changes to the individual procedures and suchlike and then get them to add the source code to the patches one they've finished. But they'd probably forget a file every now and again, and would moan about the task. We could have a build manager monitor the version control repository and generate the list of source code files that are needed for the jump between any two given tagged versions of the system and build patch files upon request. But that would mean that we would need to employ a build manager to do this job.

Ideally we want a solution that will minimise the overall work in managing the build script, not just the work for the developer.

One of the reasons why we use the patch runner is to ensure that destructive code, or code that changes the underlying structure of a database cannot be run more that once. The implication of this is that statements which can be ran multiple times without destroying structure or content, due to their intrinsic nature, do not need to form part of the patch runner.

That is, if a procedure was to be dropped and re-created, the only impact would be to temporarily invalidate any other procedures calling that procedure. Oracle can deal with this and can be set to re-compile dependencies when a procedure is called. The overall structure and content of the database is not changed by re-creating a procedure. The intrinsic nature of the procedure means that it can be re-installed without damaging the database.

This is not so when dealing with tables. If a table was to be dropped and re-created*, the impact would be to loose the data previously held in that table. The result of re-creating a table that has not changed is that the overall structure will remain the same, but the content will not.

* Re-creating a table here being: dropping the table, cascading the drop of any keys on / to that table, and re-creating both the table and those keys.
This is not a small amount of work in itself, and if this was not done, the structure would also change.

This difference is crucial in understanding which objects need to fall under the control of the patch runner and which objects do not, as well as understanding how we can then deal with the objects that do not.

Of course, there is nothing stopping you from putting any particular set of objects under the control of the patch runner if you feel it is appropriate. For example, indexes can be dropped and re-created without any risk of loss of content and so you may regard these as outside the patch runner's concerns. However, indexes on large tables can take time to create and most of the data migrations that take place in the patch runner scripts would benefit from those indexes. For these reasons it may sometimes be appropriate to put the indexes into patch scripts.

For us, the split between patch runner and non patch runner is as such:

Patch Runner:

Tables

Indexes

Materialized Views

Non Patch Runner:

Grants / Synonyms

Views

Packages / Functions / Procedures

The list isn't exhaustive by any means; it covers the objects we use. As any new object type comes into focus we get together to decide if it should be installed by the patch runner.

So, knowing which objects do not fall into the realm of the patch runner, we then need to actually deal with those objects.

As stated above, the primary factor for deciding when an object should fall under outside of the patch runner's remit is if that object can be freely replaced with a new version of that object with no further work. We use that to our advantage, and state that since all objects outside of the patch runner can be replaced when they have changed, then all objects outside of the patch runner will be replaced irrespective of whether that object has changed or not.

That is, our patch runner sits within a larger build script. In our case this build script is a master batch file and a collection of SQL scripts, though I see no reason why this shouldn't be Unix scripts, ANT tasks, or whatever. The overarching build scripts contain sections that will install objects that are required before the patch runner can execute, and others that require the patch runner to have completed its task before they are installed. In every case, each of these scripts install all objects required by the database, not caring if they previously existed in the correct form.

In some cases a single script will create the full set of a single object type; all grants are created in a single script, as are synonyms. In other cases a single script will exist for each individual object; each package specification has a script to itself, separated from its corresponding package body. In adding a new object to the database schema, the developer needs to add that object to the relevant controlling build script.

This setup allows us to separately manage the version control for each individual object where that object is sufficiently important, whilst managing the number of scripts that exist for the less verbose creation statements.

By ensuring that all objects are created during every build sequence we simplify two processes:

Implementing a change in a given object requires only that the create script is edited, therefore minimising the work a developer must do to implement that change.

Building the release does not require knowledge of the source version, only the target version, therefore minimising the work required in generating a release.

The result is that whilst the objects fall within a highly structured build script, the developer need not think about it except when adding new objects. The PL/SQL source code can be managed like any other source code, each component falling under version control as an individual component.

We acknowledge that the build script could be better, and that the controlling install scripts could be generated (run all files from directory x, or run all files with extension '.xxx'), though in reality the maintenance of this script is minimal (a single number of minutes each week). Also, putting this in place would have an impact only on the build script, not on the individual object create scripts.

Additionally, it can appear to be overkill to install the full set of source code on each build, even small patch releases. However, installing source code onto an Oracle database is generally pretty fast. We find that the build script spends far more time running data migrations than installing code.

We exclusively use packages (no standalone procedures, functions or triggers), and so our build scripts are simple: We can install all the package specifications, then all the package bodies and can be sure that they will compile; dependencies do not affect the order of installation.
Views are installed in an order dictated by dependencies, though this does not prove difficult. If it did I'm sure we could 'force create' them.

Simplicity aside, I'm certain that even if we had difficult interdependencies with a very large number of objects, we would first test the ability to manage their creation in this way.
In short, I acknowledge that for larger projects there may be difficulties with the approach in the simplest form, but I'm more than confident that the basic theory holds. If not, I'd like to hear why, since our small (ish) project will inevitably become a very large one, as is the way with database applications.

Technorati Tags: Oracle, software, development, database, upgrade, blog, agile, Robert+Baillie, patch

A little more conversation

2005-09-02T09:00:00.000+01:00

In my usual style, I'm finding it difficult to walk away from a conversation on Wilfred van der Deijl's blog, this time with a guy called Chuck.

His comment can be found here, with my reply following close behind.

I wouldn't normally link to a comment like this, but I feel the message I'm trying to get across with the Patch Runner is well illustrated by my confusions about Chuck's company's approach.

As he clearly points out, the CMM level 3 accreditation states that the process is repeatable, but not necessarily appropriate.

Technorati Tags: Oracle, patch, upgrade, software, development, CMM, Robert+Baillie, Wilfred+van+der+Deijl

WTF

2005-09-02T08:41:00.000+01:00

For a while now I've been reading the consistently amusing Daily WTF. Well now, thanks to William Robertson, James Padfield, Thai Rices and Adrian Billington we have the Oracle WTF. It's started well...

Technorati Tags: Oracle, WTF, software, development, internet,

Database Upgrade Scripts – Why all the effort?

2005-08-16T20:38:00.000+01:00

On Wilfred van der Deijl's blog, Chuck (amongst other things) effectively asks:

"How much effort do you dedicate to total automation when you still need someone smart enough to [deal with failures]".

In order to address this I think I first want to address the question, what are we trying to do with automation.
The aim of is to make it easy to re-run the upgrade with as little effort as possible, with as many configruations as possible, thus testing the upgrade as much as possible and hopefully making the upgrade solid and reliable. Of course we can't make sure that the upgrade works every time with every configuration of data, but we can make sure it runs successfully a lot more often than it doesn't.

This gives us confidence in our ability to change the databases, and in doing so makes us more likely to re-work difficult data structures rather than work around them. This means that our database is better to work with, which makes our jobs easier.

Also, whenever we make a problem free roll-out we increase the confidence the rest of the business has in our ability to do our job. It makes them more open to receiving more upgrades more often and makes them easier to roll them out when we do. This means our customers get their bug fies earlier and their enhancements earlier, and can therefore give us feedback on where to make more enhancements earlier in the software's lifespan.

In short, making the upgrades solid helps us write better software by allowing us to focus on the software rather than the software roll-out, and the customer on the possibilities of new software rather than software failure..

How does automation give us this? Well, the ability to automate builds does not, it's the way in which those automated builds are used that does: Why not make it possible for an overnight process to perform the upgrade to as many different production databases as you can gather the resources to do?

In putting together an automated build plan we want builds on test databases that occur every night, using recent (that day's?) backups of production systems. We want to be able to run through the upgrade as it would occur on live as many times as possible, and we want to do that without draining our resources to the point that it's all we are doing.

If we can have unattended backups of our live system, unattended rebuilds of that live system on a test server, unattended upgrading of that database and then unattended testing of the resultant application, then we can eaily test the upgrade of the live system from the moment the project starts to the night before the production upgrade takes place.

If we do this, then the odds are that by the time we decide to upgrade the production system we know about that last minute data problem that will cause the build to fail. We'll have fixed the problem before the upgrade started, and we'll have dealt with in when we're not in a panic, we'll have done all this without teams of users waiting for the production database to come back up.

Our aim is not to have a completely unskilled worker to perform the task of upgrading the production database, but to make sure the upgrade will go smoothly far more often than not. We want that standby DBA to stay on standby.

That seems like worth a bit of effort to me!

Technorati Tags: Oracle, software, development, database, upgrade, blog, agile, Robert+Baillie, Wilfred+van+der+Deijl, patch

A seminal essay and a difference of opinion

2005-08-12T19:46:00.000+01:00

Thanks to Wilfred van der Deijl's post here, I've just read a January 2003 article by Martin Fowler and Pramod Sadalage... and I can't believe that I had previously missed it!

Martin and Pramod very clearly discuss the overarching issues that appear once you start to look at database development in an agile manner, and you can very easily see from their discussion how important a clear and solid patch runner structure is to successful agile database development.

Not long after this article was written we were just starting to think about the same practices and many of the conclusions we came to matched. Not least the need to implement sandbox databases for each workspace and the need for consistent test data across those databases.

However, our conclusions did differ in one fundamental way, and I find it difficult to understand Martin and Pramod's (M&P) approach in this particular area.

My issue relates directly to the propagation of database changes across the development team. In M&P's approach, database changes are automatically applied to the development workspaces as they occur, managed by the team's DBA. They point out that:

'people are usually concerned that automatically updating developers databases underneath them will cause a problem, but we found that it worked just fine'

I am one of those concerned people. I am concerned in the same way that I would be if I was told that the source code I was working with would be automatically updated at regular intervals as changes occurred. Let me rephrase the above statement with that in mind:

'people are usually concerned that automatically updating developers source code underneath them will cause a problem, but we found that it worked just fine'

Suddenly the statement does not seem quite so reasonable!

In order to develop I need to clearly understand what is within my control and what is outside of my control. I need to be able to rely on the known state of my own workspace. For me, this is a founding principle behind the idea of the development workspace as a sandbox.
A given version of an application is developed to run against a given version of the database. Because of this, the database is as much a part of the workspace as the rest of the source code.

There is also the issue of the database change being applied in advance of the related code change being checked into the version control system, and the manual process therefore required. In M&P's process they will notify the DBA of the changes required once they are decided upon. At some point in the near future the DBA will then make the changes to the central databases and the changes will propagate to the development workspaces. As this happens there are three clear risks that I can see:

1.The database schema change is made a significant amount of time before or after the associated code change is committed to version control.
2.The change required by the developer is miscommunicated to the DBA and an incorrect change is applied.
3.The change is mistakenly applied to development databases that are following a different branch to that in which the change is made.

In all the above cases, the effective result is that the change may invalidate a particular component of the application due to the development (or integration) database being out of step with the source code being ran against that database.
I believe all this risks can be mitigated against by making the following changes to the described process:

1.Make the developers responsible for producing the actual patches that will be eventually be ran against all databases, development, through integration to live. These patches should form part of an clear structured patch runner, and should be placed in version control alongside the rest of the application source code.

2.Make each developer responsible for the maintenance of their own database workspace. Make it easy for a developer to upgrade their database, and make them do so using those scripts that will be ran against all other instances of the database. Make it part of the routine that immediately after updating their source code workspace they upgrade their database.

3.Put the database upgrade into the automated build / continuous integration test suite and ensure that the build being performed by the automated build is exactly the same as that which would be ran on the live system.

These practices have the added advantage of minimising the amount of work required by the DBA, who can exist in a more advisory role within the team.

Of course, there are times where data migrations will take a non trivial amount of time to complete. In these cases it is important that developers are warned of the fact and are able to schedule times when these changes take place so as to minimise the impact on their work.

Having said this, agile database development is a new venture for most everybody involved, and work such as this should not be underestimated, and cannot understated. There will likely be many different ideas on this topic, coming from many different people, and it's a pleasure to be involved. It is very easy to forget just how much database development has evolved in the last 10 years!

Technorati Tags: Oracle, agile, extreme+programming, software, development, Robert+Baillie, upgrade, database, Martin+Fowler, Pramod+Sadalage

Named Notation Parameters in easy to read unit test shock!

2005-08-10T13:57:00.000+01:00

One of the better kept secrets of Oracle is the ability to use 'named parameter passing' over 'positional parameter passing'. Oracle covers it here.

That is, if I want to call a procedure that has three parameters that have defaults, and I want to a value pass into the third parameter, I can do so by referencing the name of the third parameter….

Give the function definition:

    FUNCTION insert_employee( pn_department_id  NUMBER := NULL
                            , pn_manager_id     NUMBER := NULL
                            , pc_employee_name  VARCHAR2       ) RETURN NUMBER;

Can be called by performing the following:

    vn_employee_id := insert_employee( pc_employee_name => 'Rob Baillie' );

OK, so in the example given it may make a lot more sense to just order the parameters a little better, like have the employee name first, but you get the idea.

So, it looks cool, and you may be able to think of a few instances where it may prove useful. It can be handy if you think your procedure signatures are likely to change and you have an out parameter you want to keep to the end of the procedure definition...

    FUNCTION insert_employee( pc_employee_name IN     VARCHAR2 
                            , pn_department_id IN     NUMBER   := NULL
                            , pn_manager_id    IN     NUMBER   := NULL
                            , pc_error_message    OUT VARCHAR2         ) RETURN NUMBER;

It can be regarded as tidy make sure that error message parameter is at the bottom, but without having named parameter notation you would never get the benefit of the default parameters, and would have to change all your calls if you ever added a new field between the department and manager id.

Quite handy, but nothing really earth shattering.

The place where we’ve found the most use is in unit tests...

Lets say we’re writing a test for the function:

    FUNCTION insert_holiday( pn_employee_id   IN     NUMBER
                           , pd_from_date     IN     DATE
                           , pd_to_date       IN     DATE
                           , pc_error_message    OUT VARCHAR2  ) RETURN BOOLEAN;

Lets say that this function has many different ways it can fail: Any of the parameters being NULL, the from data being later than the to date, the total number of days over the allocation...

In each case the function will fail in the same way. Fail to insert the holiday record, return FALSE and set pc_error_message to the reason why it failed.

In order to test for this conditions we write a simple failure check procedure with a signature along the lines of:

    PROCEDURE check_insert_holiday_fails( pc_context      VARCHAR2
                                        , pn_employee_id  NUMBER    := 7438
                                        , pd_from_date    DATE      := TRUNC( SYSDATE )  + 10
                                        , pd_to_date      DATE      := TRUNC( SYSDATE )  + 17 )

Each of the insert_holiday’s input parameters are duplicated on the check procedure, and set to be valid values for that procedure.
Check_insert_holiday_fails calls insert_holiday with the passed in values and goes on to check for the correct three error conditions: No increase in number of holiday records, returning false and passing back an error message.

The pc_context is appended to each of the assertion texts in order to give a nice readout. So, for example, the check false assertion may be (using UtPlsql):

    utAssert.this( 'When ' || pc_context ||', insert_holiday returns false', NOT vb_result );

The idea is that if the procedure was called with only the context value specifed, the call to the tested insert_holiday function would be successful (and the test would fail, if that makes sense!).

So, we have a single procedure that will check that a failure state is returned, covering all the components. We can then call this with our error states, using the named parameter notation in order to only change the parameters we are interested in changing. E.G.

    check_insert_holiday_fails( 'invalid employee_id'    , pn_employee_id => -1 );
    check_insert_holiday_fails( 'NULL  employee_id'      , pn_employee_id => NULL );
    check_insert_holiday_fails( 'from date after to date', pd_from_date   => SYSDATE + 5, pd_to_date => SYSDATE + 2 );
    check_insert_holiday_fails( 'from date in past'      , pd_from_date   => SYSDATE - 1 );
    check_insert_holiday_fails( 'to date in past'        , pd_to_date     => SYSDATE - 1 );

The use of the named notation means that only the parameters important to the failure are stated, rather than the full list. This makes it easier to see each individual test case clearly. If the procedure being tested has a lot of parameters, then the advantage becomes very clear!

Technorati Tags: Oracle, unit+testing, extreme+programming, software, development, Robert+Baillie, test, testing

Update: Sorry for the chnage in permalink address. I just couldn't handle the typo in the title!

Haloscan Ping 0.3 (alpha 2) released

2005-08-06T18:07:00.000+01:00

For those that are interested, a new version of Haloscan Ping has been released. Thanks go to Andrew Beacock and Ryan Cullen for their input!

This version fixes a couple of minor bugs with the first alpha...

Atom feeds with no content, but a summary tag are now recognised correctly

NULL entries no longer cause the ping screen to crash

If an entry is shorter than the minimum number of characters, it will still be chopped off to the nearest sentence

What we're currently working on for the next version:

Ability to send multiple pings in a single submit

Automatically stripping tags from the content

As with the previous version, if anyone wants it, please mail me!

Technorati Tags: Robert+Baillie,Andrew+Beacock,Ryan+Cullen Firefox, extensions, Haloscan, trackback, Mozilla, browser, internet, web, blog, software, tech, technology

The Database Patch Runner: Table Centric Views

2005-08-06T17:16:00.000+01:00

On Wilfred van der Deijl's blog we got into a nice discussion about grabbing a table centric history of the patches applied to a database when using a functional patch centric upgrade organisation.

I think I've made it obvious in the past that I massively favour the functional grouping, since I think it makes the job of the developer and the person performing the upgrade that much clearer. However, I acknowledge the perceived need for a table history, and started thinking about how you might produce it.

Just as I was getting to some sort of conclusion, Wilfred put together a couple of great ideas... and I thought I'd give them a critical review.

Forgive me for this Wilfred... I really wanted to get this down, I think it's a great topic!

Group the patches by table, then add interpreted comments to the files in order to state which patch a given ALTER TABLE statement belongs with

I'm not sure about this solution for a simple reason. I don't like organising the source files by tables; I think this makes the developer's job that much harder.

In producing a patch the developer needs to update several files and cannot easily see the order in which these changes are applied.

I simple situation where this becomes very important is where a column needs to be moved from one table to another. In order to do so, I cannot see how this can be done without changing at least two files; one that adds the new column, and one that drops the old.. In addition, you need to find somewhere to put the data migration script. Does this go into to the DDL file for the table having the column added? Or do you could add a third file into the mix.

I have found in the past that the create column and data migration get put into the table scripts and there is a manual process added to ensure that the existing column gets dropped. The process is then forgotten in the heat of the production upgrade and the redundant column remains in the schema.

In addition, with the table centric grouping it is easy for a developer to miss a step in the DDL, a column isn't added to a table, for example. By having the functional grouping it's easier to spot this mistake since the developer has a single file that lists all the DDL applicable for that functional change.

Finally, there is a need to add that extra step... adding the comments to the table create scripts. I have an objection to this in the same way that I have an objection to JavaDoc comments. People simply don't add them, and nobody notices / cares until such time as those missing comments are crucial. By then it's too late. The comments could be policed, but whenever I see something that needs policing I see something that needs changing so the policing is no longer needed.

Another way would be to stick to your way with a SQL script per patch and just store all statements in the database... [snip] ... have system triggers that log all DDL statement against particular objects

I love this idea. I think this has got a lot going for it. The big advantage of this technique is that the person writing the patch does not need to change the way they work in order for it to be implemented.

As part of the patch runner installation it can add a system trigger to the current schema that will log the changes to the schema as they occur. This detail log can state which object that has changed, the change that was made, and the patch that issued the change.

The trigger can be notified of which patch is currently running (as stated by Wilfred) by the patch runner logging the name of the current patch in a package variable, a temporary table or some other session specific temporary store.

Once the patch runner has completed it can then produce a report stating which changes were applied by that upgrade, grouped by table. It can produce a report of all changes ever made to the database. If so required it could list the changes classified by some arbitrary grouping of objects (system functional area / table sizes).

As Wilfred states, there is a downside to this... the data is in the database, rather than version control. It is generated after the change has been applied. However, this data can be made available in version control fairly easily. The simplest solution is to have the report run regularly on a development or test server. Copy the report into version control and away you go. If you're running a nightly build then this can be done automatically each night from the test server in advance of the codeset being tagged.

Alternatively you can provide a simple user interface in order to query this information more interactively. If, for example, you're developing browser based applications this will be a fairly simple tool to produce within your normal toolset.

Additionally, such a system DDL trigger can be used to report on ANY changes made to the database, not just those applied through the official upgrade process. Having produced reports that will list object changes from the database it would be a trivial task to change those reports to list any objects that have changed without being ran through the patch runner. It may be that in a strict environment a change made outside of the patch runner (and therefore the standard application upgrade) could be blocked. Obviously, any developer or DBA that knows the structure will be able to work around it without too much work, but this then means that you are certain that the person doing so is consciously making a change outside of the accepted process, rather than because they are not aware that such a process exists.

As I've stated earlier, on Wilfred's blog, I'm not sure our organisation has a need for the table centric view of the changes made to a database. In fact I strongly urge those people who do so to ask why it is they do. I'd love to hear the reasons why... I feel as though I'm missing something!

That said, I think the second solution proposed by Wilfred is spot on, and if we were looking to produce a table centric view we would definitely follow that avenue first.

Technorati Tags: Oracle, software, development, database, upgrade, blog, agile, Robert+Baillie, Wilfred+van+der+Deijl, patch

Rant: You want my money?

2005-08-06T13:29:00.000+01:00

To all you companies out there that want to take my money, and want me to pay you by credit card, on the internet. When you ask for my credit card number, you've got three choices:

1 - Build the entryboxes so that it doesn't matter if I put spaces or dashes in, just let my format the number 1234-1234-1234-1234, strip the number out and just use that.

2 - Build the entry boxes so that I can't put spaces, dashes or too many characters in and make it clear that I can't. That way I'll type my credit card number with just the numbers.

3 - Let me type a combination of spaces, dashes and numbers in the way that's natural to me, then tell me I'm an idiot. Give me messages like "Your credit card number has too many characters, please try again", or (my personal favourite) "You have entered your card number with spaces, please enter just numbers".

Guess which option will reduce the chances of me using your company next time. Guess which option your company most likely uses.

The Database Patch Runner - Rollbacks

2005-08-05T19:29:00.000+01:00

In response to my earlier post here, Andrew Beacock asked a couple of seemingly small questions.

First of all, he asked for more detail. I'm going to address that in a later post.

The second question, I didn't really expect. He basically asked “How do you rollback a patch?”

I'm assuming that he's asked this question in relation to the point I made that went along these lines: Since the patch runner stops running patches whenever one of the fails, then recovery is simply a case of rolling back the changes that the patch had already made and then restarting the patch runner. If the runner ensures it doesn't attempt to re-run any successful patches, but does re-run the failed one then it effectively picks up from where it left off and goes on its merry little way.

The point I was trying to make was that since the patch runner stops, then you don't get into the situation where you're attempting to fix a problem that occurred half way through the upgrade by getting to the end that then picking up the mess it left in its wake. Instead you know where the upgrade failed, the patch log states it clearly; you know what the patch was trying to do, since you have a functional grouping of changes in the patch, and this gives you context for the failed change; you know that you only have to worry about the failure that is reported, not the hidden data problems that the rest of the upgrade generated because the first patch didn't do its job properly.

So how do you rollback the changes in order to get the patch up and running again?

To be honest, there's nothing I can think of other than getting your hands dirty; rolling up your sleeves and getting into the database. Picking apart what happened, piecing together the story and trying to gaffer tape up the problems.

However, you can make that job easier:

The big thing is... Think about transactions in your patch

Every DDL statement issues an implicit COMMIT, so don't pepper your patch with DDL, leaving data in a transient state committed in the database. Think about the order in which you do things in order to reduce the chances that a failure at any given point leaves data in an invalid state.

Ensure that your 'add to schema' DDL comes first. That way, if you're moving two columns between tables, and the second column's create fails, you don't get stuck in a situation where you need to reverse the first change or apply the second one manually.

Ensure that your 'remove from schema' DDL comes last. For the same reason as above. It's much easier to deal with if you know that your data migrations have completed and it's just a case of removing those columns manually and marking the patch as complete.

If you can get away with it, don't issue commits in your patch's data migration until it's completed. Admitted, if you're moving millions of rows of data around, this may not be possible. But think about it... do you really need to issue that commit every 10,000 records, or can you get away with running the whole lot through first? That way, when you get a failure, the patch runner can ROLLBACK and you know what state the data is in... the state it was before the patch started.

If you can't get away without committing regularly, or the patch simply takes to long to risk having to repeat it, think about how you can make the migration restartable. That is, store a watermark, make your patch keep track of where it was so that it can restart after a failure. Make sure that the patch can handle being restarted without it trying to re-do work it has already done. If it really is that risky, why not check that you have enough space in your rollback segments before you generate GBytes of data?

Ideally you want to make sure that your patch doesn't fail, but with a bit of thought before you write the patch, you can make your life easier when it does.

Technorati Tags: Oracle, software, development, database, upgrade, blog, agile, Robert+Baillie, patch

Comments back

2005-08-05T08:53:00.000+01:00

Comments should be running normally again now. The offending Haloscan comments javascript has been replaced with the original Blogger javascript.

I had inadvertently switched over to Haloscan comments without switching off the Blogger comments. So adding a comment when you arrived from a permalink and clicking on 'Post a comment' gave you a Blogger comment that couldn't be seen from the main page.

Ah well, you live and learn! Normal commenting has now been resumed.

Comment problems

2005-08-04T22:00:00.000+01:00

Just so you know, I'm having trouble with the comments on here...

I've managed to install two competing sets of comments.
Haloscan is dealing with the comments of the main page, then Blogger is dealing with the comments that you see when you look at an individual post. Look at this post and you'll see what I mean.

Over the weekend I'll try to make sure I fix the problem...
For now, please accept my apologies :-)

The Database Patch Runner

2005-08-04T08:40:00.000+01:00

Over here Wilfred van der Deijl has raised the time old question when dealing with databases.

How do you easily upgrade a database from an arbitrary version to a new particular version with the smallest amount of effort.

I responded with a brief description here , which caused Amihay Gohen to ask a question or two over E-mail. I thought I'd put together something a little more coherent and detailed here, rather than just send it on to Amihay directly.

This is the basic solution...

Whenever a developer / pair of developers need to change the database they write a patch. This patch contains any alter tables / data migrations / etc. A single file contains the full set of changes for a single conceptual change and may cover changes to several tables.

The patches are written in PL/SQL procedures. That is, each single set of changes is implemented in a single procedure. Each DDL command is issued in an EXECUTE IMMEDIATE, or similar, as is any DML that references a table or column that does not yet exist. The patch is written in a procedure so that we can use the Oracle exception handler to propagate fatal errors in the code to a controlling procedure, something that is far more difficult to do in a script.

These patch procedures are then installed by a "patch runner". This runner is guided by a .txt list of patches that is created by the developers. If a developer wants to add a patch then they write the procedure and add it to the patch list. The runner then uses this patch list to tell it which patches need to be installed, and in what order. The runner is written in PL/SQL, using UTL_FILE and DBMS_SQL to install the procedures.

The runner logs when a patch starts, in a table, and marks the completion or failure of that patch when it finishes. By looking in the patch log before it runs a patch, the patch runner ensures that it never attempts to run a patch that has previously been marked as complete.

In addition, if any patch fails then the patch runner aborts, stating that it failed. "Crash don’t trash!". If the patch runner is restarted then it will attempt to re-run any patches that previously failed. This makes it easier to restart a failed upgrade if ever required. You reverse out the changes already made by the failed patch and then rerun the patch runner.

The runner itself is installed in a single PL/SQL script, and we make sure that it installs all the components it needs before it starts, and tears itself down once its finished. It does not rely on any other part of the database or application in order to work. Since there are no dependencies outside of the patch runner install script, we can use a single patch runner for all of our database based applications, with no changes.

The installation and upgrade of the patch_log table is dealt with in the more traditional manner of a single CREATE TABLE at the start followed by ALTER TABLE statements as it changes over time. The patch log is not destroyed down with the runner.

So, in summary, you check out a given tagged version of the application to run against an arbitrary database and run the patch runner. It loads the list of patches that are applicable for that tagged version. It runs over each patch in turn and checks if it has previously ran. If it has not, then it runs the patch.
By the time it reaches the end of the list it has ran all the patches, none of them twice. You can even run the patch runner twice in succession and guarantee that the second run will not change the state of the database.

The key to that is then to make sure that you use the build often. Get your developers their own database workspaces and make sure that that rebuild and upgrade them often. By repeating the upgrades often you ensure that they basically work before they get moved into a test environment with the bigger worry of large volumes of data.

The patch runner is, of course, only a small part of the database build, and a smaller part of the whole application build. So the runner needs to put into the context of a larger build script. We use batch files here in order to manage our database build process, but I see no reason why other, for complete scripting languages shouldn't be used, or ANT. The runner is, after all, just a PL/SQL script.

As an aside, we also write tests for the patches, checking the pre and post conditions are valid… but that’s a very different topic!

Technorati Tags: Oracle, software, development, database, upgrade, blog, agile, patch, Robert+Baillie, Wilfred+van+der+Deijl, Amihay+Gohen

Oracle Firefox

2005-08-01T17:38:00.000+01:00

Whilst it doesn't give you anything that a bookmark doesn't give you anyway, that little search box next to the address bar still feels pretty handy.

And if you're an Oracle developer, then tahiti.oracle.com is pretty handy too.

Well, someone out there has put the two together here.

Not sure I'll use it that often, but it's worth a look!

Update: 25/01/06
Turns out I use this absolutely loads. If you haven't installed it yet... get it on!

Technorati Tags: Oracle, Firefox, extension, oracle+documentation, Mozilla, Robert+Baillie, Eddie+Awad

Some screenshots of the Haloscan Ping Firefox Extension

2005-07-30T17:38:00.000+01:00

The configuration screen...

And the Ping Screen...

Technorati Tags: Robert+Baillie,Andrew+Beacock Firefox, extensions, Haloscan, trackback, Mozilla, browser, internet, web, blog, software, tech, technology

Trackbacks are getting easier

2005-07-30T16:46:00.000+01:00

Ever since I was introduced to trackbacks a couple of months ago I've had a worry about them... they're difficult to setup. So I thought I'd do something about it.

After a bit of help from my good friend Andrew Beacock, I got my head around trackbacks and set up a Haloscan account so I could get them installed onto my own blog. Getting it up and running was pretty straight forward, but then the tricky bit arrived... actually performing a trackback ping request.

In order to make a request you need to go to the Haloscan account and enter the trackback ping URL for the post that you want to ping. You then need to enter the details of your own blog... the blog title, post link, title and extract. It all seemed like a lot of hard work to me.

So here's what I did:
I wrote a Firefox extension to send trackback ping requests to Haloscan

In order to use it you need to set it up:

Install the extension

Configure it with the address of your blog feed (rss and atom are supported). You'll find it under Tools -> Haloscan Ping
Log into Haloscan

In order to send a ping request:

Write your blog entry and publish it

Go to the blog you want to ping, and select the trackback URL

Right-click and pick Ping Haloscan

Pick the entry from your blog to ping with and click OK

Simple as that!

At the moment I'd regard the extension as being at alpha stage, so for now I'm sending it out on request. If you'd like to try it out then mail me and I'll send you a copy.
Once I've got some positive feedback for it I'll try to publish it on Mozilla Update and Mozdev.

Update:You can mail me here...

Technorati Tags: Robert+Baillie,Andrew+Beacock Firefox, extensions, Haloscan, trackback, Mozilla, browser, internet, web, blog, software, tech, technology

Index Cards

2005-07-29T13:43:00.000+01:00

In our version of XP, Stories get written on index cards.

Text on the front, in big marker pen describes the user action in a single sentence.
The aim is to get the people involved to think in small chunks, and to think about the fundamentals of each piece of work.

Text on the back, in biro describes any particular behaviour required, we call them the acceptance criteria. The aim is to get people to think about the exceptional requirements of a story, the bits that the simple view of the action doesn't cover, to think of the little gotchas that'll appear.

They work extremely well, focussing the mind on the job in hand.

But there's something I think we've missed the point of: For each story there is only one card.

Here are my thoughts...

Since there's only one card, the card needs to be with the people that are currently working on it. If the story's still being worked out, it's with the customer, if it's being developed then it's with the pair working on it.

That means that if the customer wants to change their mind after the story's been started by the development team then they need to find the pair that's currently working on it and speak to them.
Fair enough, we could have the stories stored electronically and then put a system in place to ensure that the pair are notified of any change as soon as its put onto the system. It wouldn't be difficult, the customer enters each story on the system, revising it until they think it's ready. The story gets marked as ready to start when they've had a discussion with the development team. Each pair would register which story they're working on, marking the story as in progress. And so, and so on.

Or, we could do the simple thing and use natural capabilities of the card; do the simplest thing that will work.

Simplicity is only part of the issue. The big advantage of actually tracking down the pair and speaking to them is that you then get into a conversation. A notification is one way, a conversation is two way. And things that are written down are open to interpretation.

Conversation is crucial to the success of XP, and keeping the story on the card just seems to me to be a great way of keeping that conversation going.

Technorati Tags: Robert+Baillie, Extreme+Programming, user+stories

The second saddest post ever

2005-07-27T13:59:00.000+01:00

The other thing I love about regular expressions is that you can NEVER test them enough. You always find yet another bit of text that doesn't behave itself. There's always better ways of doing what you're doing. And if you ever post a blog entry about one, you always have to follow it up with another that corrects it...


  [\w\W]{200,}?[\.!\?\)"\n\r]+

Damn!

RTFM (again)

2005-07-27T08:45:00.000+01:00

When I first started pair programming it took me about 2 hours to come to terms with the fact that sometimes you need to ask for help, and that it's OK to look in a manual. You won't lose face with your pair for doing that.

It took our boss about 2 weeks to notice the increase in traffic on uk.php.net caused by my arrival.

So why is it that whenever I'm coding on my own I will tend to try to hack through the undergrowth with a pen knife to get to the solution rather than ask someone if I can borrow their chainsaw?

The latest example is a problem I was having with a firefox extension.

I wanted the user to be able to select a piece of text, right click, pick my extension and have a dialog box where one of the fields contains the selected text.

Since I already knew how to do this I flew straight in and spent ages fiddling round in the dialog box's code with parent.getSelection(), parent.window.getSelection, parent.window.context.getSelection() and every other combination including parents, windows and selections that I could think of. Not a chance.

For some reason I couldn't get the text from the parent window. I suspect that firefox does not regard the window the user sees as the parent window, but I've not bothered looking to find out...

Anyway, quite early on in I thought to mayself:

'This would be easy if you could just pass arguments into the dialog box. I could just get the selection in the code that kicks off the dialog box and there we go'.

But since I knew window.open I knew this wasn't possible.

An hour later I decided to take a look at openDialog, which is the call I was actually making to open the window. It turned out that yep, you can pass arguments into the child dialog box.

If only I'd just accepted that I don't know everything and looked at it an hour earlier I'd have saved myself a lot of stress.

Note to self... RTFM!

CSS Rules

2005-07-26T14:16:00.000+01:00

We love CSS here, we try to use it as much as possible to ensure that the look and feel of our application is easily skinnable. We're pretty good at it, and we get by.

This guy though... now HE knows what he's doing :-)

UK Flag in CSS only

Hardware Tokens

2005-07-26T13:29:00.000+01:00

In our office, we all sit around a couple of sets of desks. You may say that we're definitely not 'geographically challenged'. Each pair can pretty clearly see every other pair.

We don't store any source code on our local machines. Since we're developing a PHP application we need to deliver it though a web server and it access a nice big Oracle database, and guess what... we've got a whole department geared up to supporting web servers and databases. So we don't have local installs of databases and webservers, we don't administrate them, and none of us want to.

So our workspaces live on a network drive. Yep, Win CVS is a bit of a pain over a network, but for now we're living with it. Our PHP files live on a web server so we can access them without running any uploads or anything.

We've given our workspaces numbers, prefixed with the name of the project. The source code lives in a folder with the name of the workspace and the database lives in a schema with the name of the workspace. Your pair is working in workspace 'Laurel1', you get set of source code 'Laurel1' and database schema 'Laurel1'. Nice and simple.

The only problem is, if no one individual owns a particular workspace, then how do we know which pair is allowed to work in which workspace?

We have mutually exclusive hardware tokens. Or, to put it another way... we have paper flags that we put into holsters taped onto the back of our flat screen monitors.
If you've got the Laurel1 flag then you've got the Laurel1 workspace, and only your pair is allowed to change the workspace.
If you 've not got the Laurel1 flag then you can't change the Laurel1 workspace.

Since everyone's living so close, it's easy to see the flags. In the very rare situation where a pair is working remotely, they phone up a proxy to get their flag, a humorous effigy is made of one of the pair, and the flag is given to the effigy.

We have a spare workspace too...

We call it integrate*. It's the workspace where the integration takes place. In order to do a large scale commit you must have the integrate workspace token. If you don't then you don't commit. The pair with the token is the pair that's currently doing an integration. It has the advantage of a separate build machine since you're forced to check in from your dev space and check out to the integrate space and everyone can see you're doing it.

Our hardware tokens make it easy to see who's working where.

And they were fun to make.

Technorati Tags: Robert+Baillie, Extreme+Programming, pair+programming, workspaces, development

* actually, we call it autotest, but if we got a new one we'd call it integrate... honest.

The saddest post ever

2005-07-25T17:23:00.000+01:00

One of the things that I had to come to terms with about 18 months ago was the fact that I needed to understand regular expressions.
The Oracle world has yet to really embrace these strange beasts, and so I was completely hidden from their glory.

Now, I've reached the point where I'm going to do possibly the saddest thing in the whole world. I'm going to post a recent regular expression onto this blog... here... now.

For a soon to arrive firefox extension, I needed to extract a number of complete sentences from a piece of text that numbered at least 100 characters in length. Here's how I did it:


     (.|\n){200,}?[\.!\?\)"\n\r]+

OK, if you use apostrophes as quotation marks it falls down a little, but then you deserve a slap anyway ;-)

For being so sad, I must now go and whip myself with a birch branch.

Updated: OK, so I'm hoping that it now deals with newlines properly as well ;-) Serves me right!