Saturday, December 13, 2014

Throw it away - Why you shouldn't keep your POC

"Proof of Concepts" are a vital part of many projects, particularly towards the beginning of the project lifecycle, or even in the pre-business case stages.

They are crucial for ensuring that facts are gathered before some particularly risk decisions are made.  Technical or functional, they can address many different concerns and each one can be different, but they all have one thing in common.  They serve to answer questions.

It can be tempting, whilst answering these questions to become attached to the code that you generate.

I would strongly argue that you should almost never keep the code that you build during a POC.  Certainly not to put into a production system.

I'd go so far as to say that planning to keep the code it is often damaging to the proof of concept; planning to throw the code away is liberating, more efficient and makes proof of concepts more effective by focussing the minds on the questions that require answers..

Why do we set out on a proof of concept?

The purpose of a proof of concept is to (by definition):

  * Prove:  Demonstrate the truth or existence of something by evidence or argument.
  * Concept: An idea, a plan or intention.

In most cases, the concept being proven is a technical one.  For example:
  * Will this language be suitable for building x?
  * Can I embed x inside y and get them talking to each other?
  * If I put product x on infrastructure y will it basically stand up?

They can also be functional, but the principles remain the same for both.

It's hard to imagine a proof of concept that cannot be phrased as one or more questions.  In a lot of cases I'd suggest that there's only really one important question with a number of ancillary questions that are used to build a body of evidence.

The implication of embarking on a proof of concept is that when you start you don't know the answer to the questions you're asking.  If you *do* already know the answers, then the POC is of no value to you.

By extension, there is the implication that the questions posed require to be answered as soon as possible in order to support a decision.  If that's not the case then, again, the POC is probably not of value to you.

As such, the only thing that the POC should aim to achieve is to answer the question posed and to do so as quickly as possible.

This is quite different to what we set out to do in our normal software development process. 

We normally know the answer to the main question we're asking (How do we functionally provide a solution to this problem / take advantage of this opportunity), and most of the time is spent focussed on building something that is solid, performs well and generally good enough to live in a production environment - in essence, not answering the question, but producing software.

What process do we follow when embarking on a proof of concept?

Since the aim of a POC is distinct from what we normally set out to achieve, the process for a POC is intrinsically different to that for the development of a production system.

With the main question in mind, you often follow an almost scientific process.  You put forward a hypothesis, you set yourself tasks that are aimed at collecting evidence that will support or deny that hypothesis, you analyse the data, put forward a revised hypothesis and you start again.

You keep going round in this routine until you feel you have an answer to the question and enough evidence to back that answer up.  It is an entirely exploratory process.

Often, you will find that you spend days following avenues that don't lead anywhere, backtrack and reassess, following a zig-zag path through a minefield of wrong answers until you reach the end point.  In this kind of situation, the code you have produced is probably one of the most barnacle riddled messes you have every produced.

But that's OK.  The reason for the POC wasn't to build a codebase, it was to provide an answer to a question and a body of evidence that supports that answer.

To illustrate:

Will this language be suitable for building x?

You may need to check things like that you can build the right type of user interfaces, that APIs can be created, that there are ways of organising code that makes sense for the long term maintenance for the system.

You probably don't need to build a completely functional UI, create a fully functioning API with solid error handling or define the full set of standards for implementing a production quality system in the given language.

That said, if you were building a production system in the language you wouldn't dream of having in incomplete UI, or an API that doesn't handle errors completely or just knocking stuff together in an ad-hoc manner.

Can I embed x inside y and get them talking to each other

You will probably need to define a communication method and prove that it basically works.  Get something up and running that is at least reasonably functional in the "through the middle" test case.

You probably don't need to develop an architecture that is clean with separation of concerns that means the systems are properly independant and backwards compatible with existing integrations. Or that all interactions are properly captured and that exceptional circumstances are dealt with correctly.

That said, if you were building a production system, you'd need to ensure that you define the full layered architecture, understand the implications of lost messages, prove the level of chat that will occur between the systems.  On top of that you need to know that you don't impact pre-existing behaviour or APIs.

If I put product x on infrastructure y will it basically stand up?

You probably need to just get the software on there and run your automated tests.  Maybe you need to prove the performance and so you'll put together some ad-hoc performance scripts.

You probably don't need to prove that your release mechanism is solid and repeatable, or ensure that your automated tests cover some of the peculiarities of the new infrastructure, or that you have a good set of long term performance test scripts that drop into your standard development and deployment process.

That said, if you were building a production system, you'd need to know exactly how the deployments worked, fit it into your existing continuous delivery suite, performance test and analyse on an automated schedule.

Production development and Proof of Concept development is not the same

The point is, when you are building a production system you have to do a lot of leg-work; you know you can validate all the input being submitted in a form, or coming through an API - you just have to do it.

You need to ensure that the functionality you're providing works in the majority of use-cases, and if you're working in a TDD environment then you will prove that by writing automated tests before you've even started creating that functionality.

When you're building a proof of concept, not only should these tests be a lower priority, I would argue that they should be *no priority whatsoever*, unless they serve to test the concept that you're trying to prove.

That is,  you're not usually trying to ensure that this piece of code works in all use-cases, but rather that this concept works in the general case with a degree of certainty that you can *extend* it to all cases.

Ultimately, the important deliverable of a POC is proof that the concept works, or doesn't work; the exploration of ideas and the conclusion you come to; the journey of discovery and the destination of the answer to the question originally posed.

That is intellectual currency, not software.  The important delivery of a production build is the software that is built.

That is the fundamental difference, and why you should throw your code away.

Thursday, October 09, 2014

The opportunity cost of delaying software releases

Let me paint a simple picture (but with lots of numbers).

Some software has been built.  It generates revenue (or reduces cost) associated with sales, but the effect is not immediate.  It could be the implementation of a process change that takes a little time to bed in, or the release of a new optional extra that not everyone will want immediately.

It is expected that when it is initially released there’ll be a small effect.  Over the next 6 months there will be an accelerating uptake until it reaches saturation point and levels off.

Nothing particularly unusual about that plan.  It probably describes a lot of small scale software projects.
Now let’s put some numbers against that.

At saturation point it’s expected to generate / save an amount equal to 2% of the total revenue of the business.  It might be an ambitious number, but it’s not unrealistic.

The business initially generates £250k a month, and experiences steady growth of around 10% a year.

What does the revenue generation of that software look like over the first 12 months?


It’s pretty easy to calculate, plugging in some percentages that reflect the uptake curve:

Period Original Business Revenue Software Revenue Generation Additional Revenue
1 £250,000.00 0.2% £500.00
2 £252,500.00 0.5% £1,262.50
3 £255,025.00 1.1% £2,805.28
4 £257,575.25 1.6% £4,121.20
5 £260,151.00 1.9% £4,942.87
6 £262,752.51 2.0% £5,255.05
7 £265,380.04 2.0% £5,307.60
8 £268,033.84 2.0% £5,360.68
9 £270,714.18 2.0% £5,414.28
10 £273,421.32 2.0% £5,468.43
11 £276,155.53 2.0% £5,523.11
12 £278,917.09 2.0% £5,578.34
Total: £51,539.34

Or, shown on a graph:




So, here’s a question:

What is the opportunity cost of delaying the release by 2 months?


The initial thought might be that the effect isn’t that significant, as the software doesn’t generate a huge amount of cash in the first couple of months.

Modelling it, we end up with this:

Period Original Business Revenue Software Revenue Generation Additional Revenue
1 £250,000.00 £-
2 £252,500.00 £-
3 £255,025.00 0.2% £510.05
4 £257,575.25 0.5% £1,287.88
5 £260,151.00 1.1% £2,861.66
6 £262,752.51 1.6% £4,204.04
7 £265,380.04 1.9% £5,042.22
8 £268,033.84 2.0% £5,360.68
9 £270,714.18 2.0% £5,414.28
10 £273,421.32 2.0% £5,468.43
11 £276,155.53 2.0% £5,523.11
12 £278,917.09 2.0% £5,578.34
Total: £41,250.69

Let’s show that on a comparative graph, showing monthly generated revenue:


Or, even more illustrative, the total generated revenue:


By releasing 2 months later, we do not lose the first 2 months revenue – we lose the revenue roughly equivalent to P5 and P6.


Why?

When we release in P3, we don’t immediately get the P3 revenue we would have got.  Instead we get something roughly equivalent to P1 (it’s slightly higher because the business generates a little more revenue overall in P3 than it did in P1).

This trend continues in P3 through to P8, where the late release finally reaches saturation point (2 periods later than the early release – of course).

Throughout the whole of P1 to P7 the late release has an opportunity cost associated.  That opportunity cost is never recovered later in the software’s lifespan as the revenue / cost we could have generated the effect from is gone.

If the business was not growing, this would amount to a total equal to the last 2 periods of the year.

In our specific example, the total cost of delaying the release for 2 months amounts to 20% of the original expected revenue generation for the software project in the first year.


And this opportunity cost is solely related to the way in which the revenue will be generated; the rate at which the uptake comes in over the first 6 months.

Or to put it another way – in this example, if you were to increase or decrease the revenue of the business or the percentage generation at which you reach saturation point the cost will always be 20%.

So, when you’re thinking of delaying the release of software it’s probably worth taking a look, modelling your expected uptake and revenue generation to calculate just how much that will cost you…