22 October 2009 | Peter Smith
It's essential to have the right supplier, but how do you decide who's fit for the job? Peter Smith examines the pros and cons of a few approaches to assessing bids and tenders
We are procurement people. We buy things on behalf of our organisations. And despite all the strategic topics that occupy our minds, the fundamental thing we do as a profession is this: we choose the best supplier to meet our organisation's specific need.
Sometimes we might help internal colleagues make the decision, but generally it will be procurement that defines the selection process. Yet when it comes down to how we do that, there is limited guidance available. Textbooks tend to discuss the two key dimensions in most supplier selection decisions - the whole-life cost; and other "qualitative" factors such as quality, service levels and the supplier's ability to deliver innovation. But the key question is how do we score these criteria, particularly cost and price, then bring them together to achieve the final supplier selection decision?
Decision-Making
I recently saw a big procurement project where the entire supplier selection hinged on how cost was scored as part of the evaluation model. In this instance there were two acceptable vendors, both with strong bids; with a slightly different scoring scheme it could have gone the other way.
In some cases, organisations are making the wrong decisions (which could be justifiably challenged) because they do not understand the implications of scoring mechanisms. Sometimes, "cheap" suppliers are chosen who do not fully meet the needs of the organisation. In other cases, expensive suppliers are picked because price is not given enough prominence in the evaluation process. So how do you get the balance right?
Some argue trying to make these decisions through any logical, structured process is a waste of time. They might say applying a scoring process is meaningless and it should come down to judgment, but that approach brings a number of problems:
-
How do you justify your decisions to the market to maintain future competition? Market confidence will decline and suppliers will quickly stop bidding if they perceive an unfair or random selection process.
-
How do you avoid corruption if supplier selection is not structured and transparent?
Non-Cost Factors
There are strong arguments that a scoring system is essential for assessing bids. And in the public sector, a structured approach and a degree of transparency is mandatory. There are many options, but most evaluations will use a scoring mechanism applied to each question asked of the supplier or criterion.
The questions are designed to test the suppliers' ability to meet the buyer's needs in terms of each of the criteria - quality, service, innovation or whatever is appropriate to that contract. A common approach is to mark the response to each question as follows:
Scores can then be weighted individually according to the question and criterion. One weakness of this process is that scores around 50-80 per cent tend to be prevalent, particularly at the final tender stage when no-hopers have already been eliminated. A final "non-cost" score is arrived at, and can be converted into a percentage if required.
Typical non-cost evaluation criteria for a service-type contract include: technical solution proposed and competence; quality systems and processes; flexibility and change management; approach to working collaboratively; and management issues such as proposals on Tupe, equality issues and more.
Financial Evaluation
Assessing non-cost factors is quite straightforward. Scoring price and cost, however, and bringing it together with the non-cost score, is more challenging.
In terms of scoring price, many private sector firms do not even try. A more judgmental process is used. The buyer might start with a table like the one overleaf (see table A*) following the non-cost evaluation process outlined above.
Then a chart would be produced (see chart*). Any scores below a predetermined minimum might be excluded on the grounds that an overall score of, say, less than 50 per cent on non-cost factors meant the supplier was not up to the task. The dotted line on the chart represents this cut-off. A more sophisticated approach could have a cut-off based on a single criterion or question. If you apply this approach, it is good practice to inform suppliers.
The chart can then be used as the basis for a discussion by the tender board, evaluation panel or steering group. The starting point (or "null hypothesis") is that the lowest cost bid, above the minimum standard, would be the winner. However, the discussion would then focus on whether the additional cost of other bids could be justified by the additional non-price score. In other words, is it worth paying an extra million pounds to get better quality service, a more flexible approach or whatever factors had driven the higher score?
In the example above, supplier A is the starting point. Clearly, supplier D cannot win because the bid is both more expensive than B and scores less on non-cost factors. But the trade-off between A and B is interesting. Is it worth paying another £2 million for the extra 30 points scored for non-price factors? Similarly, is it worth paying a further £1 million to move from B to C to gain the extra 12 points? That would be the focus of the discussion. Often, good decisions are made through this process. But it contains an element of subjectivity, and the cost/non-cost trade-offs are not based on specific weighting or marking systems.
Purchasers, particularly in the public sector, are under pressure to develop clearer evaluation mechanisms. EU regulations mean as a minimum, ranking of criteria, if not weightings, need to be made public. And ranking costs against other award criteria implies a clear marking system must exist - and not just come down to judgment as in the chart example above.
Structured Approach
Recent challenges to procurement decisions in the public sector are increasingly focusing on the detail of the evaluation process. And even in the private sector, suppliers are asking for more detail around selection decisions. So how can we respond to this?
Using the example above (and ruling out supplier D for the reasons given), we can look at some other approaches and assess how robust, fair and effective they are.
Assuming a 50:50 weighting between total cost and total noncost factors, here are three relatively simple mechanisms currently in use for scoring price. For each of them, we will mark cost against a 0-100 scale. That mark will be weighted at 50 per cent, and added to 50 per cent of the non-price score.
1. Percentage scoring
The first approach is percentage scoring. This generally starts with scoring the lowest cost bid as 100. So in the example above, A scores 100. Supplier B is 20 per cent more expensive on price, so gets a score of 80 and supplier C is 30 per cent more expensive so it scores 70 (see table B*).
This has the advantage of apparent simplicity and fairness, but has some disadvantages. A 10 per cent cost difference may not sound much but on a major project, it could equate to many million of pounds. A score only 10 per cent lower than the cheaper bid may not reflect that adequately. And even a very expensive bid may score quite well. This approach also tends to produce high scores for the cost dimension, so in comparison to the non-cost scores, it may over-weight the price element.
Another variant of this is to apply a price curve rather than a straight line - so for a larger price, the score declines more rapidly. This can make very expensive bids less likely to win.
2. Score by rank
Here, we give the highest-priced supplier 0 and the lowest 100. All others are placed at equal increments between. So if there were 6 suppliers, they would be placed at 0, 20, 40, 60, 80 and 100.
This uses the full range of scores available. It certainly distinguishes clearly between costs. But it has two serious problems. Firstly, it is unfair. It could mean that two suppliers whose bids were only a few pounds different gain vastly different scores. Secondly, it distorts the weighting. Non-cost scoring based on the sort of marking scheme we have discussed tends to result in most suppliers scoring somewhere between, say, 50 per cent and 80 per cent. So in effect only a fraction - perhaps a third - of the possible scale is used (see table C*).
If this is combined with a price scoring system that uses the whole 0-100 scale, the effect is to amplify the weighting of cost against noncost. An expensive bid, however strong the quality, just cannot win under this system. If non-cost is scored on a ranking system also, this issue of relative weighting is avoided. However, the issue of unfairness is then repeated for non-cost factors. To my mind, these disadvantages should rule out this method. However, I have seen it used on some important tenders.
3. Highest/lowest conceivable price
This introduces a somewhat artificial construct, but does have advantages. I have recommended it and seen it used to good effect on some large public sector bids. It relies on the buying organisation pre-determining highest and lowest conceivable prices. The highest conceivable cost is usually defined in terms of the budget. If it is £15 million, then arguably any higher bid should score 0 for cost, or even be eliminated. (To add a safety margin, as budgets are rarely set in stone, the 0 score could be set at 10 per cent above the budget).
At the other extreme, what is the lowest possible cost that an organisation expects to get? In this case, let's assume that £7.5 million is the lowest price feasible; that cost would score 100. Table D (see table D*) shows how the bids score on that linear £7.5-£15 million scale.
This process can feel artificial. But it reflects "usefulness" quite well in that high bids can be "punished", but the scoring is less extreme than the second method. Often, it will lead to scores around the 50-75 per cent mark, which matches well with the non-price range. Bids that are very close score similarly, which would seem to make it a fair system.
The Results
With this study, under the ranking system, the cheapest bid wins by a distance. The percentage system has a close result, with the most expensive bid just beating the other two. And matters are even closer under the lowest/highest conceivable cost method, with the middle bid wining by a short head. What does this prove? Any of our bids can win given a scoring process that is favourable to their mix of cost and non-cost attributes. And if your mechanism comes up with a result that does not feel fair or desired, it is not easy (certainly in the public sector) to overrule it because a Freedom of Information Act request or any challenge will find you out.
Conclusions
1. Making the right choice of supplier, understanding how you arrived at that decision and the ability to justify it internally and externally are all critical aspects of the procurement process.
2. Organisations need to understand how different methods of scoring price can lead to different outcomes.
3. Designers of evaluation strategies need to understand how the chosen option will affect the selection decision.
4. Quoting a "weighting" in terms of criteria is meaningless if it is not associated with a scoring method.
5. Choose your method carefully. Make sure it gives bids with unaffordable costs suitably low marks - or excludes them. But also ensure small cost differences don't drive decisions.
6. Ranking-based scoring can lead to illogical decisions and - if ranking is used only for cost - over-weighting of price compared with other criteria.
7. A percentage mechanism - but applied on a non-linear basis to reflect unacceptably high pricing - is a reasonable and simple method.
8. The "lowest/highest conceivable cost" approach probably comes the closest of the options here to reflecting true "usefulness" and decision theory.
9. Modelling your scoring method in advance is advisable - try out some possible bid combinations to see if you get reasonable results.
Finally, it would be good to see more debate and work - academic or practitioner - in this area.
* Peter Smith is director of Procurement Excellence
*To see the tables referenced is this article please go to pages 30 - 34 of Supply Management's current online digital edition.
