The Public Spending Code: D. Standard Analytical Procedures Overview of Appraisal Methods and Techniques

The Public Spending Code: D. Standard Analytical Procedures

Overview of Appraisal Methods and Techniques

D.01

Document Update Log

Document Summary: This document outlines the main appraisal methods and techniques which should be used as part of the Public Spending Code. It provides a brief introduction to each technique and contains reference material at the end of the document. This information is intended to provide a general overview of these techniques, helping to orient new Public Spending Code users and point the way to further more detailed material, both in the Public Spending Code and more generally.

1.         Overview of appraisal

The basic purpose of systematic appraisal is to achieve better spending decisions for capital and current expenditure on schemes, projects and programmes. This document provides an overview of the main analytical methods and techniques which should be used in the appraisal process. These techniques can also be used in the evaluation process. More detailed information on individual techniques can found in financial and economic textbooks, examples of which are listed at the end of this document and in other guidance material on the VFM portal.

An understanding of discounting and Net Present Value (NPV) calculations is fundamental to proper appraisal of projects and programmes. A good understanding of Cost Benefit Analysis (CBA), Internal Rate of Return (IRR), Multi Criteria Analysis (MCA) and Cost Effectiveness Analysis (CEA) is also essential for economic appraisal purposes.

2.            Analytical methods

The recommended analytical methods for appraisal are generally discounted cash flow techniques which take into account the time value of money. People generally prefer to receive benefits as early as possible while paying costs as late as possible. Costs and benefits occur at different points in the life of the project so the valuation of costs and benefits must take into account the time at which they occur. This concept of time preference is fundamental to proper appraisal and so it is necessary to calculate the present values of all costs and benefits.

Net Present Value Method (NPV)

In the NPV method, the revenues and costs of a project are estimated and then are discounted and compared with the initial investment. The preferred option is that with the highest positive net present value. Projects with negative NPV values should be rejected because the present value of the stream of benefits is insufficient to recover the cost of the project.

Compared to other investment appraisal techniques such as the IRR and the discounted payback period, the NPV is viewed as the most reliable technique to support investment appraisal decisions. There are some disadvantages with the NPV approach. If there are several independent and mutually exclusive projects, the NPV method will rank projects in order of descending NPV values. However, a smaller project with a lower NPV may be more attractive due to a higher ratio of discounted benefits to costs (see BCR below), particularly if there affordability constraints.

Using different evaluation techniques for the same basic data may yield conflicting conclusions. In choosing between options A and B, the NPV method may suggest that option A is preferable, while the IRR method may suggest that option B is preferable. However in such cases, the results indicated by the NPV method are more reliable. The NPV method should be always be used where money values over time need to be appraised. Nevertheless, the other techniques also yield useful additional information and may be worth using.

The key determinants of the NPV calculation are the appraisal horizon, the discount rate and the accuracy of estimates for costs and benefits.

Discount rate

The discount rate is a concept related to the NPV method. The discount rate is used to convert costs and benefits to present values to reflect the principle of time preference. The calculation of the discount rate can be based on a number of approaches including, among others:

  • The social rate of time preference
  • The opportunity cost of capital
  • Weighted average method

The same basic discount rate (usually called the test discount rate or TDR) should be used in all cost-benefit and cost-effectiveness analyses of public sector projects.

The current recommended TDR is 5%.  However, if a commercial State Sponsored Body is discounting projected cash flows for commercial projects, the cost of capital should be used or even a project-specific rate.

Internal Rate of Return (IRR)

The IRR is the discount rate which, when applied to net revenues of a project sets them equal to the initial investment. The preferred option is that with the IRR greatest in excess of a specified rate of return. An IRR of 10% means that with a discount rate of 10%, the project breaks even. The IRR approach is usually associated with a hurdle cost of capital/discount rate, against which the IRR is compared. The hurdle rate corresponds to the opportunity cost of capital. In the case of public projects, the hurdle rate is the TDR. If the IRR exceeds the hurdle rate, the project is accepted.

There are disadvantages associated with the IRR as a performance indicator. It is not suitable for the ranking of competing projects. It is possible for two projects to have the same IRR but have different NPV values due to differences in the timing of costs and benefits. In addition, applying different appraisal techniques to the same basic data may yield contradictory conclusions.

Benefit / Cost ratio (BCR)

The BCR is the discounted net revenues divided by the initial investment. The preferred option is that with the ratio greatest in excess of 1. In any event, a project with a benefit cost ratio of less than one should generally not proceed. The advantage of this method is its simplicity.

Using the BCR to rank projects can lead to suboptimal decisions as a project with a slightly higher BCR ratio will be selected over a project with a lower BCR even though the latter project has the capacity to generate much greater economic benefits because it has a higher NPV value and involves greater scale.

Payback and Discounted payback

The payback period is commonly used as an investment appraisal technique in the private sector and measures the length of time that it takes to recover the initial investment. However this method presents obvious drawbacks which prevent the ranking of projects. The method takes no account of the time value of money and neither does it take account of the earnings after the initial investment is recouped. For example, a project requires a €3 million investment and Option 1 returns €2 million in the first year and Option 2 returns €3 million for the same year. On this basis Option 2 is the preferred option as the payback period is shorter but if the cashflows changed in subsequent years and Option 1 returned €2 million annually  while Option 2 only earned €1 million annually, the chosen option would have been incorrect. The ordinary payback period should not be used as an appraisal technique for public investment projects.

A variant of the payback method is the discounted payback period. The discounted payback period is the amount of time that it takes to cover the cost of a project, by adding the net positive discounted cashflows arising from the project. It should never be the sole appraisal method used to assess a project but is a useful performance indicator to contextualise the project’s anticipated performance.

Sensitivity analysis

An important feature of a comprehensive CBA is the inclusion of a risk assessment. The use of sensitivity analysis allows users of the CBA methodology to challenge the robustness of the results to changes in the assumptions made (i.e. discount rate, time horizon, estimated value of costs and benefits, etc). In doing so, it is possible to identify those parameters and assumptions to which the outcome of the analysis is most sensitive and therefore, allows the user to determine which assumptions and parameters may need to be re-examined and clarified.

Sensitivity analysis is the process of establishing the outcomes of the cost benefit analysis which is sensitive to the assumed values used in the analysis.  This form of analysis should also be part of the appraisal for large projects. If an option is very sensitive to variations in a particular variable (e.g. passenger demand), then it should probably not be undertaken. If the relative merits of options change with the assumed values of variables, those values should be examined to see whether they can be made more reliable. It can be useful to attach probabilities to a range of values to help pick the best option.

Sensitivity analysis requires a degree of exploratory analysis to ascertain the most sensitive variables and should lead to a risk management strategy involving risk mitigation measures to ensure the most pessimistic values for key variables do not materialise or can be managed appropriately if they do materialise. It is important to take into account the level of disaggregation of project inputs and benefits – sensitivity analysis based on a mix of highly aggregated and disaggregated variables may be misleading.

Scenario analysis

The scenario analysis technique is related to sensitivity analysis. Whereas the sensitivity analysis is based on a variable by variable approach, scenario analysis recognises that the various factors impacting upon the stream of costs and benefits are inter-independent. In other words, this approach assumes that that altering individual variables whilst holding the remainder constant is unrealistic (i.e. for a tourism project, it is unlikely that ticket sales and café-souvenir sales are independent). Rather, scenario analysis uses a range of scenarios (or variations on the option under examination) where all of the various factors can be reviewed and adjusted within a consistent framework.

A number of scenarios are formulated – best case, worst case, etc – and for each scenario identified, a range of potential values is assigned for each cost and benefit variable. When formulating these scenarios, it is important that appropriate consideration is given to the sources of uncertainty about the future (i.e. technical, political, etc). Once the values within each scenario have been reviewed, the NPV of each scenario can then be recalculated.

Switching values

This process of substituting new values on a variable-by-variable basis can be referred to as the calculation of switching values. These can provide interesting insights such as what change(s) would make the NPV equal zero or alternatively, by how much must costs or benefits fall or rise, respectively, in order to make a project worthwhile. The switching value is usually presented as a % i.e. a 20% increase in investment costs reduces project NPV to 0.

This is very useful information and should be afforded a prominent place in any decision-making process. Moreover, given the importance of this information the switching values chosen should be carefully considered and should be realistic and justifiable. For example, for capital projects requiring an Exchequer commitment over the medium to long-term, operating and maintenance costs should always be examined. Similarly, any project reliant upon user charges should always examine the impact of changes in volumes and the level of charges.

Finally, the European Commission have suggested that when undertaking a sensitivity analysis a useful determinant of the most critical variables is those for which a 1 per cent variation (+/-) produces a corresponding variation of 5 per cent or more in the NPV.

Distributional Analysis The calculation of NPV’s makes no allowance for the distribution of costs and benefits among members of society. This is an important drawback if the intended objectives of a programme/project aimed at specific income groups. Differential impact may arise because of income, gender, ethnicity, age, geographical location or disability and any distributional effects should be explicit and quantified where appropriate. A common approach to take account of distributional issues is to divide the relevant population into different income groups and analyse the impact of the programme/project on these groups. Weights can be attached to the different groups to reflect Government policy. Carrying out a distributional analysis can be a difficult task because costs and benefits are redistributed in unintended ways.

3.         Economic appraisal techniques

Economic analysis aims to assess the desirability of a project from the societal perspective. This form of appraisal differs from financial appraisal because financial appraisal is generally done from the perspective of a particular stakeholder e.g. an investor. Sponsoring Authority or the Exchequer. Economic analysis also considers non-market impacts such as externalities.

CBA

The general principle of cost benefit analysis is to assess whether or not the social and economic benefits associated with a project are greater than its social and economic costs. To this end, a project is deemed to be desirable where the benefits exceed the costs. However, should the benefits exceed the costs, this does not necessarily imply that a projects will proceed as other projects with a higher net present value (NPV) may be in competition for the same scarce resources. In addition, there are affordability constraints which mean that projects should not proceed even if the NPV is positive.

In cost-benefit analysis all of the relevant costs and benefits, including indirect costs and benefits, are taken into account. Cash values, based on market prices (or shadow prices, where no appropriate market price exists) are placed on all costs and benefits and the time at which these costs/benefits occur is identified. The analytic techniques outlined above (i.e. NPV method, IRR method, etc.) are applied using the TDR. The general principle of cost-benefit analysis is that a project is desirable if the economic and social benefits are greater than economic and social costs. It is vital that cost-benefit analysis is objective. Its conclusions should not be prejudged. It should not be used as a device to justify a case already favoured for or against a proposal. Factors of questionable or dubious relevance to a project should not be introduced into an analysis in order to affect the result in a preferred direction. 

A more detailed guide on how to carry out a CBA is set out in Public Spending Code D.03 – Guide to Economic Appraisal: Carrying out a CBA.

Cost Effectiveness Analysis (CEA)

It is difficult to measure the value to society of public investment in social infrastructure because the outputs may be difficult to specify accurately and to quantify, and are not frequently marketed. In cases like these, the cost of the various alternative options should be first determined in monetary terms. A choice can then be made as to which of the options (if they all achieve the same effects) is preferable. CEA is not a basis for deciding whether or not a project should be undertaken. Rather, it is concerned with the relative costs of the various options available for achieving a particular objective.  CEA will assist in the determination of the least cost way of determining the capital project objective. A choice can then be made as to which of these options is preferable.

Evaluating options in CEA is best done by applying the principles of the NPV method to the stream of cash outflows or costs. The recurring costs of using facilities as well as the capital costs of creating them should be taken into account, particularly if they differ between alternative options. Usually, the aim will be to select the option which minimises the net present cost.

There is a particular need for consistency in the assumptions and parameters adopted for CBA and CEA appraisals. CEA is most applicable to healthcare, scientific and educational projects where benefits can be difficult to evaluate.

Cost Utility Analysis (CUA)

CUA is a variant of CEA that measures the relative effectiveness of alternative interventions in achieving two or more objectives. It is often used in health appraisals. In a CUA, costs are expressed in monetary terms and outcomes/ benefits are expressed in utility terms e.g. outcomes are often defined in quality adjusted life years (QALYs). This outcome measure is a combination of duration of life and health related quality of life. Whereas in a CBA, there is a requirement to attempt to place a monetary value on all benefits, CUA allows for a comparison of the benefits of health interventions without having to place a financial value on health states.

Multi Criteria Analysis (MCA)

Multi-criteria analysis (MCA) establishes preferences between project options by reference to an explicit set of criteria and objectives. These would normally reflect policy/programme objectives and project objectives and other considerations as appropriate, such as value for money, costs, social, environmental, equality, etc. MCA is often used as an alternative to appraisal techniques because it incorporates multiple criteria and does not focus solely on monetary values.

MCAs often include “scoring and weighting” of the relevant criteria reflecting their relative importance to the objectives of the project. Care should be taken to try and minimise the subjectivity of decision making in an MCA as this is a common problem with carrying out MCA’s. The relative importance of objectives and criteria to achievement of the project will vary from sector to sector. The Sponsoring Agency should agree these with the Sanctioning Authority.

In constructing a multi criteria analysis scorecard and determining the weightings to be given to criteria the aim should be to achieve an objective appraisal of project options and consistency in decision making. Judgments regarding the scoring of investment options should be based on objective, factual information. The justification for scoring and weighting decisions must be documented in detail. In this regard, the system should be capable of producing similar results if the selection criteria were applied by different decision makers. 

The main steps in the MCA process include:

  1. Identify the performance criteria for assessing the project
  2. Devise a scoring scheme for marking a project under each criterion heading
  3. Devise a weighting mechanism to reflect the relative importance of each criterion
  4. Allocate scores to each investment option for each of the criteria
  5. Document the rationale for the scoring results for each option
  6. Calculate overall results and test for robustness
  7. Report and interpret the findings

The importance of explaining the weights and scores fully, and interpreting the results carefully, cannot be over-stressed.

Sources for further reading

Brealey, R. A. and Myers, S. C., Principles of Corporate Finance, Ninth Edition.

Commonwealth of Australia, Handbook of Cost Benefit Analysis, 2006 European

Commission, Regional Policy, Guide to Cost -Benefit Analysis of Investment Projects, July 2008 Edition.

Gray, A. W., EU Structural Funds and Other Public Sector Investments – A Guide to Evaluation Methods, 1995.

HM Treasury, ‘The Green Book’, Appraisal and Evaluation in Central Government, HMSO, 2003.

HM Treasury, ‘The Magenta Book, Guidance for Evaluation , 2011.

IPA edited by Michael Mulreany, Cost Benefit Analysis Readings (2002),

New ZealandTreasury, Cost Benefit Analysis Primer, The Treasury, July 2005

Comments are closed.