The Public Spending Code:
C. Implementation and Post-Implementation
Reviewing and Assessing Expenditure Programmes
VFM & Policy Reviews (VFMPRs) are now a well-established feature of the evaluation landscape in Irish public policy-making. The Reviews, which are conducted in accordance with detailed guidelines laid down in a 2007 Guidance Manual, are generally carried out thoroughly and are useful in addressing the standard VFM questions that are relevant for any such review, including:
- What is the rationale and the objectives for the scheme?
- Are the objectives still relevant, in light of evolving policy priorities?
- Has the scheme achieved its objectives?
- How efficiently has the scheme been delivered?
- How does the scheme rate against alternative ways of achieving the same objectives?
However, the VFMPR process has not achieved its full initial ambitions, in terms of breadth of coverage and direct relevance for the resource allocation process. Some shortcomings that have been identified by practitioners include the following:-
- The VFMPR process can be quite time-consuming and administratively burdensome. This does not lend itself to timely turnaround of reports, and indeed some VFMPRs have in the past taken several years from start to completion.
- Related to this, it is difficult for the VFMPR process to cover a broad range of spending areas in any one or two year period. This problem, which is exacerbated in some areas by the shortage of staff with relevant analytical expertise, can make the VFMPR process seem removed from the regular Estimates cycle, whereby policy-makers must form an overall judgement about how resources should be prioritised and allocated.
- The VFMPRs do not share a common format or presentation, and there is no uniform standard for reporting the outcomes of a Review. A more standardised approach would enable policymakers to digest the findings of a report more readily, and would help to orient the VFMPR so that it provides clear answers to the key questions.
For these reasons, the Government has decided to update and streamline the VFMPR process in a number of ways, and to supplement it with more focused policy assessments, which can be conducted more quickly by trained evaluators within Departments / Offices and by the Central Expenditure Evaluation Unit (CEEU). These measures draw upon the experiences of conducting the 2011 Comprehensive Review of Expenditure, and are detailed below.
2. Updates to VFMPR Process
2.1 Existing VFMPR Procedures
Up to now, the VFMPR procedure has been governed by a 2007 Guidance Manual which has been updated on an ad hoc basis, and the key provisions of which can be outlined as follows:
(a) Selection of topics for review: All VFMPRs should be targeted at areas of significant expenditure where there is the greatest potential for them to add value and influence policy developments. The Minister for Public Expenditure & Reform prepares an annual schedule of reviews for approval by Government, taking account of suggestions prepared by Departments.
(b) Steering Committees: Each VFMPR should be overseen, managed and delivered by a Steering Committee appointed by the relevant Department, with an independent chair, and include representatives of the Department of Public Expenditure & Reform.
(c) Terms of Reference: The Steering Committee prepares the ToR for each review, on the basis of standard template drawn up by the VFMPR Central Steering Committee.
(d) Evaluation Framework: Under the ‘programme logic model,’ evaluators must have a clear sense from the outset of the rationale for a spending programme, expressed in terms of inputs, activities, outputs, results / impacts; their linkage to specific strategic and programme objectives; the performance indicators that can be used for these purposes; and the evaluation criteria to be used (rationale, continued relevance, efficiency, effectiveness etc).
(e) Planning: Decide upon evaluation criteria, methodologies, involvement of stakeholders; and then manage the conduct of the VFMPR – including data collection, analysis and evaluation – within budget, on schedule, and to proper quality standards.
(f) Methodologies: Detailed guidance is provided on methodological approaches for conducting the VFMPR, by reference to the programme logic model of the Evaluation Framework.
(g) Content: Guidance is also provided on the standard elements for inclusion in a VFMPR Report.
These VFMPR procedures and guidelines remain valid, but need to be updated and streamlined in a number of respects to allow for more timely conduct of Reviews, and more direct linkage to the annual and multi-annual processes of expenditure allocation. The intention is that the VFMPR process, and other supporting processes outlined in this document, will be more effective in helping Departments / Offices to remain within the fixed expenditure allocations set out as part of the Medium-Term Expenditure Framework (MTEF).
2.2 Changes to VFMPR Procedures
(a) Selection of Topics for Review / Coverage of Reviews
Under the pre-existing arrangements of the VFMPR process, the areas of Health, Education, Social Protection and Justice & Equality are required to conduct one review per year, while all other line Departments plus the Office of the Revenue Commissioners and the Office of Public Works carry out 2 reviews each in the three-year period. The new approach will satisfy the following objectives:-
- All Strategic Programmes of expenditure – which form the basis of the performance budgeting framework – should be subject, in whole or in part, to rigorous evaluation over the three-year period. While it may not be practicable to evaluate the entirety of spending under each Strategic Programme, significant elements of expenditure should be covered and VFMPRs should not be focused on schemes that account for very minor elements of spending (the Focused Policy Assessments outlined in section 3 below may have a role in that respect). Departments should focus in particular on the more discretionary areas of programme expenditure, where issues of both effectiveness and efficiency feature strongly. For Departments / Offices with a relatively high proportion of “non-discretionary” ongoing expenditure, VFMPRs will still have a key role to play in assessing issues of efficiency, scheme design / alternative modes of delivery etc.
- To facilitate this breadth of coverage, all VFMPR analyses should be completed to report stage within a 6 to 9 month timescale, as a rule, and should be planned and managed accordingly within each Department / Office.
- Following discussion between the line departments and the Department of Public Expenditure & Reform (D/PER) and following consultation with Government, the Minister for Public Expenditure & Reform will decide a comprehensive VFMPR schedule of topics to cover the coming year and the following two years (consistent with the new Medium Term Expenditure Framework or MTEF). The schedule will be made public and will be delivered upon by all Departments / Offices. The schedule will allow for limited flexibility to adjust topics on an annual basis, mainly to take account of viewpoints put forward by relevant Oireachtas Committees.
In proposing suitable topics for evaluation, Departments should have regard to the over-arching objective of facilitating the prioritisation, and re-allocation, of expenditure in support of Government Programme commitments and consistent with MTEF spending ceilings. Accordingly, it will be necessary to target evaluation resources at areas of significant expenditure, for which – on the basis of previous reports, the CRE or other analyses – there may be a prima facie case for critical examination of the current scale of resource allocation. In this regard, Departments/Offices will be required to clearly indicate the estimated total expenditure that the proposed reviews will cover.
Departure from the list of review topics approved by Government is only allowed in exceptional circumstances, and then only where a suitable topic of at least equal significance is substituted for a review being dropped.
(b) Steering Committee Membership and Meetings
Based on the experience of the 2009-11 round of VFMPRs, the Steering Committees of Reviews should be more focussed and limited to key relevant officials. The aim, where possible, should be no more than 5 officials on the Steering Committee, comprising the Chairperson, the lead evaluator, the D/PER representative and two other senior officials – whether from within the Department / Office or from elsewhere – with knowledge and experience that is relevant to the subject matter of the review and/or to the Department’s review/audit process more generally. External evaluation expertise on the Steering Committee can also be considered but stakeholders that are beneficiaries or sectional interests should not be members. Their views will be sought as required as part of the evaluation.
With the exception of the lead evaluator and chairperson the work of Steering Committee ordinary members is additional to their normal ‘desk jobs’. For this reason, care should be taken to minimise the demands upon their time. The Steering Committees, particularly in the case of smaller reviews, should hold fewer meetings at key stages in the process e.g. one/two to agree on ToR /work programme, two mid-review meetings to discuss progress and a final meeting to finalise the draft.
The work programme and role of the Steering Committee should be linked to key milestones in the lifecycle of a VFM. A timetable for delivery of key milestones should be agreed. The meetings of the Steering Committee will be dictated by the production of the required deliverable. If deliverables are produced in accordance with the agreed timetable then the VFMPR can meet its target delivery date. The role of the Committee is to sign-off deliverables associated with the key milestones and to give direction on work needed to produce the next deliverable.
|Sign-off: – Terms of Reference – Objectives of Programme being Reviewed – Programme Logic Model – Methodology incl. data required – Timetable for Deliverables||These are the basic foundation stones of a good evaluation and no further work should be done until these are signed-off. They should be signed-off after a maximum of two meetings of the Steering Committee.|
|Preliminary analysis of data gathered.||This allows the Steering Committee to form an opinion on whether the data received matches the expectations they had when the methodology was agreed. It also allows the Steering Committee to give advice/direction to the Evaluator on report drafting and further analysis of the data to support the findings that will emerge. One meeting of the Steering Committee|
|First draft of key chapters of the Report||Steering Committee gives its views on first draft of the key chapters. Gives advice and direction on the findings, on structure and drafting. One meeting of the Steering Committee|
|Final Report||Finalisation of the Report may take one or two meetings of the Steering Committee.|
(c) Approval of Terms of Reference
The first key task of a VFM Review Steering Committee is to draw up the Terms of Reference for the review. When a draft of the ToRs has been discussed with the Steering Committee, it should be forwarded for consultation to the relevant Vote Section in the Department of Public Expenditure and Reform. This can be done informally through the Department of Public Expenditure and Reform representative from the relevant Vote section on the Steering Committee. However, it should be noted that Vote Sections should submit all draft ToRs for approval at Assistant Secretary level before advising line Departments/Offices of agreement to proceed.
Following this consultation, any amendments that might be agreed with the Department of Public Expenditure and Reform should be made to the draft ToRs before they are submitted to the Secretary General/Head of Office who is carrying out the review. Where a matter of dispute exists, direct consultation should take place between the Secretary General/Head of Office and the relevant Department of Public Expenditure and Reform Assistant Secretary with the aim of resolving the point at issue. Once the Secretary General/Head of Office is satisfied with the draft ToRs, he or she will give authority to formally commence the review.
(d) Role of the Evaluator
The lead evaluator has a pivotal role in the evaluation. S/he is not a secretary to the Steering Committee but is conducting the VFMPR under their direction. The lead evaluator should have sufficient analytical capacity to deliver the review. It is a person’s knowledge/experience of evaluation and the VFMPR process that is important and not their prior knowledge of the programme being reviewed. The lead evaluator cannot be working in the area that is being reviewed. It is the lead evaluator’s responsibility to produce the deliverables required for the Committee. In advance of the first meeting the evaluator should have prepared a background document/presentation on the area being reviewed as well as draft TORs, PLM and methodology. This will facilitate an efficient running of the review. Meetings of the Steering Committee will be dictated by production of deliverables by the evaluator rather than a set timetable of meetings. The Committee should not need to meet with stakeholders or conduct field visits. They may do so if they wish but it is quicker to give direction to the evaluator on who should be consulted. The lead reviewer should keep the Chairperson of the Steering Committee informed of progress in the review and of any issues that may require resolution, so that timely decisions can be made and reports finalised in time.
(e) Use of paid consultants
The engagement of paid consultants to carry out VFMPRs, as a rule, is not permitted. The VFMPR work should be conducted from within the evaluation / policy analysis resources that are developed and maintained within each Department / Office. Exceptions can only be justified on the basis that a particularly complex piece of analysis is required and that the necessary skills are not available internally. Even where this is the case it is not a justification to outsource the whole review.
(f) Independent Chairpersons
It is best practice to have a fully independent Chairperson in charge of each Steering Committee. The Independent Chairperson is responsible for driving the review within schedule and within its Terms of Reference and acts as a key channel between the lead reviewer and the Steering Committee. It is the responsibility of the independent Chairperson to see that the review deadline is met. The Chairperson should not be the lead reviewer.
The CEEU will maintain the existing central list of suitably-qualified retired officials at Principal Officer grade (or higher) to act as independent Chairpersons, and will also compile a list of serving Principal Officers who have evaluation and VFM experience and who would be available to chair Steering Committees.
(g) Role of CEEU and publication of Assessments
The Vote Section in the D/PER will continue to be represented on the review Steering Committee. The CEEU of the D/PER will no longer be directly involved in reviews and will instead be involved in carrying out its own quality assessments of Reviews at terms of reference / work plan stage, interim and final draft report stages. These assessments will be made available to the Steering Committee and the final assessment will be made available online.
The evaluator should send the TORs, Objectives, PLM, Methodology, Timetable, first draft key chapters and first final draft, to the CEEU prior to the Steering Committee meeting at which these deliverables will be signed-off, and in reasonable time to allow the CEEU to return their written comments to the Steering Committee. The evaluator is free to avail of advice from the CEEU on a less formal basis prior to formally sending any deliverable. The CEEU may be requested by the Chairperson to attend to exchange views at particular meetings of the Steering Committee or to engage more fully on certain aspects of the review process, where in the Chairperson’s view this would be helpful; as a rule, the CEEU will endeavour to accede to such requests.
The CEEU will publish a simple tracking document on all VFMPR deliverables on its website in the format below. It will also publish its final review of the VFMPR here.
Tracking Table for VFMPRs
|Department||VFMPR||TORs, Objs’ PLM, Methodology rec’d||CEEU sign-off||First draft of key chapters||CEEU sign-off||First final draft||CEEU sign-off|
|Dept of Industry||Employment Grants||
|Dept of Sport||Sports Facilities||
(h) Timetabling of reviews
In order to ensure the relevance of evaluations, the annual cycle of VFMPRs will be aligned more closely within the new, ‘whole-of-year’ approach to setting expenditure allocations (see section 4 below). This involves the following elements:-
- The schedule of VFMPRs should be decided, following consultation with Government, by the Minister for Public Expenditure and Reform during the autumn of each year.
- Work on the VFMPRs for the year ahead should get under way immediately with appointment of Steering Committee, agreement of Terms of Reference and commencement of fieldwork. VFMPRs for later years, as specified in the multi-year schedule, can be planned for in advance, but the precise timetabling of these future reviews will be subject to revision in light of views expressed by the relevant Oireachtas Committees.
- The VFMPRs should be concluded within a timeframe (6-9 months from their commencement) that allows for the final Reviews to be submitted to the Oireachtas Committees during the course of the year, to inform discussions and debate of the following year’s Estimates.
- Accordingly, an end-date for each VFMPR should be specified from the outset of each Review, and this deadline will be regarded as fixed and binding.
(i) Compliance with Timeframes / Sanctions
In order for the evaluation process to be effective it is essential that, insofar as possible, that timeframes are strictly adhered to. If the Chairperson considers that the agreed timeframes may not be adhered to, he/she should notify the Head of Department/Office immediately, who in turn must request an extension of the deadline from the Head of the CEEU in the Department of Public Expenditure & Reform. In this context, the reason for slippage should be explained and a new deadline will be fixed, which will not exceed 3 months from original deadline.
(i) Principle of transparency
A primary rationale for the VFMPR process is to facilitate better resource allocation decisions by bringing to light, and testing, the evidential basis for spending programmes. Complementary to this is the general principle of transparency in relation to how public money is allocated, used and evaluated. The CEEU will maintain a central repository of all reports including terms of reference, timescale, status update and letters (if any) seeking extensions to deadlines on the http://publicspendingcode.per.gov.ie website. As a guiding principle, all of the background material that would be released in response to an FOI request should be made public at the same time as the VFMPR report is published.
(k) Completing the Report & Memorandum for Government
The final report should be submitted by the Steering Committee Chairperson to the Secretary General and Minister of the relevant Department for publication. A copy of the report should also be circulated to the Minister for Public Expenditure and Reform at this stage. Before publication of the final report the relevant Minister should bring a Memorandum to Government. This Memorandum should outline the main findings and recommendations of the report and the proposed responses to address any issues arising. The Memorandum should be submitted to Government within one month of the finalisation of the report. The following steps should be taken once the report has been cleared for release:
- lay the report before both Houses of the Oireachtas, ideally, along with the response of the Department/Office to the report’s recommendations; the Oireachtas Library requires six copies of the document together with the completed form. For further information, contact Oireachtas Library.
- copies of each review must also be forwarded to the relevant Dáil Select Committee. The Clerk of the Committee will be able to advise how many copies the Committee will need;
- the report (and the Department’s/Office’s response) should be published on the website of the Department/Office;
- two copies should be forwarded to the Central Expenditure Evaluation Unit;
- a copy of the report should be sent to the Department of Public Expenditure Vote Section;
(l) Uniform Reporting: ‘Balanced Scorecard’
As highlighted in the 2011 Comprehensive Expenditure Report, in order to bring greater uniformity and standardisation to the evaluation process, each VFMPR will include a ‘Balanced Scorecard’ which will be used to assess the programme against a range of criteria of use to decision makers. This standard approach will represent one key, recognisable output of the Reviews for all programmes, and will to some extent facilitate performance comparisons across programmes and across Departments. A draft approach to the Scorecard is outlined in Box 1 and will be further developed in consultation with the Public Service Evaluation Network.
3. Focused Policy Assessments (FPAs)
Building upon the experience of the 2011 Comprehensive Review of Expenditure (CRE), the full VFM & Policy Reviews will also be complemented with sharper and more narrowly focused assessments designed to answer specific issues of policy configuration and delivery. The experience of the Comprehensive Review of Expenditure – including the major analyses conducted by each Department, and the cross-cutting and thematic evaluations undertaken by the Central Expenditure Evaluation Unit (CEEU) in the Department of Public Expenditure & Reform – showed that it is possible to get a quicker turnaround, to high standards of quality, when specific timelines and specific policy questions are set.
These Focused Policy Assessments (FPAs) can play a useful role in addressing the following types of policy issue:-
- Cross-cutting issues of relevance to one or more department; typically conducted by the CEEU or by evaluation staff from relevant Departments working together;
- Evaluation of a discrete expenditure programme, to answer specific questions of programme design and delivery, by reference to one or more evaluation criteria;
- Preliminary evaluation of a more complex programme or inter-connected set of programmes, to scope issues that may benefit from full VFMPR.
To optimise the effectiveness of the FPAs, it is intended that the overarching process will be flexible and not overly prescriptive, however it is envisaged that the FPAs:
– Operate under a clear mandate from the relevant official with responsibility for Programme area and the Head of CEEU.
– Are conducted by a Department’s evaluation unit and / or by an evaluator from CEEU. Ideally there should be no more than one or two evaluators.
– Have tightly framed terms of reference focusing on the key issue at hand.
– Do not require a steering committee; the responsibility of the evaluation should be under the management of the head of the departmental evaluation unit or the head of CEEU, as appropriate.
– Are completed within tight timeframes, 3 months as a rule.
– Are routinely published on http://publicspendingcode.per.gov.ie subject to any necessary redactions arising under FOI legislation. Redactions should be kept to the minimum necessary and a justification for redactions should be published with the document.
4. Role of the Oireachtas and its Committees
The Comprehensive Expenditure Report 2012-2014 set out range of reforms and an enhanced role for the Oireachtas. As can be seen from the timetable below the Oireachtas and its Committees will now play an ongoing part in the new ‘Whole of Year’ budgetary process.
|Input from the Oireachtas: A New Annual Estimates Timetable|
|Under the new arrangements Estimates allocations will be determined in the following manner.
Start of year: Multi-annual expenditure ceilings are known
Spending allocations are set for each Department not just for the forthcoming year (n), but also for years (n+1) and (n+ 2). Ministers and officials have up to two years to plan their affairs so as to achieve policy objectives within these allocations.
Spring of each year: Engagement with Oireachtas Committees on allocations / Estimates
It is open to the Oireachtas Committees, from the early part of each year, to engage with Ministers and their Departments to exchange views on how the fixed allocations for future years should be allocated to best effect. These perspectives can be taken into account by Government as the Estimates allocations are considered over the remainder of the year.
April: Stability Programme Update
Just as the November 2011 Medium-Term Fiscal Statement set out the Government’s overall fiscal adjustment path for the 2012-2015, the Stability Programme Update (SPU) published in April each year will adjust these targets as necessary to reflect economic developments, input from the assessments of the independent Fiscal Advisory Council and indeed the views of the Oireachtas Committees. In this context, the multi-year fiscal planning horizon will be extended by a further year, including the new overall expenditure figures.
Autumn of each year: Further engagement on expenditure policy
As the Government’s annual Estimates process becomes more advanced, Oireachtas Committees will have further opportunities to engage on specific policy proposals. The Committees will be informed by the range of VFM Reviews and focused policy analyses generated on an ongoing basis as part of the Government’s new Public Spending Code.
End of each year: Estimates are finalised
The Estimates for the coming year will be published as part of the annual Budget process, having been informed by the input of the Oireachtas Committees over the preceding year.
February of the following year: Revised Estimates and “Performance Budgets”
More detailed versions of the annual Estimates, which will now include key performance information, will be published and referred to Dáil Select Committees for consideration. In this context, Ministers and public service managers can expect to be held to account for delivery – or non-delivery – of the targets and objectives spelled out previously.
This new approach allows greater opportunities for Oireachtas members, as representatives of the public, to play a more substantive role throughout the entire budgetary process, from initial allocation of funds, through to holding Ministers and public service managers to account for the achievement – or non-achievement – of stated performance targets. The VFMPRs in particular will be used to assist Oireachtas Committees in their assessment of resource allocation priorities. Completion of these reviews will therefore have to be more closely aligned with this timetable. The Oireachtas Select Committees can also play a role in setting the agenda of topics and programmes to be reviewed in the VFM process, and holding Departments to account for timely progress.
Each Department should avail of the opportunity presented by the new process to work in a proactive way – including through submitting lists of topics for the annual and multi-annual review cycle to Committees and soliciting their feedback, and through timely completion and submission of reviews during the course of the year to facilitate Committee consideration.
|‘Balanced Scorecard’: A New Standard for Programme Evaluation|
|A criticism of the VFM & Policy Reviews is that they are each conducted differently, the various Reports are presented differently from one another, and it is hard for policy-makers to form a common view of how particular programmes rate relative to other programmes.As part of the new process, all Reviews will have to include a standard report – a ‘balanced scorecard’ – based upon a number of important criteria that are common to all evaluations. These criteria include:-
Quality of Programme Design
Implementation of Programme / Scheme
This approach allows for an overall, standardised quality score to be put in place, providing a programme rating that is of use to policy-makers and to those – including Oireachtas Committees and the general public – scrutinising the cost-effectiveness of spending. In other countries, more general programme ratings using the ‘traffic light’ system are found to be useful:- HIGH Score (Green light) – the programme is well-specified, achieving its objectives, and cost-effective in general terms. INTERMEDIATE Score (Amber light) – the programme scores highly in some areas, poorly in others: scheme re-design or efficiency improvements must be considered. LOW Score (Red light) – poor evidence of delivery of objectives; scheme funding should be available for reallocation to other priority areas.