Quality Assurance – Compliance with the Public Spending Code
The Public Spending Code imposes obligations, at all stages of the project/programme life-cycle on organisations that spend public money. These obligations apply to those that have responsibility at the different stages i.e. those within the Sponsoring Agency or Sanctioning Authority responsible for appraising, planning, approving, implementing or reviewing.
An additional obligation of the Public Spending Code is that each Department should put in place an internal, independent, quality assurance procedure involving annual reporting on how organisations are meeting their Public Spending Code obligations. This new Quality Assurance procedure replaces and updates the “spot check” requirements previously laid down in Circular letter dated 15th May 2007. The old procedure required a report with five sections – (i) Steps taken to disseminate the Guidelines (ii) Description of current systems for appraisal and management (iii) Coverage of the spot-checks and the findings (iv) measures in place to ensure compliance and (v) the views and responses of Departments and Agencies regarding the spot-check findings.
This new Quality Assurance Process aims to be easier to understand, more of an aid to compliance and easier to complete. The QA process should not place an undue burden on organisations. QA does not involve doing or redoing any of the appraisal, evaluation or review work that is required elsewhere in the Code. QA reviews pieces of ex-ante appraisal, management, evaluation or review work done by others.
The Quality Assurance procedure is made up of five steps:
- Draw up inventories of projects/programmes at the different stages of the Project Life Cycle. It is expected that the Organisation’s Finance Unit is best placed to draw up this inventory. They may have to consult with others to ensure that they have the full picture on projects that are at the appraisal/planning stage i.e. have yet to incur expenditure. The person responsible for the Quality Assurance process should be satisfied that they have a full and complete inventory.
- The Organisation’s Finance Unit should publish summary information on its website of all procurements in excess of €2m, related to projects in progress or completed in the year under review. A new project may become a “project in progress” during the year under review if the procurement process is completed and a contract is signed. Department’s should also publish details of the website references where its agencies have placed information on procurements over €2m.
- Complete the checklists contained in this guidance document. Only one of each checklist per Department/Agency is required. Checklists are not required for each project/programme. The QA process is based on a sample.
- Carry out a more in-depth check on a small number of selected projects/programmes
- Complete a short summary report for the Department of Public Expenditure & Reform. The report, which will be generated as a matter of course through compliance with steps 1-4, involves minimum administrative burden and should be submitted by the end of February in respect of the previous calendar year.
Step 1 was not formally a part of the old process but it would have had to be completed in order to select the projects that were to be checked. The second step is new but should not be a significant burden as the inventory compiled as part of Step 1 will provide the master list. The set of checklists to be completed for Step 3 serve as prompts that allow organisations to self-assess how compliant they are at a general level and will allow them to identify areas that need attention. They can also measure progress from one year to another. Step 4 is the most detailed step. Organisations are required to look in detail at a small number of projects/areas of expenditure. The detailed checks will verify whether the assessments made when completing the checklists are accurate or not. Organisations may think that they are very compliant based on the initial surface checks but find that when the detailed checks are undertaken that the practice does not live up to the theory or vice-versa. This may prompt a revisit to the checklist assessments.
Responsibility for Quality Assurance
The Quality Assurance requirement rests mainly with the Sponsoring Agency. The questions in the self-assessment questionnaires have to be answered by the organisation that is responsible for the appraisal or management of an area of expenditure i.e. the Sponsoring Agency.
Departments are usually Sanctioning Authorities in respect of one part of their budget and Sponsoring Agencies for the remainder. Where the Department is the Sponsoring Agency it carries out Steps 1-5 of the QA process in respect of that part of its expenditure.
Departments in their role as Sanctioning Authorities must choose how they will implement the QA process for agencies that they fund. The Sanctioning Department could require those that they fund to complete the QA process and report it into them or the Sanctioning Authority could decide to take a hands-on role in part of the QA process particularly Step 4 where the Sanctioning Authority chooses to undertake one or more of the in-depth reviews.
Only Departments are required to send an annual QA report to the Department of Public Expenditure & Reform
Who Quality Assures compliance with the Public Spending Code?
The Quality Assurance process should be undertaken by internal staff that are as independent as possible of the areas responsible for appraisal, planning and implementation e.g. staff from the economic /evaluation units, financial management units, internal audit, staff from an evaluation unit in another Department or academics on a pro bono basis. The process should be led by a small group chaired at senior level (minimum PO).
Supplementary Quality Assurance by the CEEU
In addition to the quality assurance checks undertaken by Departments themselves, the Central Expenditure Evaluation Unit (CEEU) in the Department of Public Expenditure & Reform may undertake its own quality assurance checks from time to time. This exercise, which aims to promote a consistent approach to Quality Assurance and VFM enforcement across the public service, will not replicate the internal quality assurance process but may instead involve in-depth reviews of the processes followed for specific projects or programmes.
The five steps in the Quality Assurance procedure are described in more detail below.
1. Drawing up Inventories of projects/programmes
For Departments to know that they are compliant with the Public Spending Code they first need to be aware of the areas of expenditure to which the Code applies in their Department. The first step in the process is to draw up or update your inventories of:
(i) Expenditure being considered:
– Capital projects that are or were under consideration during the year. These should be broken down by their anticipated cost (between €0.5 – €5m, between €5m – €20m, greater than €20m). Grant schemes for capital purposes should also be also included here. – New Current expenditure programmes or significant extensions to existing programmes that will involve annual expenditure of €0.5m or more.
(ii) Expenditure being incurred
– Capital Projects (> €0.5m) that are at the implementation stage
– Capital Grant Schemes (> €0.5m) that are incurring expenditure
– Current expenditure schemes or programmes (> €0.5m) that are incurring expenditure
(iii) Expenditure that has recently ended
– Capital Projects (> €0.5m) that were completed in the year being reviewed
– Capital Grant Schemes (> €0.5m) that were completed or were discontinued
– Current expenditure schemes or programmes (>€0.5m) that were completed or were discontinued
It is expected that the Organisation’s Finance Unit is best placed to draw up this inventory. They may have to consult with others to ensure that they have the full picture on projects that are at the appraisal/planning stage i.e. have yet to incur expenditure. The person responsible, for the Quality Assurance process, should be satisfied that they have a full and complete inventory.
2. Publish summary information on your website of all procurements in excess of €2m, whether new, in progress or completed
Drawing from the inventory compiled or updated in Step 1 the organisation should publish, annually on its website, summary details of all procurements (capital and current) where the value exceeds €2m. This information should appear under the standard heading PROCUREMENTS/PROJECT PROGRESS on all Departmental websites. This information should be published concurrently with the quality assurance report i.e. by the end of February each year. The table below should be published for each project/procurement >€2m:
|Name of Contracting Body|
|Name of Project/Description|
|Tender advertised in:|
|EU contract award notice date|
|Expected Date of Completion per Contract:|
|Spend in Year under Review:|
|Cum Spend to end Year:|
|Projected final Cost:|
|Value of Contract variations:|
|Date of Completion:|
|Expected Output on completion (e.g. X km of road, No. units)|
|Output achieved to date (e.g. Y km of road, no. units)|
There should be an entry for all new projects and projects still in progress. Completed projects feature for the last time in respect of the year that they were completed.
The presentation of this information can be in tabular or spreadsheet if that is more convenient.
3. Checklists to be completed in respect of the different stages
Step 3 involves completing a set of basic checklists covering all expenditure. These are high level checks that should be readily completed within each organisation. The objective of the exercise is to provide local and senior management, and the public more generally, with a self-assessment summary overview of how compliant the organisation is with the Public Spending Code. More in-depth checks are carried out as part of Step 4.
The first checklist captures obligations/good practice that apply to the organisation as a whole. Each of the remaining checklists listed below (checklists in the Appendices) might apply to a number of projects/areas of expenditure. Only one of these checklists is required for each organisation. Organisations are asked to estimate their compliance on each item on a 3 point scale (Scope for significant improvements = a score of 1, compliant but with some improvement necessary = a score of 2, broadly compliant = a score of 3). This self-assessed estimate of compliance can be based on an appropriate sample of the projects/areas of expenditure that are relevant to the checklist. The sample could be 5-10% of projects/programmes. The sample should rotate from year to year. Using a sample, to form a view on what should be included for the organisation in the Checklist answers, is in keeping with the intention that the QA process does not become over burdensome.
Checklist 1: General Obligations not specific to individual projects/programmes
Checklist 2: Capital Projects or Capital Grant Schemes being considered
Checklist 3: Current expenditure being considered
Checklist 4: Capital Expenditure being incurred
Checklist 5: Current Expenditure being incurred
Checklist 6: Capital Expenditure completed
Checklist 7: Current expenditure completed
4. Carry out a more in-depth check on a small number of selected projects/programmes
Parts 1 & 3 of the Public Spending Code Quality Assurance process will give an organisation a good overview of how compliant its processes are with the Public Spending Code. Quality Assurance Step 4 is about examining in more detail a small subset of its practices to see if the practices used are of a high standard. This step requires a higher level of analysis and judgement than previous steps in the QA process. It may for example involve drawing conclusions on whether the CBA used to appraise a proposal for a large project was satisfactory or not.
Selection of subset for closer examination:
Over a 3-5 year period every organisation should ensure that every stage of the project life-cycle and every scale of project is subject to a closer examination. In any given year this may involve looking at a couple of large projects at appraisal/planning, implementation or review stages or looking at a larger selection of smaller projects. Not every organisation has a large project every year so where large projects, in the year under review, are at the appraisal stage, implementation stage or have recently been completed it is opportune to select them for closer examination. In other years when large projects may not be a feature there is an opportunity to select a number of smaller scale projects. The value of the projects selected per annum, should be at least 5% of the total value of all projects in the inventory. This includes projects at the appraisal stage that have yet to incur expenditure. A subset of more than 5% may be needed for large organisations or because of the way that expenditure is divided a 5% sample would not give good coverage. To allow flexibility the minimum of 5% can be achieved as an average over a three year period e.g. 8%, 4%, 3%. The same projects should not be selected more than once in a three year period unless it is as a follow-up to serious deficiencies discovered previously.
Where there is a scheme that involves a large number of grants then it is the scheme itself that is the unit that is examined, not all of the individual grants i.e. it will not be necessary as part of this QA process to check 5% of all grants paid. The appraisal work on the scheme itself might be reviewed i.e. was there sufficient analysis to reach a conclusion that introducing the scheme was the best option to meet the objectives pursued? A small number of individual grants might be checked to confirm (i) that the conditions attaching to a grant matched the scheme design e.g. is this the subset of the population that we intended to target?, and (ii) that there was reasonable evidence that the scheme conditions were complied with.
This approach leaves organisations the greatest flexibility to cover the whole spectrum of projects and life-cycle phases over a number of years but also allows them to focus on large items at the most appropriate time.
What is expected of a more in-depth check?
Step 4 will look at a small subset and probe the quality of the work carried out. Step 3 above looks for basic indicators of compliance with the Public Spending Code i.e. if the project is over €20m, a CBA is required. Step 3 does not involve an assessment of whether or not the CBA is up to standard. Step 4, in contrast, looks in more detail at the quality of the appraisal, planning or implementation work done. This may mean:
– examining a CBA for a large project,
– an appraisal of a project under the €20m threshold,
– looking at how the outputs and outcomes for a current expenditure programme are defined and whether the data exists for on-going monitoring and evaluation
– examining how a large project was managed or
– looking at a post-project review
and making a judgement on whether the CBA, post-project review etc. was of an acceptable standard. Adverse findings might be that the estimated number of users of the proposed project was too optimistic, that the value of the benefit was overstated or unfounded, that other realistic options were not considered, that all costs including lifetime costs were not included, that the outputs were not defined prior to implementation or that data was not gathered during implementation to allow ongoing monitoring etc.
Step 4 may highlight, that while processes are in place and the organisation looks very compliant as per the checklists, there are deficiencies when more detailed checks are made.
Step 4 is a in depth look at how the organisation complies with the Public Spending Code. It is different from a Value for Money Policy Review (VFMPR). Step 4 looks at how the decision was made initially, was it soundly based, was it well managed and reviewed in more depth when necessary.
The VFMPR looks at whether the intervention chosen worked or not or whether it was efficiently implemented. An organisation can do everything right as per the Code and come through this Quality Assurance check with a clean bill of health but an intervention it has chosen to fund may be shown in a VFMPR to have failed in spite of the best appraisal, planning and management. They are two separate exercises. If a VFMPR found that an intervention failed then continued compliance with the Public Spending Code should mean that the intervention is either abandoned or redesigned to address the deficiencies.
5. Complete a short report for Department of Public Expenditure & Reform.
The final step in the Quality Assurance process is the completion of a report to be submitted to the Department of Public Expenditure & Reform by the end of February in respect of the previous calendar year. The report should contain:
– the inventory of project/programmes, current & capital as compiled by the organisation’s Finance unit;
– the website reference where details of procurements over €2m are published;
– completed checklists as per Step 3;
– the Department’s judgement on the adequacy of the appraisal/planning, implementation or Review work that it examined as part of Step 4 and the reasons why the Department formed these judgements; and – the Department’s proposals to remedy any inadequacies found during the Quality Assurance process.
This report should be certified by the Accounting Officer and published on the Department’s website.
The Quality Assurance Process should serve as an aid to each Department in its ongoing task of achieving the best value for money. The Quality Assurance process takes stock of how well an organisation does its job as steward of a significant block of public expenditure. Compiling and submitting a report will allow the Department of Public Expenditure & Reform to be of greater assistance in how it supports the achievement of this objective. It will also allow the D/PER and Departments generally to assess how appropriate this Quality Assurance Process is in practice and to make whatever adjustments may be required, in the context of the broader Consultation and Review procedures that are now integrated into the Public Spending Code itself.
CEEU Review of Compliance with Public Spending Code
The CEEU may make an annual assessment of each Department’s compliance with the Public Spending Code and may publish this assessment on its website. The assessment will be based on Departments’ Quality Assurance Reports, their record in completion of VFMs and any reviews that the CEEU itself conducts in Departments. Rather than focus only on deficiencies and shortcomings, it is important that instances of good practice be acknowledged, and that due credit should be given to Departments when they themselves identify and address deficiencies as part of the internal Quality Assurance process.
When completing the checklists, organisations should consider the following points.
- The scoring mechanism for the checklists is a follows:
- Scope for significant improvements = a score of 1
- Compliant but with some improvement necessary = a score of 2
- Broadly compliant = a score of 3
- For some questions, the scoring mechanism is not always strictly relevant. In these cases, it may be appropriate to mark as N/A and provide the required information in the commentary box as appropriate.
- The focus should be on providing descriptive and contextual information to frame the compliance ratings and to address the issues raised for each question. It is also important to provide summary details of key analytical outputs covered in the sample for those questions which address compliance with appraisal / evaluation requirements, i.e. the annual number of CBAs, VFMs/FPAs and Post Project Reviews. Key analytical outputs undertaken but outside of the sample should also be noted in the report.
Checklist 1 – To be completed in respect of general obligations not specific to individual projects/programmes.
|General Obligations not specific to individual projects/programmes||Self-Assessed Compliance Rating: 1 – 3||Discussion/Action Required|
|Does the organisation ensure, on an ongoing basis, that appropriate people within the organisation and its agencies are aware of their requirements of the Public Spending Code (incl. through training)?|
|Has internal training on the Public Spending Code been provided to relevant staff?|
|Has the Public Spending Code been adapted for the type of project/programme that your organisation is responsible for? i.e., have adapted sectoral guidelines been developed?|
|Has the organisation in its role as Sanctioning Authority satisfied itself that agencies that it funds comply with the Public Spending Code?|
|Have recommendations from previous QA reports (incl. spot checks) been disseminated, where appropriate, within the organisation and to agencies?|
|Have recommendations from previous QA reports been acted upon?|
|Has an annual Public Spending Code QA report been submitted to and certified by the organisations Accounting Officer and published on the organisation’s website?|
|Was the required sample of projects/programmes subjected to in-depth checking as per step 4 of the QAP?|
|Is there a process in place to plan for ex post evaluations?|
|How many formal evaluations been completed in the year under review? Have they been published in a timely manner?|
|Is there a process to follow up on the recommendations of previous evaluations?|
|How have the recommendations of VFMs, FPAs and other evaluations informed resource allocation decisions?|
Checklist 2 – To be completed in respect of capital projects/programmes & capital grant schemes that were under consideration in the past year.
|Capital Expenditure being Considered – Appraisal and Approval
|Self-Assessed Compliance Rating: 1 – 3||Comment/Action Required|
|Was a preliminary appraisal undertaken for all projects > €5m?|
|Was an appropriate appraisal method used in respect of capital projects or capital programmes/grant schemes?|
|Was a CBA/CEA completed for all projects exceeding €20m?|
|Was the appraisal process commenced at an early stage to facilitate decision making? (i.e. prior to the decision)|
|Was an Approval in Principle granted by the Sanctioning Authority for all projects before they entered the planning and design phase (e.g. procurement)?|
|If a CBA/CEA was required was it submitted to the relevant Vote Section in DPER for their views?|
|Were the NDFA consulted for projects costing more than €20m?|
|Were all projects that went forward for tender in line with the Approval in Principle and if not was the detailed appraisal revisited and a fresh Approval in Principle granted?|
|Was approval granted to proceed to tender?|
|Were procurement rules complied with?|
|Were State Aid rules checked for all supports?|
|Were the tenders received in line with the Approval in Principle in terms of cost and what is expected to be delivered?|
|Were performance indicators specified for each project/programme which will allow for a robust evaluation at a later date?|
|Have steps been put in place to gather performance indicator data?|
Checklist 3 – To be completed in respect of new current expenditure under consideration in the past year.
|Current Expenditure being Considered – Appraisal and Approval
|Self-Assessed Compliance Rating: 1 – 3||Comment/Action Required|
|Were objectives clearly set out?|
|Are objectives measurable in quantitative terms?|
|Was a business case, incorporating financial and economic appraisal, prepared for new current expenditure?|
|Was an appropriate appraisal method used?|
|Was an economic appraisal completed for all projects exceeding €20m or an annual spend of €5m over 4 years?|
|Did the business case include a section on piloting?|
|Were pilots undertaken for new current spending proposals involving total expenditure of at least €20m over the proposed duration of the programme and a minimum annual expenditure of €5m?|
|Have the methodology and data collection requirements for the pilot been agreed at the outset of the scheme?|
|Was the pilot formally evaluated and submitted for approval to the relevant Vote Section in DPER?|
|Has an assessment of likely demand for the new scheme/scheme extension been estimated based on empirical evidence?|
|Was the required approval granted?|
|Has a sunset clause been set?|
|If outsourcing was involved were procurement rules complied with?|
|Were performance indicators specified for each new current expenditure proposal or expansion of existing current expenditure programme which will allow for a robust at a later date?|
|Have steps been put in place to gather performance indicator data?|
Checklist 4 – To be completed in respect of capital projects/programmes & capital grants schemes incurring expenditure in the year under review.
|Incurring Capital Expenditure
|Self-Assessed Compliance Rating: 1 – 3||Comment/Action Required|
|Was a contract signed and was it in line with the Approval in Principle?|
|Did management boards/steering committees meet regularly as agreed?|
|Were programme co-ordinators appointed to co-ordinate implementation?|
|Were project managers, responsible for delivery, appointed and were the project managers at a suitably senior level for the scale of the project?|
|Were monitoring reports prepared regularly, showing implementation against plan, budget, timescales and quality?|
|Did projects/programmes/grant schemes keep within their financial budget and time schedule?|
|Did budgets have to be adjusted?|
|Were decisions on changes to budgets / time schedules made promptly?|
|Did circumstances ever warrant questioning the viability of the project/programme/grant scheme and the business case incl. CBA/CEA? (exceeding budget, lack of progress, changes in the environment, new evidence, etc.)|
|If circumstances did warrant questioning the viability of a project/programme/grant scheme was the project subjected to adequate examination?|
|If costs increased was approval received from the Sanctioning Authority?|
|Were any projects/programmes/grant schemes terminated because of deviations from the plan, the budget or because circumstances in the environment changed the need for the investment?|
Checklist 5 – To be completed in respect of current expenditure programmes incurring expenditure in the year under review.
|Incurring Current Expenditure
|Self-Assessed Compliance Rating: 1 -3||Comment/Action Required|
|Are there clear objectives for all areas of current expenditure?|
|Are outputs well defined?|
|Are outputs quantified on a regular basis?|
|Is there a method for monitoring efficiency on an ongoing basis?|
|Are outcomes well defined?|
|Are outcomes quantified on a regular basis?|
|Are unit costings compiled for performance monitoring?|
|Are other data complied to monitor performance?|
|Is there a method for monitoring effectiveness on an ongoing basis?|
|Has the organisation engaged in any other ‘evaluation proofing’ of programmes/projects?|
Checklist 6 – To be completed in respect of capital projects/programmes & capital grant schemes discontinued in the year under review.
|Capital Expenditure Recently Completed||Self-Assessed Compliance Rating: 1 – 3||Comment/Action Required|
|How many post project reviews were completed in the year under review?|
|Was a post project review completed for all projects/programmes exceeding €20m?|
|Was a post project review completed for all capital grant schemes where the scheme both (1) had an annual value in excess of €30m and (2) where scheme duration was five years or more?|
|Aside from projects over €20m and grant schemes over €30m, was the requirement to review 5% of all other projects adhered to?|
|If sufficient time has not elapsed to allow for a proper assessment, has a post project review been scheduled for a future date?|
|Were lessons learned from post-project reviews disseminated within the Sponsoring Agency and to the Sanctioning Authority? (Or other relevant bodies)|
|Were changes made to practices in light of lessons learned from post-project reviews?|
|Were project reviews carried out by staffing resources independent of project implementation?|
Checklist 7 – To be completed in respect of current expenditure programmes that reached the end of their planned timeframe during the year or were discontinued.
|Current Expenditure that (i) reached the end of its planned timeframe or (ii) was discontinued||Self-Assessed Compliance Rating: 1 – 3||Comment/Action Required|
|Were reviews carried out of current expenditure programmes that matured during the year or were discontinued?|
|Did those reviews reach conclusions on whether the programmes were efficient?|
|Did those reviews reach conclusions on whether the programmes were effective?|
|Have the conclusions reached been taken into account in related areas of expenditure?|
|Were any programmes discontinued following a review of a current expenditure programme?|
|Were reviews carried out by staffing resources independent of project implementation?|
|Were changes made to the organisation’s practices in light of lessons learned from reviews?|