A-04

Quality Assurance – Compliance with the Public Spending Code

A-04 

Document Update Log

Document Summary: The Public Spending Code will only be of use if it is complied with by those that are responsible for expenditure at the appraisal, planning, implementation or post implementation stages. This document describes what is expected of the internal independent team that will carry out quality assurance checks and produce the annual quality assurance report.

The Public Spending Code imposes obligations, at all stages of the project/programme life-cycle on organisations that spend public money. These obligations apply to those that have responsibility at the different stages i.e. those within the Sponsoring Agency or Sanctioning Authority responsible for appraising, planning, approving, implementing or reviewing.

An additional obligation of the Public Spending Code is that each Department should put in place an internal, independent, quality assurance procedure involving annual reporting on how organisations are meeting their Public Spending Code obligations. This new Quality Assurance procedure replaces and updates the “spot check” requirements previously laid down in Circular letter dated 15th May 2007. The old procedure required a report with five sections – (i) Steps taken to disseminate the Guidelines (ii) Description of current systems for appraisal and management (iii) Coverage of the spot-checks and the findings (iv) measures in place to ensure compliance and (v) the views and responses of Departments and Agencies regarding the spot-check findings.

This new Quality Assurance Process aims to be easier to understand, more of an aid to compliance and easier to complete. The QA process should not place an undue burden on organisations. QA does not involve doing or redoing any of the appraisal, evaluation or review work that is required elsewhere in the Code. QA reviews pieces of ex-ante appraisal, management, evaluation or review work done by others.

The Quality Assurance procedure is made up of five steps:

  1. Draw up inventories of projects/programmes at the different stages of the Project Life Cycle. It is expected that the Organisation’s Finance Unit is best placed to draw up this inventory. They may have to consult with others to ensure that they have the full picture on projects that are at the appraisal/planning stage i.e. have yet to incur expenditure. The person responsible for the Quality Assurance process should be satisfied that they have a full and complete inventory.
  2. The Organisation’s Finance Unit should publish summary information on its website of all procurements in excess of €2m, related to projects in progress or completed in the year under review. A new project may become a “project in progress” during the year under review if the procurement process is completed and a contract is signed. Department’s should also publish details of the website references where its agencies have placed information on procurements over €2m.
  3. Complete the checklists contained in this guidance document. Only one of each checklist per Department/Agency is required. Checklists are not required for each project/programme. The QA process is based on a sample.
  4. Carry out a more in-depth check on a small number of selected projects/programmes
  5. Complete a short summary report for the Department of Public Expenditure & Reform. The report, which will be generated as a matter of course through compliance with steps 1-4, involves minimum administrative burden and should be submitted by the end of February in respect of the previous calendar year.

Step 1 was not formally a part of the old process but it would have had to be completed in order to select the projects that were to be checked. The second step is new but should not be a significant burden as the inventory compiled as part of Step 1 will provide the master list. The set of checklists to be completed for Step 3 serve as prompts that allow organisations to self-assess how compliant they are at a general level and will allow them to identify areas that need attention. They can also measure progress from one year to another. Step 4 is the most detailed step. Organisations are required to look in detail at a small number of projects/areas of expenditure. The detailed checks will verify whether the assessments made when completing the checklists are accurate or not. Organisations may think that they are very compliant based on the initial surface checks but find that when the detailed checks are undertaken that the practice does not live up to the theory or vice-versa. This may prompt a revisit to the checklist assessments.

Responsibility for Quality Assurance

The Quality Assurance requirement rests mainly with the Sponsoring Agency. The questions in the self-assessment questionnaires have to be answered by the organisation that is responsible for the appraisal or management of an area of expenditure i.e. the Sponsoring Agency.

Departments are usually Sanctioning Authorities in respect of one part of their budget and Sponsoring Agencies for the remainder. Where the Department is the Sponsoring Agency it carries out Steps 1-5 of the QA process in respect of that part of its expenditure.

Departments in their role as Sanctioning Authorities must choose how they will implement the QA process for agencies that they fund. The Sanctioning Department could require those that they fund to complete the QA process and report it into them or the Sanctioning Authority could decide to take a hands-on role in part of the QA process particularly Step 4 where the Sanctioning Authority chooses to undertake one or more of the in-depth reviews.

Only Departments are required to send an annual QA report to the Department of Public Expenditure & Reform

Who Quality Assures compliance with the Public Spending Code?

The Quality Assurance process should be undertaken by internal staff that are as independent as possible of the areas responsible for appraisal, planning and implementation e.g. staff from the economic /evaluation units, financial management units, internal audit, staff from an evaluation unit in another Department or academics on a pro bono basis. The process should be led by a small group chaired at senior level (minimum PO).

Supplementary Quality Assurance by the CEEU

In addition to the quality assurance checks undertaken by Departments themselves, the Central Expenditure Evaluation Unit (CEEU) in the Department of Public Expenditure & Reform may undertake its own quality assurance checks from time to time. This exercise, which aims to promote a consistent approach to Quality Assurance and VFM enforcement across the public service, will not replicate the internal quality assurance process but may instead involve in-depth reviews of the processes followed for specific projects or programmes.

The five steps in the Quality Assurance procedure are described in more detail below.

1. Drawing up Inventories of projects/programmes

For Departments to know that they are compliant with the Public Spending Code they first need to be aware of the areas of expenditure to which the Code applies in their Department. The first step in the process is to draw up or update your inventories of:

(i) Expenditure being considered:

–          Capital projects that are or were under consideration during the year. These should be broken down by their anticipated cost (between €0.5 – €5m, between €5m – €20m, greater than €20m). Grant schemes for capital purposes should also be also included here. –          New Current expenditure programmes or significant extensions to existing programmes that will involve annual expenditure of €0.5m or more.

(ii) Expenditure being incurred

–          Capital Projects (> €0.5m) that are at the implementation stage

–          Capital Grant Schemes (> €0.5m) that are incurring expenditure

–          Current expenditure schemes or programmes (> €0.5m) that are incurring expenditure

(iii) Expenditure that has recently ended

–          Capital Projects (> €0.5m) that were completed in the year being reviewed

–          Capital Grant Schemes (> €0.5m) that were completed or were discontinued

–          Current expenditure schemes or programmes (>€0.5m) that were completed or were discontinued

It is expected that the Organisation’s Finance Unit is best placed to draw up this inventory. They may have to consult with others to ensure that they have the full picture on projects that are at the appraisal/planning stage i.e. have yet to incur expenditure. The person responsible, for the Quality Assurance process, should be satisfied that they have a full and complete inventory.

2. Publish summary information on your website of all procurements in excess of €2m, whether new, in progress or completed

Drawing from the inventory compiled or updated in Step 1 the organisation should publish, annually on its website, summary details of all procurements (capital and current) where the value exceeds €2m. This information should appear under the standard heading PROCUREMENTS/PROJECT PROGRESS on all Departmental websites. This information should be published concurrently with the quality assurance report i.e. by the end of February each year. The table below should be published for each project/procurement >€2m:

Project Details:

Year
Parent Department
Name of Contracting Body
Name of Project/Description

Procurement Details:

Advertisement Date:
Tender advertised in:
Awarded to:
EU contract award notice date
Contract Price:

Progress:

Start Date:
Expected Date of Completion per Contract:
Spend in Year under Review:
Cum Spend to end Year:
Projected final Cost:
Value of Contract variations:
Date of Completion:

Outputs:

Expected Output on completion (e.g. X km of road, No. units)
Output achieved to date (e.g. Y km of road, no. units)

There should be an entry for all new projects and projects still in progress. Completed projects feature for the last time in respect of the year that they were completed.

The presentation of this information can be in tabular or spreadsheet if that is more convenient.

3. Checklists to be completed in respect of the different stages

Step 3 involves completing a set of basic checklists covering all expenditure. These are high level checks that should be readily completed within each organisation. The objective of the exercise is to provide local and senior management, and the public more generally, with a self-assessment summary overview of how compliant the organisation is with the Public Spending Code. More in-depth checks are carried out as part of Step 4.

The first checklist captures obligations/good practice that apply to the organisation as a whole. Each of the remaining checklists listed below (checklists in the Appendices) might apply to a number of projects/areas of expenditure. Only one of these checklists is required for each organisation. Organisations are asked to estimate their compliance on each item on a 5 point scale (0. Not Done, 1. < 50% compliant, 2. 50-75% compliant, 3. > 75% compliant or 4. 100% Compliant). This self-assessed estimate of compliance can be based on an appropriate sample of the projects/areas of expenditure that are relevant to the checklist. The sample could be 5-10% of projects/programmes. The sample should rotate from year to year. Using a sample, to form a view on what should be included for the organisation in the Checklist answers, is in keeping with the intention that the QA process does not become over burdensome.

Checklist 1:                   General Obligations not specific to individual projects/programmes Checklist 2:                   Capital Projects or Capital Grant Schemes being considered Checklist 3:                   Current expenditure being considered

Checklist 4:                   Capital Expenditure being incurred

Checklist 5:                   Current Expenditure being incurred

Checklist 6:                   Capital Expenditure completed

Checklist 7:                   Current expenditure completed

4. Carry out a more in-depth check on a small number of selected projects/programmes

Parts 1 & 3 of the Public Spending Code Quality Assurance process will give an organisation a good overview of how compliant its processes are with the Public Spending Code. Quality Assurance Step 4 is about examining in more detail a small subset of its practices to see if the practices used are of a high standard. This step requires a higher level of analysis and judgement than previous steps in the QA process. It may for example involve drawing conclusions on whether the CBA used to appraise a proposal for a large project was satisfactory or not.

Selection of subset for closer examination:

Over a 3-5 year period every organisation should ensure that every stage of the project life-cycle and every scale of project is subject to a closer examination. In any given year this may involve looking at a couple of large projects at appraisal/planning, implementation or review stages or looking at a larger selection of smaller projects. Not every organisation has a large project every year so where large projects, in the year under review, are at the appraisal stage, implementation stage or have recently been completed it is opportune to select them for closer examination. In other years when large projects may not be a feature there is an opportunity to select a number of smaller scale projects. The value of the projects selected per annum, should be at least 5% of the total value of all projects in the inventory. This includes projects at the appraisal stage that have yet to incur expenditure. A subset of more than 5% may be needed for large organisations or because of the way that expenditure is divided a 5% sample would not give good coverage. To allow flexibility the minimum of 5% can be achieved as an average over a three year period e.g. 8%, 4%, 3%. The same projects should not be selected more than once in a three year period unless it is as a follow-up to serious deficiencies discovered previously.

Where there is a scheme that involves a large number of grants then it is the scheme itself that is the unit that is examined, not all of the individual grants i.e. it will not be necessary as part of this QA process to check 5% of all grants paid. The appraisal work on the scheme itself might be reviewed i.e. was there sufficient analysis to reach a conclusion that introducing the scheme was the best option to meet the objectives pursued? A small number of individual grants might be checked to confirm (i) that the conditions attaching to a grant matched the scheme design e.g. is this the subset of the population that we intended to target?, and (ii) that there was reasonable evidence that the scheme conditions were complied with.

This approach leaves organisations the greatest flexibility to cover the whole spectrum of projects and life-cycle phases over a number of years but also allows them to focus on large items at the most appropriate time.

What is expected of a more in-depth check?

Step 4 will look at a small subset and probe the quality of the work carried out. Step 3 above looks for basic indicators of compliance with the Public Spending Code i.e. if the project is over €20m, a CBA is required. Step 3 does not involve an assessment of whether or not the CBA is up to standard. Step 4, in contrast, looks in more detail at the quality of the appraisal, planning or implementation work done. This may mean:

–          examining a CBA for a large project,

–          an appraisal of a project under the €20m threshold,

–          looking at how the outputs and outcomes for a current expenditure programme are defined and whether the data exists for on-going monitoring and evaluation

–          examining how a large project was managed or

–          looking at a post-project review

and making a judgement on whether the CBA, post-project review etc. was of an acceptable standard. Adverse findings might be that the estimated number of users of the proposed project was too optimistic, that the value of the benefit was overstated or unfounded, that other realistic options were not considered, that all costs including lifetime costs were not included, that the outputs were not defined prior to implementation or that data was not gathered during implementation to allow ongoing monitoring etc.

Step 4 may highlight, that while processes are in place and the organisation looks very compliant as per the checklists, there are deficiencies when more detailed checks are made.

Step 4 is a in depth look at how the organisation complies with the Public Spending Code. It is different from a Value for Money Policy Review (VFMPR). Step 4 looks at how the decision was made initially, was it soundly based, was it well managed and reviewed in more depth when necessary.

The VFMPR looks at whether the intervention chosen worked or not or whether it was efficiently implemented. An organisation can do everything right as per the Code and come through this Quality Assurance check with a clean bill of health but an intervention it has chosen to fund may be shown in a VFMPR to have failed in spite of the best appraisal, planning and management. They are two separate exercises. If a VFMPR found that an intervention failed then continued compliance with the Public Spending Code should mean that the intervention is either abandoned or redesigned to address the deficiencies.

5. Complete a short report for Department of Public Expenditure & Reform.

The final step in the Quality Assurance process is the completion of a report to be submitted to the Department of Public Expenditure & Reform by the end of February in respect of the previous calendar year. The report should contain:

–          the inventory of project/programmes, current & capital as compiled by the organisation’s Finance unit;

–          the website reference where details of procurements over €2m are published;

–          completed checklists as per Step 3;

–          the Department’s judgement on the adequacy of the appraisal/planning, implementation or Review work that it examined as part of Step 4 and the reasons why the Department formed these judgements; and –          the Department’s proposals to remedy any inadequacies found during the Quality Assurance process.

This report should be certified by the Accounting Officer and published on the Department’s website.

The Quality Assurance Process should serve as an aid to each Department in its ongoing task of achieving the best value for money. The Quality Assurance process takes stock of how well an organisation does its job as steward of a significant block of public expenditure. Compiling and submitting a report will allow the Department of Public Expenditure & Reform to be of greater assistance in how it supports the achievement of this objective. It will also allow the D/PER and Departments generally to assess how appropriate this Quality Assurance Process is in practice and to make whatever adjustments may be required, in the context of the broader Consultation and Review procedures that are now integrated into the Public Spending Code itself.

CEEU Review of Compliance with Public Spending Code

The CEEU may make an annual assessment of each Department’s compliance with the Public Spending Code and may publish this assessment on its website. The assessment will be based on Departments’ Quality Assurance Reports, their record in completion of VFMs and any reviews that the CEEU itself conducts in Departments. Rather than focus only on deficiencies and shortcomings, it is important that instances of good practice be acknowledged, and that due credit should be given to Departments when they themselves identify and address deficiencies as part of the internal Quality Assurance process. Checklist 1:          – to be completed by all Departments

General Obligations not specific to individual projects/programmes   Self-Assessed Compliance Rating:  0 – 4 Comment/Action Required
Does the Department ensure, on an ongoing basis that appropriate people within the Department and in its agencies are aware of the requirements of the Public Spending Code?
Has training on the Public Spending Code been provided to relevant staff?
Has the Public Spending Code been adapted for the type of project/programme that your Department is responsible for? i.e. have adapted guidelines been developed?
Has the Department in its role as Sanctioning Authority satisfied itself that agencies that it funds comply with the Public Spending Code?
Have recommendations from previous Quality Assurance exercises (incl. old Spot-Checks) been disseminated, where appropriate, within the Department and to your agencies?
Have recommendations from previous Quality Assurance exercises been acted upon?
Has an annual Public Spending Code Quality Assurance Report been submitted to the Department of Public Expenditure & Reform?
Was the required sample subjected to a more in-depth Review i.e. as per Step 4 of the QA process
Has the Accounting Officer signed off on the information to be published to the website?

Self-Assessed Ratings: 0 – Not Done, 1 – < 50% compliant, 2 – 50-75% Compliant, 3 – > 75% Compliant, 4 – 100% Compliant
Checklist 2: – to be completed in respect of capital projects or capital programme/grant scheme that is or was under consideration in the past year.

Capital Expenditure being considered – Appraisal and Approval Self-Assessed Compliance Rating:  0 – 4 Comment/Action Required
Was a Preliminary Appraisal undertaken for all projects > €5m
Was an appropriate appraisal method used in respect of each capital project or capital programme/grant scheme?
Was a CBA completed for all projects exceeding €20m?
Were all Programmes with an annual value in excess of €30m and of 5 years or more duration subjected to an ex-ante evaluation?
Was an Approval in Principle granted by the Sanctioning Authority for all projects before they entered the Planning and Design Phase?
If a CBA was required was it submitted to the CEEU for their view?
Were the NDFA Consulted for projects costing more than €20m?
Were all projects that went forward for tender in line with the Approval in Principle and if not was the detailed appraisal revisited and a fresh Approval in Principle granted?
Was approval granted to proceed to tender?
Were Procurement Rules complied with?
Were State Aid rules checked for all supports?
Were the tenders received in line with the Approval in Principle in terms of cost and what is expected to be delivered?
Were Performance Indicators specified for each project/programme which will allow for the evaluation of its efficiency and effectiveness?
Have steps been put in place to gather the Performance Indicator data?

Self-Assessed Ratings: 0 – Not Done, 1 – < 50% compliant, 2 – 50-75% Compliant, 3 – > 75% Compliant, 4 – 100% Compliant
Checklist 3: – New Current expenditure or expansion of existing current expenditure under consideration

Current Expenditure being considered – Appraisal and Approval Self-Assessed Compliance Rating:  0 – 4 Comment/Action Required
Were objectives clearly set?
Are objectives measurable in quantitative terms?
Was an appropriate appraisal method used?
Was a business case prepared for new current expenditure?
Has an assessment of likely demand for the new scheme/scheme extension been estimated based on empirical evidence?
Was the required approval granted?
Has a sunset clause been set?
Has a date been set for the pilot evaluation?
Has the methodology and data collection requirements for the pilot evaluation been agreed at the outset of the scheme?
If outsourcing was involved were Procurement Rules complied with?
Were Performance Indicators specified for each new current expenditure proposal or expansion of existing current expenditure which will allow for the evaluation of its efficiency and effectiveness?
Have steps been put in place to gather the Performance Indicator data?

Self-Assessed Ratings: 0 – Not Done, 1 – < 50% compliant, 2 – 50-75% Compliant, 3 – > 75% Compliant, 4 – 100% Compliant
Checklist 4: – Complete if your organisation had capital projects/programmes that were incurring expenditure during the year under review.

Incurring Capital Expenditure Self-Assessed Compliance Rating:  0 – 4 Comment/Action Required
Was a contract signed and was it in line with the approval in principle?
If a construction or ICT project was the contract for a fixed price?
Are suitable management structures in place, commensurate with the scale of projects?
Did management boards/steering committees meet regularly as agreed?
Were Programme Co-ordinators appointed to co-ordinate implementation?
Were Project Managers, responsible for delivery, appointed and were the Project Managers at a suitable level for the scale of the project?
Were monitoring reports prepared regularly, showing implementation against plan, budget, timescales and quality?
Did the project keep within its financial budget and its time schedule?
Did budgets have to be adjusted?
Were decisions on changes to budgets or time schedules made promptly?
Did circumstances ever warrant questioning the viability of the project? (exceeding budget, lack of progress, changes in the external environment)

Y/N

If circumstances did warrant questioning the viability of a project was the project subjected to adequate examination?
If costs increased was approval received from the Sanctioning Authority?
Were any projects terminated because of deviations from the plan, the budget or because circumstances in the environment changed the need for the investment?

Y/N

For projects > €20m were quarterly reports on progress submitted to the MAC or Management Board and to the Minister?
Were prescribed annual tables on projects, completed or in progress and > €20m submitted to the Department of Public Expenditure & Reform?

Self-Assessed Ratings: 0 – Not Done, 1 – < 50% compliant, 2 – 50-75% Compliant, 3 – > 75% Compliant, 4 – 100% Compliant Checklist 5: – For Current Expenditure

Incurring Current Expenditure Self-Assessed Compliance Rating:  0 – 4 Comment/Action Required
Are there clear objectives for all areas of current expenditure?
Are outputs well defined?
Are outputs quantified on a regular basis?
Is there a method for monitoring efficiency on an ongoing basis?
Are outcomes well defined?
Are outcomes quantified on a regular basis?
Is there a method for monitoring effectiveness on an ongoing basis?
Have formal VFM evaluations or other evaluation been completed in the year under review?

Y/N

Are plans for new evaluations made in good time to ensure that they are completed in time to feed into the annual budget cycle?
Are unit costings compiled for performance monitoring?

Self-Assessed Ratings: 0 – Not Done, 1 – < 50% compliant, 2 – 50-75% Compliant, 3 – > 75% Compliant, 4 – 100% Compliant Checklist 6: – to be completed if capital projects were completed during the year or if capital programmes/grant schemes matured or were discontinued.

Capital Expenditure Completed   Self-Assessed Compliance Rating:  0 – 4 Comment/Action Required
Were the required post-project reviews carried out?
Was a post project review completed for all projects/programmes exceeding €20m?
If sufficient time has not elapsed to allow a proper assessment of benefits has a post project review been scheduled for a future date?
Were lessons learned from post-project reviews disseminated within the Sponsoring Agency and to the Sanctioning Authority?
Were changes made to the Sponsoring Agencies practices in light of lessons learned from post-project reviews?
Was project review carried out by staffing resources independent of project implementation?

Self-Assessed Ratings: 0 – Not Done, 1 – < 50% compliant, 2 – 50-75% Compliant, 3 – > 75% Compliant, 4 – 100% Compliant Checklist 7: – to be completed if current expenditure programmes that reached the end of their planned timeframe during the year or were discontinued.

Current Expenditure that (i) reached the end of its planned timeframe  or (ii) Was discontinued Self-Assessed Compliance Rating:  0 – 4 Comment/Action Required
Were reviews carried out of, current expenditure programmes that matured during the year or were discontinued?
Did those reviews reach conclusions on whether the programmes were effective?
Did those reviews reach conclusions on whether the programmes were efficient?
Have the conclusions reached been taken into account in related areas of expenditure?
Were any programmes discontinued following a review of a current expenditure programme?
Was the review commenced and completed within a period of 6 months?