Evaluation and Review

Evaluation is the systematic and objective assessment of a program, project or policy and plays a key role in decision-making by helping us understand whether the program, project or policy is relevant (are we doing the right thing), effective (are the objectives being achieved), efficient (are the objectives being achieved economically), having impact (are real differences being made) and sustainable.

A comprehensive evaluation of an initiative will include elements of process, outcome, and economic evaluation. Process evaluation examines implementation and delivery, outcome evaluation is concerned with whether initiatives delivered intended change, and economic evaluation is interested in identifying and measuring impacts relative to costs so as to provide an assessment of value for money and/or net social benefit.

Reviews are also an assessment of a program, project or policy but are typically performed periodically or on an ad hoc basis. Often reviews will be undertaken to assess progress (to inform improvement) or to collect sufficient data in a timely manner (to inform decision-making).

If you are planning and/or commissioning an evaluation or review of a program or project, check out our selection of useful Resources for Evaluation and Review.

Our work in this area includes:

• Economic evaluation

  • Cost-benefit analysis

  • Cost-effectiveness analysis

• Impact assessment/evaluation

• Delivery of training workshops for the Australian Evaluation Society on economic evaluation and cost-benefit analysis.

• Planning data collection for economic/impact evaluation before program, project or policy implementation

• Agency and program reviews

• Post-implementation reviews

• Development of benefits databases

• Benefits realisation

• Value for money reviews

Would you like to discuss how we can help you?

Examples of recent work include:

NSW National Parks and Wildlife Service | Economic Evaluation of Enhanced Bushfire Management Program 2017-2022

NSW National Parks and Wildlife Service (NPWS) manages 893 NSW national parks and reserves, covering over 7 million hectares of land.

The Enhanced Bushfire Management Program supports work to improve bushfire response, increase hazard reduction and develop research to reduce the risk of bushfire across New South Wales.

As part of an evaluation consortium, we undertook the ex-post economic component of the summative evaluation of the Enhanced Bushfire Management Program 2017-2022 and made recommendations for enhancing economic evaluation in the next phase of the program.

For more information about NPWS and its work in managing fire in park and reserves visit NSW NPWS.

NSW Department of Planning and Environment | Evaluation of Fast-Tracked Assessment Program

The Fast-Tracked Assessment Program (the Program) was established at the beginning of the COVID-19 pandemic to accelerate the assessment and determination of ‘shovel ready’ projects that would inject investment into the NSW economy and keep people in jobs during and after the COVID-19 pandemic.

The program included projects that required State planning approval such as State Significant Developments, State Significant Infrastructure, major projects, and planning proposals.

101 projects were selected for fast-tracking with the potential for 50,000 jobs, 25,000 new dwellings, investment value of $25 billion, and 400 hectares of open space.

We undertook an evaluation of the Program which focused on the costs, benefits and outcomes.

Destination NSW | Ex-Post Cost-Benefit Analysis

Destination NSW is the lead government agency for the NSW tourism and major events sectors, influencing and shaping the NSW visitor economy through strategic research, policy and thought leadership, stakeholder engagement and consultation, commercial partnerships, and investment in visitor experiences, business support, marketing, and events. 

We have undertaken ex-ante and ex-post cost-benefit analyses of investments in marketing campaigns and major events, and helped develop a methodology for undertaking cost-benefit analysis that can be applied to marketing campaigns and major events on either an ex-ante or ex-post basis.

A performance audit of Destination NSW’s support for major events favorably commented that ‘Destination NSW has recently incorporated cost-benefit analysis principles into its event assessment methodology. This approach has the potential to further increase the rigour of event assessment because the cost-benefit analysis methodology considers a range of costs and benefits beyond the direct economic impact’.

NSW Rural Fire Service | Post-Acquisition Financial Analysis of Aviation Assets

To inform a strategic review that NSW Rural Fire Service was undertaking in relation to its aviation needs, we undertook independent financial analysis of all aircraft that had been acquired.

Our analysis included consideration of costs and benefits, and assessment of whether the acquisition and ownership had delivered value for money.

NSW Rural Fire Service has since gone on to expand its fleet of aircraft to eleven.

Achieving value for money is a primary consideration in every procurement. Information and guidance on assessing value for money in procurement can be found here.

Resources for Evaluation and Review

If you are planning and/or commissioning an evaluation of a program or project, useful resources include:

Evaluation Communities of Practice

• BetterEvaluation - an international collaboration to improve the practice and theory of evaluation.

• Australian Evaluation Society - a member based organisation which exists to improve the theory, practice and use of evaluation in Australia for people involved in evaluation.

• Australian Centre for Evaluation - a quick guide to the Australian Government’s plans from 2023–24 to establish a central evaluation function within Treasury to provide leadership and improve evaluation capability across Government, including support to agencies and leading a small number of flagship evaluations each year.

• American Evaluation Association - professional association of evaluators devoted to the application and exploration of evaluation as a profession, and with more than 5,000 members representing all 50 states in the United States as well as more than 80 foreign countries.

• European Evaluation Society - promoting the theory, practice, and utilisation of high-quality evaluation in Europe and beyond.

Australian States & Territories Guidance on Evaluation

• NSW Evaluation Toolkit - a resource designed for NSW public sector managers who are responsible for a program evaluation.

• NSW Treasury TPG22-22 Policy and Guidelines: Evaluation (2023) - sets out mandatory requirements, recommendations and guidance for NSW General Government Sector agencies and other government entities to plan for and conduct the evaluation of policies, projects, regulations, and programs. TPG22-22 is supported by online workbooks and resources designed to be used flexibly as relevant to the initiative, the context in which it is being evaluated, and the resources allocated to the evaluation.

• NSW Government Guide to Economic Evaluation and Ex-Post CBA for TfNSW Evaluation Programs (2024) - sets out the requirements, process for managing an Economic Evaluation, and high-level guidance for an ex-post CBA, for all suitable projects / programs nominated in the TfNSW Evaluation Schedule.

• NSW Government Technical Note on Ex-Post Cost-Benefit Analysis (TPG22-22) - supplementary guidance for undertaking ex-post cost-benefit analysis when an initiative is underway (interim or ‘in media res’ ex-post CBA) or completed (final ex-post CBA) to assess the net social benefits of an implemented initiative.

• NSW Government Grants Administration Guide (2022) - provides an overview of the grants administration process in NSW including evaluation requirements.

• Queensland Government Program Evaluation Guidelines (2020) - broad principles that are expected to underpin the planning and implementation of evaluations for programs funded by the Queensland Government. Includes supplementary information sheet on economic evaluation.

• Western Australia Program Evaluation Guide (2020) - a resource developed for Western Australian Government agencies undertaking the evaluation of government funded programs.

• Northern Territory Government Program Evaluation Framework and Toolkit (2022) - framework and toolkit to be used by Territory Government agencies to help plan, commission and use evaluations. Also provides useful guidance for Territory Government service delivery partners and external evaluators of Territory Government programs.

• ACT Government Evaluation Policy and Guidelines (2010) - a resource designed to support ACT Government agencies in planning for and executing evaluations.

Commonwealth Guidance on Evaluation

• Evaluation in the Commonwealth - overview of evaluation concepts and approaches

• Commonwealth Evaluation Toolkit - templates, tools, and other resources, to help plan and conduct an evaluation.

• Australian Government Post-Implementation Reviews (2020) - a guidance note to support Australian Government agencies that need to undertake a post-implementation review of any regulatory changes that have a substantial or widespread impact on the economy. Did the implemented policy operate as intended and is it effectively and efficiently meeting the Government’s objectives in addressing the original problem?

Guidance on Health Economic Evaluation

• Commissioning Economic Evaluations: A Guide (2017) - guidance provided by NSW Health for commissioning economic evaluations of health programs, particularly those in population health.

• Reviewing Economic Evaluations: A Checklist (2021) - guidance provided by NSW Health that focuses on the core principles of economic evaluation and how each can be used in appraising economic methods found in a range of documents such as peer-reviewed journal articles, grey literature, project proposals and reports.

• Consolidated Health Economic Evaluation Reporting (CHEERS) Standards (2013) - a consolidation and update of previous health economic evaluation guidelines efforts into one current, useful reporting guidance. The primary audiences for the CHEERS statement are researchers reporting economic evaluations and the editors and peer reviewers assessing them for publication.

Guidance on Value for Money

• Evaluating Methods for Assessing Value for Money (2013) - paper developed by a Working Group convened through the Australian Evaluation Society which provides information on the topic of Value for Money and presents a range of methods for assessing Value for Money.

• On Efficiency and Effectiveness: some definitions (2013) - research note from the Australian Productivity Commission setting out how it defines efficiency and effectiveness along with related concepts such as cost-effectiveness and productivity.

• Value for Money - NSW Government guidance on assessing value for money in procurement and evaluation.

UK & NZ Guidance on Evaluation

• UK Government: The Magenta Book (2020) - HM Treasury guidance on what to consider when designing an evaluation.

• New Zealand Government: making sense of evaluation: A handbook for everyone (2017) - handbook has been developed to provide people new to evaluation with an overview of evaluation concepts and processes so that they are sufficiently equipped to work in this space (often, but not always, with a professional evaluator).

Guidance on Impact Assessment/Evaluation

• CSIRO: Impact Evaluation Guide (2020) - articulates a common framework for all CSIRO impact evaluations. The Guide has been publicly released as CSIRO believes that it is beneficial for the broader innovation system for Australia’s publicly funded research organisations to use a common approach to the assessment of the outcomes and impacts of their research.

• ACIAR: Guidelines for assessing the impacts of ACIAR's research activities (2008) - provides a common approach for making three types of assessment of the impact of ACIAR investments - quick assessment of expected impact, adoption studies (observed and anticipated impacts), and full impact assessments. ACIAR has a long history of undertaking impact assessments. It’s well-designed frameworks and approach can be applied more widely to other research activities.

• Interact: Approaches, ToR and methods for impact evaluation (2016) - overview of the different approaches, methods and techniques for impact evaluation.

Wellbeing Frameworks & Outcome Budgeting

• Australian Government: Measuring What Matters (2023) - Australia’s first ever national wellbeing framework which sets out the factors that are important to Australians’ individual and collective wellbeing across all phases of life in five broad themes – healthy, secure, sustainable, cohesive and prosperous.

• NSW Government TPP18-09 Outcome Budgeting (2018) - provides guidance on the first two elements of the Outcome Budgeting Framework (i) developing and maintaining outcome structures and (ii) financial and non-financial requirements for performance reporting purposes.

Research Sources

• Australian Institute of Family Studies - Australian Government statutory agency established in 1980 under the Family Law Act 1975 with a mission is to conduct high-quality, impartial research into the wellbeing of Australian families, to inform government policy and promote evidence-based practice in the family services sector. Also incorporates the Australian Gambling Research Centre (AGRC), which was established under the Gambling Measures Act 2012.

• APO (Australian Policy Online) - an open access evidence platform, making public policy research and resources accessible and useable.