How to show impact

 

What is impact and how do you show it?

Impact is any affect arising from an intervention. This can include shorter-term outcomes as well as any broader and longer-term affects.  Showing impact is about demonstrating changing risk, the likelihood of someone reoffending and generally improving outcomes. There are leading indicators linked to the likelihood of reducing risk and behaviour change. Additionally, there are lag indicators, such as the outcome stars http://www.outcomesstar.org.uk/. 

Key evidence-based dimensions of reoffending impact include:

  • Assessments
  • Accommodation
  • Education training and employability
  • Relationships
  • Lifestyle and associates
  • Drug misuse
  • Alcohol misuse
  • Thinking and behaviour
  • Attitudes

 

What do you need to do and think about?

Providers need to consider impact from the perspective of the service user and how services/interventions reduce reoffending. A second consideration is the impact on the public and ensuring they are protected.

Impact should look at the “system” and/or way the person may move through your services/interventions. Your services/interventions can be thought of as a “programme” of activities which have four basic components that need to be considered:

  • At what point along the offender pathway does the service user enter the intervention
  • Upon entry into the intervention, how are service users’ needs assessed
  • What are the outcomes that the intervention hopes to achieve for this individual
  • What measures will be used to monitor progress towards reaching the desired outcomes

The critical point is whether the service user has now formed supportive relationships and/or is more or less likely to change their behaviour.

The amount of time and resource you put into the person is the direct cost. That cost can be spread over a number of service users. This is an important consideration when assessing value for money and return on investment.

 

What data sources can you use?

There are a number of data sources that can be accessed to provide valuable data for measuring impact, such as:

  • Internally produced data: Data collected by the organisation for administrative purposes that includes details on individual service users, their level of participation in the intervention and their progress against standardised measures. Cost data is also important for understanding value for money and return on investment.
  • Data.gov: Is a searchable website that provides raw data from all central government departments and a number of other public sector bodies and local authorities. It provides a number of datasets such as reoffending data and population data (ONS).
  • Justice Data Lab: The Justice Data Lab gives organisations working with offenders access to central reoffending data. The service provides this information to help organisations assess the impact of their work on reducing reoffending. Organisations complete a ‘data upload template’ which details information on service users and the services provided. The template is submitted to the Justice Data Lab and if the information meets the minimum criteria the Justice Data Lab will issue a report on the data.  These minimum requirements include the need to provide details on 60 individuals and for the template to be sent by secure email (gsi or CJSM). This analysis assesses the impact of the programme against various measures and provides results in a clear and easily understandable format. The results from the Justice Data Lab are also published to enable all those working with offenders to see clearly what works and to help create a culture of best practice and transparency.
  • Justicemeasures.co.uk: This site contains data (up to 2011) on a number of justice measures including the adult reoffending rate, cost per community order, detected crime rate per 1,000, detection rate, and guilty and caution outcomes per 100,000  across different localities in England and Wales.

 

What are the potential data issues?

To successfully measure impact, the emphasis should be on getting some simple measures right and focusing on a programmatic view of the interventions. That said, there are a number of considerations that need to be considered when selecting the data that will be collected for specific measures.

  • Identifying trusted data sources – how will the reliability of data sources be assessed?
  • Quality – Will the data be sufficiently timely to provide a true indication of impact?
  • Sample size – is the sample size sufficiently large to generate robust findings?
  • Analysis/manipulation of data – what methods will be used to analyse the data and calculate the impact of the intervention?
  • Data security – how will your data be stored?

There is also the need to bear in mind the administrative burden of data collection to measure impact. The process may start as a simple process but as the burden grows there is a need to find simple solutions to leverage technology. Organisations may be able to access funds and support with certain aspects of achieving collaboration and analysis in a secure environment. This includes initiatives such as local partnerships’ technology funds.