Contents | 1. Essence | 2. Aproaches | 3. Process | 4. Measuring outcomes | 5. Paradoxes
6. Examples | 7. Jargon | 8. Checklist | 9. Practice tips | 10. Resources

3. The process of evaluation

Some of the essential elements of the process of evaluation are:

1. Clarity of purpose and questions
2. Dialogue and reflection strategies
3. Data gathering strategies
4. Data collation and analysis
5. Reporting

 1. Clarity of purpose and questions

The essence of an evaluation process is

• Having a purpose
• Asking a question
• Identifying the information needed to answer the question
• Designing and testing a method for collecting the information
• Collecting the information
• Analysing the information
• Determining the answer to the question
• Using the answer.

Having a clear purpose is an essential ingredient of a good evaluation.

Having a clear purpose will:

• Help identify the users of the evaluation
        Who will use the evaluation to achieve this purpose?
• Help identify relevant questions for the evaluation to address
        What are the questions that these users will find useful?
• Help identify the extent of the evaluation
        How extensive does the evaluation need to be for this purpose?
• Help identify what an answer for the evaluation would look like?
         What would those using the evaluation need to read for their questions to be answered so they can use the answers?
• Help focus the evaluation on using the answer
         How can we facilitate those using the evaluation to use it?

Common evaluation purposes include:

• Making a decision about whether to continue, expand or scale back a service
• Refocusing a service, ie, changing the nature or the service or its programs to better meet client and community needs
• Improving the quality of a service
• Ensuring funding body requirements are met so the service can continue to be funded
• Developing models of best practice

2. Dialogue and reflection strategies to give meaning

Dialogue and reflection is a core part of any human service evaluation process.

Dialogue and reflections should be embedded in all phases of the evaluation process from the initial description of the purpose and questions through to the data gathering and analysis and the asking of the big questions: So what? What does it mean?

Some of the spaces that can be created for dialogue and reflection include:

• Steering committee meetings
• Staff meetings
• Case conferences
• Quality committees.

Some of the questions and rules of thumb to keep in mind are:

Have all the stakeholders been involved in processes of dialogue and reflection in identifying

• The purpose of the evaluation
• The values that will underpin it
• The questions that will be asked
• What the findings mean.

Data does not speak for itself. Useful questions for stakeholders are.
In the findings from the evaluation we have noticed such and such.... Is this good or not good?

Key questions
Who are the stakeholders in this evaluation process?
What are the opportunities for them to dialogue and reflect on the evaluation purpose, process, findings, interpretations and recommendations?

3. Data gathering

Data gathering and data analysis are tightly integrated processes. They are separated here for convenience. Some ways of gathering data are:

Observing

Observation
Being a participant observer is often a useful evaluation strategy. Keep your eyes and ears open. Reflect on what you see and hear.

Photos and video
Photos and video of settings, service processes, surroundings, etc.

Listening and dialogue

Interviews
Formal and informal interviews are a important way of gaining information for use in evaluation processes. Different groups can be interviewed including:

• Service users
• Staff
• Other service providers
• People in the community.

There are many different approaches to interviews, for example informal conversational interviews, standardised open-ended interviews, standardised closed question interviews.

Focus groups
Groups of people can be asked to a meeting where they can talk about their experiences of the service, what was helpful and not helpful. This can be run by a person not associated with the service so that the comments are more likely to be frank and honest.

Case studies
Case studies provide the richness of what is happening in the lives of people and what the service has meant to them. Some questions case studies can explore are:

What does the service mean for the people who benefit from it? What are some of the real life complexities of providing services to clients? What impact do the beneficiaries see the service has had in their life?

Service users telling their stories is usually preferable to staff telling stories of service users.

Documents

Examples of documents that can be gathered include:
Plans including, strategic plans, work plans, case plans
Organisational policy
Client records

 Counting and measuring

Information systems
Most services have some kind of hard data information systems. This data usually meets multiple purposes within the service or its auspicing organisation. It can be drawn on and analysed in evaluation processes.

Questionnaires
Questionnaires are a useful tool especially because they are cost effective way of gaining views from a large number of people in formats that are readily collatable and analysable. Questionnaires can be used in many different situations:

• Questionnaires at the end of group sessions
• Staff questionnaires
• Client follow up questionnaires
• Questionnaires to other service providers
• Advisory Committee questionnaires
• Questionnaires for the worker to help focus the work.

Key numbers
Services (managers and management committees) often need one page of key numbers for regular review. In human services the numbers probably need to cover the following areas:

• Clients and client characteristics
• Services/activities
• Processes
• Outcomes
• Inputs, e.g. staff and volunteer time, $
• Ratios, e.g. $/hour service

Census and other population data
Painting a profile of the local community and comparing your service users with the profile of the local community is very useful in identifying who you are serving and who you are not serving.

Individual, family and social functioning indicators
There are many indicators that can be used. Indicators are often developed by specialists and made available for others to use. For example many questionnaires have been developed to measure self-esteem and family functioning; in recent years there has been work done on indicators of social well-being. Several measures of social capital have been developed.

Key questions
What ways do you currently gather data?
What other ways are there?

4. Data collation and analysis

The distinction between qualitative and quantitative data is not as clear cut as is often thought. Quantitative data is often also qualitative data and qualitative data can be quantitative data. However it is still a useful ‘common sense’ distinction when thinking about how to collate, analyse and report on data.

For further details
      Qualitative data analysis
      Quantitative data analysis

5. Reporting

The first step in an evaluation process is identifying the purpose of the evaluation. The report needs to be useful for this purpose.
There are many different kinds of evaluation reports. For example:

• a memo to a manager
• a report to the staff
• a report to clients and other stakeholders
• a report to a funding body
• an internal report
• a public report
• an interim report
• a final report

While there is no one way to write an evaluation report. It is useful to think about the following areas and how they will best be communicated:

Purpose - what was the purpose of the evaluation?
The service - what is being evaluated?
Process - what were the steps in the evaluation process?
Description of findings - what data was gathered and what was found?
Interpretation - What do the findings mean? What issues are emerging?
Valuing - what is the value of what has been happening?
Recommendations - what should happen now?

It is useful to write the table of contents of the evaluation report before the evaluation is begun and also to write the number of pages each section of the report will have.

This process helps focus the evaluation in relation to:

• What you are looking for?
• How you are going to communicate it?
• How extensive it needs to be?

Key questions
Who is the report for?
What is the table of contents of the report?
How long will the report be?
How long will each section/chapter of the report be?