1. Essence | 2.
Aproaches | 3. Process | 4.
Measuring outcomes | 5. Paradoxes
Examples | 7.
Jargon | 8.
Checklist | 9.
Practice tips | 10.
The process of evaluation
Some of the essential
elements of the process of evaluation are:
Clarity of purpose and questions
Dialogue and reflection strategies
collation and analysis
1. Clarity of purpose and questions
The essence of an evaluation process is
- Asking a question
- Identifying the information needed to answer the question
- Designing and testing a method for collecting the information
- Collecting the information
- Analysing the information
- Determining the answer to the question
- Using the answer.
Having a clear purpose is an essential ingredient of a good evaluation.
Having a clear purpose will:
identify the users of the evaluation
Who will use the evaluation to achieve this purpose?
- Help identify relevant questions for the evaluation to address
What are the questions that these users will find useful?
- Help identify the extent of the evaluation
How extensive does the evaluation need to be for this purpose?
- Help identify what an answer for the evaluation would look like?
What would those using the evaluation need to read for their questions
to be answered so they can use the answers?
- Help focus the evaluation on using the answer
How can we facilitate those using the evaluation to
Common evaluation purposes include:
a decision about whether to continue, expand or scale back a service
- Refocusing a service, ie, changing the nature or the service or its
programs to better
- Improving the quality of a service
- Ensuring funding body requirements are met so the service can continue
- Developing models of best practice
and reflection strategies to give meaning
Dialogue and reflection is a core part of any human service evaluation
Dialogue and reflections should be embedded in all phases of the evaluation
process from the initial description of the purpose and questions
through to the data gathering and analysis and the asking of the
So what? What does it mean?
Some of the spaces that can be created for dialogue and reflection
- Steering committee meetings
- Staff meetings
- Case conferences
- Quality committees.
Some of the questions and rules of thumb to keep in mind are:
Have all the stakeholders been involved in processes of dialogue
and reflection in identifying
- The purpose of the evaluation
- The values that will underpin it
- The questions that will be asked
- What the findings mean.
Data does not speak for itself. Useful questions for
In the findings from the evaluation we have noticed such
and such.... Is this good or not good?
Who are the stakeholders in this evaluation process?
What are the opportunities for them to dialogue and reflect on the evaluation
purpose, process, findings, interpretations and recommendations?
Data gathering and data analysis are tightly integrated processes. They
are separated here for convenience. Some ways of gathering data are:
Being a participant observer is often a useful evaluation strategy.
Keep your eyes and ears open. Reflect on what you see and hear.
Photos and video
Photos and video of settings, service processes, surroundings,
Listening and dialogue
Formal and informal interviews are a important way of gaining
information for use in evaluation processes. Different groups
can be interviewed
- Service users
- Other service providers
- People in the community.
There are many different approaches to interviews, for example
informal conversational interviews, standardised open-ended
Groups of people can be asked to a meeting where they can talk
about their experiences of the service, what was helpful and
This can be run
by a person not associated with the service so that the comments
are more likely to be frank and honest.
Case studies provide the richness of what is happening in the
lives of people and what the service has meant to them. Some
What does the service mean for the people who benefit from
it? What are some of the real life complexities of providing
do the beneficiaries see the service has had in their life?
Service users telling their stories is usually preferable
to staff telling stories of service users.
Examples of documents that can be gathered include:
Plans including, strategic plans, work plans, case plans
Counting and measuring
Most services have some kind of hard data information systems.
This data usually meets multiple purposes within
the service or its auspicing
can be drawn on and analysed in evaluation processes.
Questionnaires are a useful tool especially because
they are cost effective way of gaining views from
are readily collatable and analysable. Questionnaires
can be used in many
- Questionnaires at the end of group sessions
- Staff questionnaires
follow up questionnaires
to other service providers
- Advisory Committee questionnaires
- Questionnaires for the worker to help focus the
Services (managers and management committees) often
need one page of key numbers for regular review.
need to cover the
- Clients and client characteristics
- Inputs, e.g. staff and volunteer time, $
- Ratios, e.g. $/hour service
Census and other population data
Painting a profile of the local community and
comparing your service users with the profile
of the local
very useful in
identifying who you
are serving and who you are not serving.
Individual, family and social functioning indicators
There are many indicators that can be used.
Indicators are often developed by specialists
and made available
For example many
questionnaires have been developed to measure
self-esteem and family functioning; in recent
years there has been work done on indicators
social well-being. Several measures of social
What ways do you currently gather data?
What other ways are there?
Data collation and analysis
The distinction between qualitative and quantitative data is not
as clear cut as is often thought. Quantitative data is often
data and qualitative data can be quantitative data. However it is still
a useful ‘common
sense’ distinction when thinking about how to collate, analyse
and report on data.
For further details
Qualitative data analysis
Quantitative data analysis
The first step in an evaluation process is identifying the purpose of the
evaluation. The report needs to be useful for this purpose.
There are many different kinds of evaluation reports. For example:
memo to a manager
- a report to the staff
- a report to clients and other stakeholders
report to a funding body
- an internal report
- a public report
- an interim report
- a final report
While there is no one way to write an evaluation report. It is useful
to think about the following areas and how they will best be communicated:
Purpose - what was the purpose of the evaluation?
The service - what is being evaluated?
Process - what were the steps in the evaluation process?
Description of findings - what data was gathered and what was found?
Interpretation - What do the findings mean? What issues are emerging?
Valuing - what is the value of what has been happening?
Recommendations - what should happen now?
It is useful to write the table of contents of the evaluation report
before the evaluation is begun and also to write the number
of pages each section
of the report will have.
This process helps focus the evaluation in relation to:
- What you are looking for?
- How you are going to communicate it?
- How extensive it needs to be?
Who is the report for?
What is the table of contents of the report?
How long will the report be?
How long will each section/chapter of the report be?