Contents | 1. Essence | 2. Aproaches | 3. Process | 4. Measuring outcomes | 5. Paradoxes
6. Examples | 7. Jargon | 8. Checklist | 9. Practice tips | 10. Resources

2. Approaches to evaluating human services and community development

There are many ways of conceptualising evaluation. If one looked at the human service and community development evaluations that have been done over the past decade there are eight typical approaches being used:

1. Program evaluation: Evaluating the program (eg the Community Services Grants Program (CSGP), the Families First Program, the Communities for Children program and the Home and Community Care (HACC) Program.)

2. Program monitoring and review: Gathering data to monitor and review what is happening in the program (eg the CSGP funding program or the Families First funding program and the HACC program)

3. Service network capacity evaluation: Evaluating the capacity of the network of services in a particular area or region to effectively meet community needs.

4. Agency, service provider or service periodic evaluation (eg evaluating the local family service, child care centre or aged service through a one off or periodic evaluation process)

5. Agency, service provider or service internal ongoing evaluation (eg continuous improvement within a local family service, child care centre or aged service)

6. Client focussed evaluation: Evaluating the work with individual clients (individuals and families).

7. Community focussed evaluation: Evaluating the work with the community.

8. Policy evaluation: Evaluating public or organisational policy, for example, evaluating immigration policy.

Evaluation from one perspective may be used in another. For example, client focussed evaluation may be one element in program evaluation. Agency or service provider focussed evaluation may be one element in service network capacity evaluation.

The framework above is primarily based on what is being evaluated (ie, program, agency, service, client, community, policy).

Categories 1 & 2 and 4 & 5 are also distinguished by purpose. Category 2 is about monitoring and review (for a program) and Category 5 about service improvement. These two sub-categories have been singled out because considerable investment and energy is invested in those two approaches by governments and agencies.

In a more comprehensive description we might include the following five purposes for each category of what (ie, program, agency, service, client, community, policy):

• Are we doing the right things the best way? - Direction
• Are we clear about what we are doing? - Clarity
• Do we know what’s happening? - Monitoring
• Can we improve what we are doing? - Improvement
• Are we making a difference? - Impact

The eight typical approaches above are described in more detail below.

 1. Program evaluation

Questions
Program evaluation focusses on the program as a whole and will ask questions like:
What is the impact of the program?
Is the program achieving its outcomes?
Are the outcomes the right outcomes?
Has the program been implemented as planned?
How could the program be improved?
For example what is the impact of the CSGP program? What is the impact of the Families First program? Has the Families First program been implemented as planned?

Strategies
Program evaluation often involves a specialist evaluation team with specific terms of reference from the funding body, for example the Families First evaluation program and the Communities for Children evaluation program.

2. Program monitoring and review
Program monitoring and review focusses on the program as a whole.

Questions
Program monitoring and review answers questions about what is happening in the program, such as:
How many people are using the program?
Are they in the target group?
How much service is being provided?
What are the costs per unit of service?

Strategies
Strategies typically used for program monitoring and review include:
Service agreements with conditions for service providers to provide service data
Service providers' yearly plans being submitted to the funding body.
Standardised data collection systems including minimum data sets.
For example, the CSGP has a service framework, service agreements and work is underway for standardised data collection. The Supported Accommodation Assistance Program incorporates standardised client data collection for monitoring purpose.

3. Service network capacity evaluation

Questions
Service network capacity evaluation will ask questions like:
Does the network of services in this area or region have the capacity to effectively deliver services to meet the needs of the community?
Are there effective referral processes between agencies?
Do the agencies in the network understand each others roles?
Do the agencies in the network support each other in their roles?
Is the network of services extensive and comprehensive enough to meet the needs in the community?

Strategies
Evaluation of service network capacity often involves a specialist evaluation team with specific terms of reference from a funding body. For example the Families First evaluation program has a review of service network capacity as one component of the program evaluation.

4. Agency, service provider or service focussed periodic evaluation
Agency, service provider or service focussed periodic or one off evaluations focus of the service within a particular organisation or at a particular location rather than the program as a whole.

Questions
Questions asked are similar to the kinds of questions in program evaluation and program monitoring and review but asked about a particular provider. Typical questions are:
Is the service achieving its aims and objectives?
Is the service well run?
How could the service be improved?
How many people are using the service?
Are they in the target group?
How much service is being provided?
What are the costs per unit of service?

Strategies
There are two kinds of strategies that are typically used for periodic external evaluation of services.
An external evaluation consultant or team being appointed and given specific terms of reference. The consultant(s) would usually work to a steering committee.
Accreditation processes - where agreed standards and indicators have been agreed for the program and a system established for identifying whether or not service providers meet the agreed standards and indicators.

5. Agency, service provider or service focussed internal ongoing evaluation
Internal ongoing evaluation focusses on the services at a particular location (not the program as a whole). It is developed and undertaken by service providers.

Questions
Each service needs to be able to answer questions such as: Do we know enough to know whether or not we are providing a quality service? Do we know enough to improve the quality of our service?
Specific questions include:
What is the profile of our local community?
Who requests what information and services?
What services have been provided?
To whom?
Did the service make a difference? In whose eyes? How do we know?
What are clients views of the services?
Is the service provision working collaboratively with other agencies?
What is the community perception of the service?
What community consultation work and inter-agency collaboration is undertaken?
How can the service be improved?

Strategies
Each local service needs to put in place a wide range of strategies to ensure they can answer the above questions. Ongoing strategies include: an organisational manual, supervision of staff, client feedback mechanisms, staff feedback mechanisms, focus groups, peer review, etc

6. Client focussed evaluation
Client focussed evaluations may be part of an agency focussed evaluation.

Questions
Questions include:
Who requests what information and services?
What services have been provided?
To whom?
Did the service make a difference? In whose eyes? How do we know?
What are clients views of the services?
How can the service be improved?

Strategies
Strategies include the use of case assessment, planning and review tools such as: referral form, assessment form; staff supervision; case conferences; comparing clients with the profile of the community; analysis of service processes, for example, analysing the referral and assessment process to ensure that all those entitled to receive service have an equal chance of being referred and assessed.

7. Community focussed evaluation
Community focussed evaluations focus on the community.

Questions
Questions include:
Who is the community?
What is the community’s story?
What community development processes have been undertaken?
Did they make a difference? In whose eyes? How do we know?
What are community’s views of what has happened?
How can the community be further developed?

Strategies
Strategies include: community consultations, community surveys, focus groups, demographic and other population data collection and analysis, supervision of community development workers, work collaboratively with other services, a community profile that identifies needs and priorities and an implementation plan for meeting these needs.

8. Policy evaluation

Questions
Some of the key questions that it may be useful to ask in evaluating policy are:
What is the policy and what is the background behind the policy?
What problem was the policy trying to solve?
On what values is the policy based?
What processes were used in developing the policy?
Who was consulted in the process of developing the policy?
Who is the legitimate authority making the policy?
Who benefits from the policy (in theory)?
Who is disadvantaged by the policy (in theory)?
How will the policy be implemented?
Who are the winners and losers in practice when the policy is implemented?
How will the relevant people find out about the policy?

Strategies
Strategies usually include secondary research strategies and interviews and focus groups with stakeholders.

Key questions
In relation to your service, who is evaluating what?
Who is using what evaluation approaches?
What approach are you using?