3. How can we evaluate family support services?

A model of Family Support services

To evaluate family support services it is useful to have a model of family support.

Here is a picture of a family support service:

It is an oversimplification, but it helps highlight some aspects of a family support human service process:

A. There is a context of community services.
B. The client is part of a social fabric of family, neighbourhood and local community.
C. Clients have a particular material and physical life situation - particular housing, employment, income, etc.
D. Clients have particular characteristics such as language, gender, age, cultural background.
E. Clients have particular issues effecting them or of concern to them (eg. domestic violence, gambling, drug or alcohol abuse, etc).
F. Clients are referred to the service.
G. Clients needs are assessed.
H. A level and approach to service delivery is determined.
I. Clients bring with them to the service their networks and interconnections with their family, friends and local community, their issues and characteristics.
J. Clients receive family worker services.
K. The services are provided in interactions between staff and clients.
L. Goals are identified to work towards.
M. The clients may receive other services from the family support service (eg groups).
N. The service is provided within the context of other services (eg. housing, health, etc).
O. The clients may be receiving other services from other service providers (eg child care, respite care, etc).
P. The services are provided in a particular location from a particular service provider.
Q. Outcomes are achieved.
R. These outcomes may have longer term impacts.

The provision of family support services may be part of a government funding program.

Evaluation perspectives

Within this framework evaluating family support services can be undertaken from different perspectives, for example:

Program evaluation: Evaluating the ‘family support program’ in NSW (eg the CSGP funding program)

Program monitoring and review: Gathering data to monitor and review what is happening in the program (eg the CSGP funding program or the Families First funding program)

Service capacity evaluation: Evaluating the network of services supporting families in a particular area or region of NSW

Agency focussed evaluation: Evaluating the local family support service

Client focussed evaluation: Evaluating the work with individual clients.

Evaluation from one perspective may be used in another. For example, client focussed evaluation may be one element in program evaluation. Agency focussed evaluation may be one element in service capacity evaluation.

Program evaluation

Questions

Program evaluation focusses on the program as a whole and will ask questions like:

What is the impact of the program?
Is the program achieving its outcomes?
Are the outcomes the right outcomes?
Has the program been implemented as planned?
How could the program be improved?

For example what is the impact of the CSGP program? What is the impact of the Families First program? Has the Families First program been implemented as planned?

Strategies

Program evaluation often involves a specialist evaluation team with specific terms of reference from the funding body. For example the Families First evaluation program.

Program Monitoring and Review

Program monitoring and review focusses on the program as a whole.

Questions

Program monitoring and review answers questions about what is happening in the program, such as:

How many people are using the program?
Are they in the target group?
How much service is being provided?
What are the costs per unit of service?

Strategies

Strategies typically used for program monitoring and review include:

Service agreements with conditions for service providers to provide service data
Service providers' yearly plans being submitted to the funding body.
Standardised data collection systems including minimum data sets.

For example, the CSGP has a service framework, service agreements and work is underway for standardised data collection.

Service network capacity evaluation

Questions

Service network capacity evaluation will ask questions like:
Does the network of services in this area or region have the capacity to effectively deliver services to meet the needs of the community?

Strategies

Evaluation of service network capacity often involves a specialist evaluation team with specific terms of reference from a funding body. For example the Families First evaluation program has a review of service network capacity as one component of the program evaluation.

Agency focussed evaluation

Agency evaluation includes:

• periodic external evaluations
• internal ongoing evaluation.

Periodic external evaluations

One off external evaluation of services focus on the service at a particular location rather than the program as a whole.

Questions

Questions asked are similar to the kinds of questions in program evaluation and program monitoring and review but asked about a particular provider. Typical questions are:

Is the service achieving its aims and objectives?
Is the service well run?
How could the service be improved?
How many people are using the service?
Are they in the target group?
How much service is being provided?
What are the costs per unit of service?

Strategies

There are two kinds of strategies that are typically used for periodic external evaluation of services.

An external evaluation consultant or team being appointed and given specific terms of reference. The consultant(s) would usually work to a steering committee.

Accreditation processes - where agreed standards and indicators have been agreed for the program and a system established for identifying whether or not service providers meet the agreed standards and indicators.

At present there are no specific plans to develop an accreditation process for the family support services in NSW.

Internal ongoing evaluation of service providers

Internal ongoing evaluation focusses on the services at a particular location (not the program as a whole). It will be developed and undertaken by service provider.

Questions

Each family support services needs to be able to answer questions such as: Do we know what we need to, to know whether or not we are providing a quality service and how we could improve the quality of our service.

Specific questions include:

What is the profile of our local community?
Who requests what information and services?
What services have been provided?
To whom?
Did the service make a difference? In whose eyes? How do we know?
What are clients views of the services?
How can the service be improved?
Is the service provision working collaboratively with other agencies?
What is the community perception of the service?
What community consultation work and interagency collaboration is undertaken?

Strategies

Each local service needs to put in place a wide range of strategies to ensure they can answer the above questions. Strategies include:

An organisational manual

Supervision of staff

Client feedback mechanisms

Staff feedback mechanisms

Informal ongoing discussion with clients using services.

Independent interviews with clients. Having a person outside the program interview client about such questions as whether they are getting the support they want.

Focus groups. Having an independent person facilitate a focus group about the service. The group could discuss such questions as: What do people most like about the service? What do they least like? What could be improved?

Case studies. Statistical data provides useful summary data about such things as what services are provided and what are clients views about these services. However this type of collated data does not provide insight into what difference the service is making in people’s lives. To understand this one needs to get clients to tell their stories and have these stories written up as case studies.

Peer review. Family workers working with each other to review their work with individual families.

Profile of the service within the community. Contacting other services for feedback on how they see the service.

Links with agencies, literature and reflection. Strategies could include: hearing what others are doing; reading about approaches to evaluation; reading about results from evaluation studies and social research; reflecting on the findings and their implications for the local program; and attending relevant conferences and staff training.

Client focussed evaluation

Client focussed evaluations may be part of an agency focussed evaluation.

Questions

Questions include:

Who requests what information and services?
What services have been provided?
To whom?
Did the service make a difference? In whose eyes? How do we know?
What are clients views of the services?
How can the service be improved?

Strategies

Strategies include

Referral form
Assessment form
Service provision data collection
Supervision of family workers
Case conferences and other ongoing review of the work with each family.
Monitor changes in client families over time
Work collaboratively with other services
Hear the views of clients and other service providers about services.
A community profile that identifies needs and priorities and an implementation plan for meeting these needs
Analysis of service processes, for example, analysing the referral and assessment process to ensure that all those entitled to receive service have an equal chance of being referred and assessed.

Evaluation supported by research

Some of the evaluation questions it would be good to ask will not be able to be answered unless research is undertaken.

For example considerable research was required to show the link between smoking and health issues. It would not be possible to evaluate the effectiveness of an anti-smoking campaign if there was no evidence to show the links between smoking and health issues.

In the same way research may be required to show the links between family support services and long term outcomes for children. This research may involve longitudinal studies or other research strategies that would be outside the typical scope of program evaluation, agency focussed evaluation or client focussed evaluation.

Evaluating family support services

To comprehensively effectively evaluate family support services in New South Wales will require work on each of:

1. Program evaluation
2. Program monitoring and review
3. Evaluation of service network capacity
4. Agency focussed evaluation (both external periodic and internal ongoing)
5. Client focussed evaluation.

These evaluations would need to be underpinned by adequate research, particularly research showing the cause and effect links between services provided and outcomes achieved.

To comprehensively evaluate family support services would require involvement from:

Clients and family workers
Family Support Services
Peak organisations such as NSW Family Services
Government
Other organisations such as universities.

It will require resources.

It will require a thorough understanding of both evaluation and the nature of family support service processes and the inherent paradoxes and dilemmas involved.

It will have an impact on how each person in family support services works.