management jargon, political marketing,
or one small element in developing quality services?
What are performance indicators?
Putting performance indicators in their place
Performance indicators - The FINE print
Is your organisation ready for performance indicators?
Will you use performance indicators wisely?
article was originally published in Caring, the Association of Children's
Welfare Agencies Newsletter (September 1991).
have reached plague proportions in NSW. They are overrated
by the NSW Premier and Government. They are misunderstood and misused
by Government departments. They are undervalued by the welfare sector.
They are over-promoted by consultants. They are also one small important
element in the development of quality services.
within NSW in the past few years can be crystallised by the State
Government's slogan "Putting People First by Managing Better". With the
slogan has come management by objectives, demands for higher levels of
accountability and the need to sell to the electorate the fact that the
state is managed better and is performing better. The need to sell the
idea that performance is improving has overshadowed the need to actually
There is a genuine
need for NSW Government Departments, funding programs and welfare
organisations to focus on performance, ie what they are
achieving, rather than strategies, ie, the activities that are undertaken
to achieve results. The emphasis on performance is very useful in bureaucracies
and organisations that have tended to justify their existence by the mere
fact that everyone is busy rather than what they are actually achieving.
are one way of reinforcing a legitimate emphasis on performance however
their implementation in Government departments and the welfare
sector leaves much to be desired. Some Government Departments
just don't understand how to set objectives and develop performance indicators
(See for example the Corporate Plan of the Department of Planning 1990-1993).
wanting to measure performance of funded organisations are now starting
to specify in their funding agreements that performance indicators have
to be used and/or developed. Many community organisations funded through
the Community Services Grants Program are developing performance indicators
to keep the funding body happy rather than to improve performance.
For funded bodies
increasing demands for performance indicators are made in a climate of
no growth, government funding cut backs, and the rationalisation of services
to distribute the funding dollar 'equitably'. Funded organisations are
often feeling powerless in the face of the decisions of the bureaucracy
and there is a growing breakdown in trust between the funded organisations
and their funding bodies.
An emphasis in organisations
on what is being achieved is important. However to actually improve
performance many other things besides performance indicators are needed
as well including: commitment to providing and developing quality services;
staff who are willing and encouraged to ask questions about why they do
what they do; Management Committee members and staff who are focussed
on the clients' need and meeting their clients' needs; performance indicators
that are seen for what they are, just one small element in developing
quality services; and adequate support services for example staff training
and administrative support systems.
in services that aim at changing the person receiving the service are
more difficult to develop and use effectively than performance indicators
in for example the manufacturing sector. In human services
there are complex measurement and cause and effect issues to be considered
in developing and using performance indicators.
In human services
if performance indicators are used as the judges of performance
then they will be misused and abused and it is likely that performance
will remain static or even decline.
If performance indicators
are seen as some clues among many to asking questions
about how performance can be improved then it is possible that performance
is for the Government, Government Departments, funding programs and funded
organisations to actually improve their performance. To do this may require
a new attitude to performance indicators and a much better understanding
of the issues involved.
are Performance Indicators?
are one of many tools to help answer the question: How do you know what
you are achieving?
One definition of
a performance indicator from an NCOSS publication is: A numerical
measure of the degree to which the objective is being achieved.
are usually seen as numerical measures of achievement that are easy to
collect and use. In theory they can only be derived for things over which
you have control, however in practice people don't have absolute control
over anything and so 'having control' is really a matter of whether there
is enough control for your purpose.
A more sophisticated
definition from the Office of Public Management is: A performance
indicator defines the measurement of a piece of important and useful information
about the performance of a program expressed as a percentage, index, rate
or other comparison which is monitored at regular intervals and is compared
to one or more criterion.
one of your objectives is to train people so they are able to gain employment
then one performance indicator can be the proportion of trainees that
gain employment after the training is completed.
one of your objectives is to refer people to services appropriate to their
needs then one performance indicator could be the proportion of people
who contacted you that considered the referred service relevant to themselves
and their needs.
one of your objectives is to improve the parenting skills of parents then
one performance indicator could be the proportion of parents who feel
they are coping better with parenting their children at the end of the
program (than they were at the beginning).
order to develop performance indicators it is essential to:
identify the organisations values and philosophy;
identify the clients and their needs;
aims and objectives which specifically state what is to be achieved
in relation to these needs;
each step in the service delivery process and what is to be achieved
at each step in the process as well as how it is to be achieved.
for example, an organisation committed to the empowerment of families
may have an aim of improving the functioning of families and a service
delivery process based around the following outcomes:
the client to acknowledge their need;
the client to articulate the need;
the client to identify what needs to be done to meet the need;
the client to increase their self esteem / confidence so it is possible
for them to act to meet their needs;
the client to increase their skills and develop appropriate attitudes
so the client can do what they see they need to do;
the client to do what needs to be done to meet their needs.
performance indicator for each of these steps could be the proportion
of total clients where both the worker and the client agree that the outcome
has been successfully achieved.
that could be used to achieve these outcomes could include providing information
to clients, interviews, group work, training and so on.
There is a commonly
held view that performance indicators are used to judge performance. While
this may seem self evident it leads to many difficulties in human service
organisations. If performance indicators are used as the judge of performance
in an unthinking way, wrong conclusions will be drawn.
For example, one of
the performance indicators for labour market programs in TAFE is whether
or not people in the program gain employment at the end of the course.
If the number of people gaining employment at the end of the course reduces
has the performance of the course reduced? Possibly, but not necessarily.
Alternative explanations are that: there could be less jobs available
now than there were previously; the people coming into the course may
be less skilled than previous intakes; or a larger proportion than previously
may be going on to further education.
Some people have the
view that performance indicators should be used as the basis for rewards
(and punishments) for individuals or organisations. This may also seem
self evident but it is even more seriously flawed than the previous proposition.
It assumes that performance indicators and performance are the same, and
in most cases they are not.
So where rewards or
funding is based on performance indicators the individuals and organisations
being judged and rewarded typically start to fudge the figures to obtain
the rewards irrespective of performance. The larger the rewards the greater
the pressures to fudge the figures.
Fudging the figures
can be done in a variety of ways. Sometimes borderline or doubtful cases
are reclassified as successful outcomes. "That client really did achieve
something didn't they?"
Sometimes the way
in which clients are counted is not appropriate. Some organisation have
been know to count clients who come to groups each time they come. So
if 10 clients come to a group that meets once a week for 10 weeks there
are 100 clients counted not 10.
Sometimes the definition
of a successful client outcome is changed. For example last year we only
counted the outcome as successful when both the worker and the client
agreed that the service was successful where as this year we count the
success whenever the client thinks there was a success irrespective of
the workers views.
performance indicators to be used wisely they need to be seen as numerical
indicators that require interpretation. They need to be used as some of
the many clues available to help in asking questions about performance
and the improvement of performance.
Departments, Management Committees and staff who wish to use performance
indicators wisely will need to be firmly committed to provision and development
of quality services and see mistakes (including poor performance) as an
opportunity for improving performance rather than an opportunity for 'punishment'(e.g.
reprimanding staff or cutting funds). This orientation is only possible
where there is a basic trust between the key players (for example staff
management committees and funding bodies).
Performance Indicators in their Place
the last few years in NSW performance indicators appear to have been over-rated.
They have been blow up out of proportion. Their rightful place is as one
small element in the process of planning for quality services.
the reviewing of funding programs there has been strong presses for funded
organisations to develop performance indicators. Performance indicators
cannot be developed in a vacuum. To effectively develop and wisely use
performance indicators it is probably essential to deal with each of the
following steps that are required in planning for quality services.
to Quality Services
Management Committee Members and staff need to be committed to providing
quality services and the continual improvement of the quality of those
services if there is to be an appropriate context for the development
of performance indicators.
the current political climate this commitment to quality service provision
must be able to withstand the pressures that will be felt to 'just keep
the funding body happy'.
organisations values must be made explicit if there are to be firm foundations
for the organisation.
values the organisation adopts will affect the needs it wishes to address
and clients to whom it will give priority.
will affect approaches to service delivery and answers to questions such
are quality services?
do you get quality services?
do you know you are providing quality services?
example if a child care centre's values were built around a core value
of 'children should be seen and not heard' then a quality service might
include teaching children to be quiet and orderly, service strategies
might be highly structured and controlling and a quiet, clean child care
centre could be seen as a 'measure' of success.
if a child care centre's values were based around the core value of 'individual
creativity' then a quality service might include helping children explore,
the services would be provided by children being encouraged to express
themselves and a measure of success might be the level of noise, fun and
mess during the day.
Peoples needs are
the reason for existence for human service organisations. Despite this
organisations often have difficulties clearly identifying the needs they
wish to meet.
If your organisation
is a direct service organisation the need will be a client's need, for
example, the need for food, shelter, friends or self-esteem. If it is
a need it is probably something you have experienced or can imagine experiencing.
Training is not a
need; we may have or feel a need to understand and undertake some training
as a way of meeting the need. Counselling is not a need; we may have or
feel a need to get on better with those around us and go to a counsellor
to help us meet our need.
Some examples of client
- Need for friends
and social contacts;
- Need for a safe
- Need to have an
- Need to know;
- Need to be able
to relate well with those around me;
- Need for parents
to have a break from their children.
If the need is clearly
identified then a lot can be learnt about what the aims and objectives
of the organisation should be, what sort of services are appropriate,
where they might be located and so on.
By way of analogy
imagine yourself as a 'seller of hats'. What need are you meeting for
the people who buy your hats? Is it the need for a hat?
If you sell hats there
are many needs you could be meeting. People may have a need to feel good
in front of others; they may have a need for protection from the sun;
they may have a need for protection from the snow; they may have a need
for disguise; they may have a need to meet work regulations (for uniforms).
If you were meeting
the need of people to feel good in from of others (fashion) this description
of the need would give you a lot of clues about what hats you would sell,
to whom, at what price, in what location. If you were meeting people needs
for hats for uniforms you would be selling different hats, probably to
different people, at a different price, in a different place.
In the same way if
an organisation is able to clearly and specifically identify the need
it is meeting there will be many clues about the clients, the services
they might be require, the location and so on.
Once the clients and
their needs are clearly identify you will need to ask: What do we want
to achieve in relation to those needs? The answer to this question will
be your aims and objectives.
Aims are general statements
of what you intend to achieve. In direct service organisations aims will
often include or imply what is to be achieved in relation to client needs.
Some examples of service
aims are: to strengthen and enhance the capacity of families to meet their
needs in child rearing; to enhance the capacity of older people to live
full and independent lives within the community; to increase the accessibility
and quality of social infrastructure for disadvantaged groups.
make a serious error when they are writing their aims and objectives.
They write strategies but call them aims and objectives. The following
are not examples of aims: to run parenting groups (this is a strategy
that would meet the first aim above); to run independent living courses
for those over 65 (this is a strategy that could be used to meet the second
aim above). These aims do not tell you what the organisation intends to
achieve in relation to client needs.
Objectives are specific
statements of what you intend to achieve. It is very useful if objectives
are written so they are
- Results oriented
(ie written as something to be achieved);
- Time bound.
Some examples of objectives
are: to provide 35 parents each week with a 2 hour break from their children
for the next 12 months; within 18 months to significantly increase the
level of self-esteem of all the parents of children (within such and such
a geographic area) who are at high risk of child abuse.
Sometimes the specific
needs of clients are very different from one client to the next. In these
services each clients' need has to be assessed on an individual basis.
An alternative approach to the above is to write objectives such as:
- For the client
to articulate the need;
- For the client
to identify what needs to be done to meet the need;
- For the client
to increase their self esteem / confidence so it is possible for them
to act to meet their needs;
- For the client
to do what needs to be done to meet their needs.
Once the aims and
objectives are identified a service delivery process needs to be developed.
The service delivery process will include:
* A list of what is
to be achieved at each step in the process;
For example three
initial outcomes in a counselling service delivery process might be:
- Clients are identified
- Client needs are
- Clients are prioritised
* Explicit assumptions
that you are making about how the outcomes in your service delivery process
For example in the
above process the assumption could be that clients will be prioritised
on the basis of their needs and that is why the needs have to be identified
before clients are prioritised.
* A list of the strategies
that will be used to achieve each outcome in the service delivery process.
For example the strategy
used to identify clients could be to wait for referrals;
The strategy used
to identify client needs could be an interview and completing of an assessment
The strategy used
to prioritise clients could be a staff meeting.
* Explicit assumptions
that you make when you choose a particular strategy for each outcome in
your service delivery process.
For example waiting
for clients to be referred assumes that referring agencies know your services
are available and that clients in need will contact either your or the
agencies that refer clients to you.
What's Going On
need information that can help them know both whether their aims and objectives
are being achieved and where their strategies are being effectively implemented.
information is information you can use to help answer the question: Are
we achieving our aims and objectives?
information is information you can use to answer the question: What do
we know about what are we doing? Are we carrying out our strategies?
information can be quantitative or qualitative. Performance indicators
are one kind of quantitative performance information.
information can be qualitative or quantitative.
of the statistics that organisations typically keep are quantitative activity
information. For example, the number of clients, the number of groups
run, the number of referrals. This is not primarily performance information.
Performance information would include information about what the clients
achieved and what the groups achieved.
- The FINE Print
When you read the
fine print things often turn out not to be as simple as they first appear.
Performance indicators require a lot of fine print. Here is some of it.
of Activity/Workload Indicators
definition performance indicators must be about performance rather than
the activities undertaken or the level of workload. Occasionally activities
and performance are directly related, for example, the number of people
vaccinated is a good indicator of both the performance achieved (people
vaccinated) and the activities undertaken by medical staff (the number
of injections given), however often outcomes and activities are not so
many organisations and government departments either do not write their
aims and objectives as statements of what they will achieve or do not
develop performance indicators that relate to achievements. For example
the Department of Planning has as one of its objectives "To encourage
community participation in environmental planning and assessment process,
and to disseminate information about the work of the department to the
public" and the performance indicators are "Number of publications produced,
exhibitions held, articles published, public enquires handled..." Where
are the performance indicators of actual community participation? What
is the information dissemination designed to achieve?
reliable is the indicator?
is about asking the question if I measured the same thing again would
I get the same result?
I were to measure the floor area of a room, got out a tape measure, measured
the size of the room and calculated the area and the room was 20.35 square
metres what is the chance that if you measure the same room with the same
tape measure that you will get 20.35 square metres? The chance of that
would be remote. There is always error in measurement. It is simply a
question of how much error, not whether there is any.
your organisation develops performance indicators you need to make an
estimate of how reliable they are and decide whether they are reliable
enough for your purposes.
human services the outcomes are often changes in the clients. Measures
of these changes, for example, changes to the level of self-esteem are
likely to be far less reliable than measuring the size of a room.
valid is the indicator?
is about answering the question: Are we measuring what we say we are measuring?
you know how valid your performance indicators are? Do you know how valid
they need to be for the use you are going to put them to?
example if I were to measure the height of 20 children aged from 6 months
to 10 years by putting them on a set of bathroom scales my measure would
not be valid. However if I only wanted to identify the short and taller
children weighing them might be adequate for my purposes because there
is a reasonable degree of relationship between weight and height in children.
The measure might be valid enough for my purpose.
of Control and Cause and Effect
theory you need to have control over the outcome you are intending to
achieve if you are to develop a performance indicator for the outcome.
NCOSS publication "Performance Indicators for Community Organisations"
in Chapter 2 The Rule" states "There are five basic rules which need to
be remembered about performance indicators...3. Performance indicators
can only be derived for things over which YOU have control."
in practice people don't have absolute control over anything and so 'having
control' is really a matter of whether there is enough control for your
example most family support services would have as an objective for many
of their clients an increase in the level of self-esteem for the parent(s).
Even assuming that one could develop a valid and reliable measure for
self-esteem it must be asked whether self-esteem would be a good performance
difficulty arises because there are a multitude of causes that affect
the clients self-esteem that do not relate to services provided by family
support services. How can a counselling service show that there is a cause
and effect relationship between the service provided and the changes in
the client. For most services this would be almost impossible to prove
with absolute certainty. Even with major research efforts proving cause
and effect relationships can be very difficult if you have to convince
people who hold a different view from yours. Look at the history of the
debate about whether smoking causes cancer. What chance has a small community
based organisation got to prove that its service delivery process is the
cause of the change in the client, especially when the people needing
to be convinced have a different value base or don't trust you?
difficulty is intrinsic to all human service organisations. In human service
organisations the service is intended to have an impact on the person
receiving services. However the service is only one among many causes
effecting the client and often the real outcome that is being sought will
only be observed years or more after the service has been provided.
Performance with Performance Indicators
is one example of what can happen when performance is equated with the
Commission, 1861, represents an early attempt to use evaluation to change
learning patterns. The commission had been set up in order to raise the
standards of achievement in the basic subjects. It reached the following
There is only on
way of achieving this result, which is to institute a searching examination
by competent authorities of every child in every school to which grants
are to be paid, with a view of ascertaining whether these indispensable
elements of knowledge are thoroughly acquired, and to make the prospects
and position of the teacher dependent, to a considerable extent, on the
results of this examination. (Newcastle Commission 1861)
"The report of
this commission led to the institution in England of the 'payment by results'
system which was to last forty years in formal terms but whose effects
are still visible. The examination in the schools were conducted by inspectors,
who tested in a very limited area of computation together with one problem.
The grants were paid on the basis of the number of pupils attaining a
pass mark. The consequences were disastrous: the already limited teaching
became narrower still, concentrating on the rote learning of simple number
facts and ignoring and applications. There was no encouragement for higher
achievement and thus the main effort was spent on cramming the weaker
children to get a pass mark and neglecting the needs of the average and
more able pupils. The effects were thus quite reverse of the intentions,
contributing to a very limited achievement coupled with a lasting distaste
performance indicator was the pass mark. Schools were rewarded for an
increase in the performance indicator. Teachers taught to the test. However
education which was not the same as the test result suffered.
(and performance indicators) can be affected by inputs into a program.
How clients are selected can change an organisations outcomes even if
the processes of service delivery remain effectively the same.
example in the New Enterprise Incentive Scheme Program unemployed people
are provided with training and an allowances approximately equal to the
unemployment benefit to help them set up new businesses. A performance
indicator is the number that are still in business at the end of 12 months.
the target group of this program includes migrants, single parents and
long term unemployed so long as the performance indicator is used as a
reward people will be accepted into the program on their ability to set
up the business. One may ask would they have been able to set up the businesses
anyway and may it have not been more useful to work with those on the
margins so that people who might not otherwise set up a small business
do so. In this latter case the performance indicator would be lower but
the overall gain to society would be greater.
indicators can be very useful tools in helping you answer the questions:
How do you know what you are achieving and how can your service improve
if they are used simplistically the results can be disastrous.
Your Organisation ready for Performance Indicators?
your staff and management Committee all
committed to providing quality services?
you have a written statement of the needs
your organisation is wanting to meet and a
statement of whose needs they are?
you have a written statement of your
organisations values and philosophy that has
been formally adopted by the organisation?
you have a written statement of aims (what
you intend to achieve in relation to the needs
you wish to meet)?
you have a written statement of objectives
(specific, measurable, achievable, results
oriented and time bound statements of what
you intend to achieve in relation to the clients
you have a written statement of the key
steps in the service delivery process and what
you are trying to achieve in each step of the
you have a written statement of the key
assumptions that you are making in your service
delivery process; for example assumptions
about the logic behind your service delivery
process or assumptions about the strategies you
are choosing to achieve a particular end?
you a systems for collecting statistics
that everyone in the organisation uses both in
terms of putting information into the system and
getting useful information out of it?
the members of your organisation like asking
questions about why they do things?
your staff and management committee members
see mistakes as opportunities for improving your
1-3 Your organisation
cannot develop useful performance indicators; the ground work is not in
4-6 Don't give
up yet but you have a lot of ground work to do before getting onto performance
7-9 With a
little more ground work you might be in a position where you could develop
10 Your organisation
probably has a sound basis for developing performance indicators
Your Organisation use Performance Indicators Wisely?
- Has your Board
or Committee and staff discussed and agreed on answers to:
- What is a quality
- How do we get
a quality service?
- How do we know
we have a quality service?
- Will performance
indicators be used as the judge of your organisation's
success or failure?
- Will performance
indicators be used as the basis for rewards to staff?
- Will performance
indicators be used as clues to asking questions about performance and
- Will you be using
performance indicators for each outcome in each key step in your service
- Will staff, client
and committee perceptions be as important as performance indicators?
- Will your organisation
be rewarded by the funding body for improving performance indicators?
- Do you know how
reliable your performance indicators are?
- Do you know how
valid your performance indicators are?
- Could you convince
a reasonable person that what you do causes what you are trying to achieve?
1-3 Don't use
performance indicators you will only make things worse.
4-6 don't use
performance indicators unless you come to a much better understanding
of the issues?
7-9 Don't use
performance indicators yet, but don't give up hope, there is a chance
you might use performance indicators wisely if you improve your understanding
of them a little more.
10 There is
a good chance you might use performance indicators wisely in your organisation.
and Further Reading
Donovan, F and Alun
C. Managing Human Service Organisations. Prentice Hall.
Mayo, Toni. Performance
Indicators for Community Organisations, NCOSS, 1990
Gilbert, Roy. Reglomania,
The Curse of Organisational Reform and how to Cure it. Prentice
Hall. Sydney, 1991
Hughes, Phillip. Two
Roles of Evaluation: Compatible or Conflicting? Evaluation
Journal of Australasia Volume 2, Number 3, 1990.
Paul Bullen explores
the issues in four articles in this issue of Caring. The first article
is a provocative Editorial looking at the politics of
performance indicators. The second article answers the question What
Are They? The third article Putting Them in Their Place
locates them as one small element in the process of planning and developing
quality services. The fourth article, The Fine Print
explores some of the complexities of using performance indicators. There
is also two self rating Questionnaires to give you clues
on whether your organisation is ready for performance indicators and whether
or not you will use them wisely.
The four articles
are a reflection on practice. They arise out of consultancy experiences
within NSW over the past three years. They are written specifically in
relation to human service organisations within NSW, particularly those
organisations that are providing services to change the clients themselves.