Measuring up: performance indicators for better healthcare

Page 18 

David Rooney

Industrial countries spend a large proportion of GDP on healthcare. But how can they be sure they are getting value for money? One way is to use performance indicators, but these can have their drawbacks, as well as their advantages.

There is no doubt vaccination works, yet 20,000 people die every year in developed countries from vaccine-preventable diseases, according to WHO estimates. Numerous studies over the years have found no evidence that antibiotics help treat the common cold, yet 40-60% of doctors still prescribe them. These are just two examples of the gap between knowledge and performance in our healthcare systems.

Why does this gap exist? A key problem is the sheer weight of new knowledge. In the mid-1960s, about 100 articles from randomised clinical trials were published. By the 1990s, approximately 10 000 such articles were being published every year and nearly half (49%) of all the extant medical literature has been published in the past five years. The result is an estimated 15-20 year time lag between identifying more efficient treatment and incorporating it into routine practice. The gap applies across the board, from complicated clinical conditions to routine medical problems, such as the common cold.

Other key problems include the complexity of healthcare systems, the increasing demands fuelled by patient expectation, ageing and new technologies, as well as the reality of constraints and poor distribution of resources. Improving the performance of healthcare systems is becoming a priority in many countries as a result of economic pressure to optimise health spending, as well as increasing evidence of the scale of deficiencies in quality of care.

The scientific evidence is accompanied by widespread concern about eroding performance. In a 1998 survey of five countries, more than half the doctors questioned in Canada (59%), the United States (57%) and New Zealand (53%) reported that their ability to provide quality care had worsened over the past five years. Some 46% of UK physicians and 38% of Australians told the same story. Only a quarter or less of the physicians surveyed reported that their ability to provide high quality care had improved over the period. A similar survey of nurses in Canada, Germany, Scotland, England and the US in 1998-99 found that between 17% and 44% of them believed quality had deteriorated in the past year.

These findings cannot be dismissed as simply the complaints of demoralised clinicians, since patients respond similarly. In the 1998 survey, the public indicated overwhelmingly that the health systems in their country required fundamental change.

Consensus is emerging on the principal areas to cover in performance measurement: namely, effectiveness, efficiency, responsiveness and equity. But once you have defined your terms, how can performance measurement actually improve quality? There are numerous possible methods, but scant evidence available to provide a basis for selecting one. Factors such as underlying values, financing and organisational arrangements come into play.

The choice of method will also depend on whose behaviour you are trying to change: providers, professional bodies, citizens or managers. Identifying a ¡§best method¡¨ may not be realistic, but being aware of the possible approaches, their strengths and limitations, and the experience of countries that have tried them, can help in making a choice.

Performance indicators are employed for four basic functions: facilitating accountability; monitoring healthcare systems and services as a regulatory responsibility; modifying the behaviour of professionals and organisations at both a macro (population) and micro (patient) level; and forming policy initiatives.

Public demand for accountability is rife and nations are responding in their own ways. OECD countries generally apply three models of accountability in healthcare ¡V professional, economic and political. They use various combinations of these, all of them relying on performance indicators to some degree.

Professional accountability, dominant in most health systems historically, views the physician as the key to controlling quality and uses certification, accreditation, licensing and litigation as instruments for enforcement. But the professional model of accountability is increasingly regarded as insufficient unless accompanied by one of the other two.

The economic model, of which the US is the clearest example, is based on the idea that the competitive market can be used to enforce accountability. Health plans can influence physicians¡¦ choice of treatment by declining to fund some practices or encouraging others. And accountability through public reporting is believed to have resulted in improved performance in certain areas. For example, the rates of beta-blocker prescribing following heart attacks rose from 62% in 1996 to 85% in 1999 after standardised reporting was introduced. The political model meanwhile views the citizen as receiving a public good, so the government¡¦s role is to act as an agent of change on behalf of the public.

Objective measures of performance are increasingly used at several levels. They can dramatically influence policy, for instance. UK Prime Minister Tony Blair¡¦s decision to invest significant new resources in the National Health Service (NHS) was influenced, in part, by data showing his country to be spending a lower proportion of GDP on health than most northern European countries. And the US was influenced to train more general practitioners when data showed that it trains a higher proportion of specialists than most European countries.

Importantly, performance indicators can help to make policy priorities explicit, for example by defining national priorities and then identifying specific performance targets within those priorities. Australia has had a system of identified national health priorities since 1996; namely asthma, depression, diabetes, cardiovascular and injury, with reports describing the best available data in most priority areas.

The UK¡¦s NHS, after finding that outstanding claims for alleged clinical negligence in its hospitals have reached $5.6 billion, has set targets for reducing serious harm, such as a 40% reduction in prescribing errors by 2005.

Assisting healthcare professionals in practicing evidence-based medicine is a key objective for improving quality. Performance indicators, embedded in clinical guidelines and peer reviews, are among the most common approaches aimed at bridging the knowledge gap, but have limited effectiveness when used alone to change physician behaviour.

Still, scepticism remains, as was demonstrated in a recent survey of more than 100 doctors in England. Some 85% said they would ignore the guidance of the newly established National Institute for Clinical Evidence, responsible for developing the evidence basis for guidelines and protocols, if they thought it was wrong. There are promising signs of potential in incorporating practice guidelines into computer support systems. Indeed, analysis shows that the use of computer-generated prompts can improve preventive services and how drugs are prescribed.

Experience in the US suggests that institutions too can use performance data to improve care processes, to identify poor performers, and to respond to patients¡¦ preferences or complaints. One often-cited example is the decision by the New York State Health Department to publish mortality rates after coronary artery bypass graft (CABG). In the first five years of the programme (1987-92), the mortality rate in New York declined twice as fast as the national average. But in the face of this dramatic success, so far there has been little effort to achieve similar results with this method either for CABG in other states or for other types of treatment in New York. What¡¦s more, nobody is sure why this approach has not been more widely adopted ¡V is this a problem of professional resistance, technical capability or resources?

A US movement sometimes referred to as informed consumerism has been relied upon as a means of using competitive market forces and individual choice to drive up performance. But an article in JAMA in 1999 found that patients/consumers have made very little use of performance data when making healthcare decisions, continuing largely to rely on word-of-mouth. One reason may be that most of the published performance data was very specialised and not useful for most patients, such as CABG mortality rates.

However, performance data designed specifically for consumers, such as information about success rates for common procedures and treatments, may be the most efficient way of delivering data to the public.

Patient empowerment is not just politically correct, it can cut costs and improve quality. There is now a body of literature showing that better-informed patients have better outcomes, choose less risky procedures and avoid equivocal treatments. This should increase our confidence that patients can not only make constructive use of performance data designed for them, but can also be reliable informants for performance assessment.

Performance data designed specifically for consumers, such as information about success rates for common procedures and treatments, may be the most efficient way of delivering data to the public.

Another issue is cost. Since employers are the dominant buyers of healthcare in the US, they theoretically have both the motive and clout to buy health services or insurance coverage based on performance. But in reality, price trumps all other performance data. Two studies, which between them looked at more than 1 500 employers across the US, concluded that their use of performance data was limited, again at least partly because the data was not packaged to be useful to them.

Performance measurement and reporting have clearly made dramatic advances in the past decade but more needs to be done so that fair and accurate assessments can be predictably and credibly provided. Performance indicators, whether length of waiting lists or choice of treatment for a particular ailment, may light the way forward, but face significant challenges.

The state of the art of performance measurement is embryonic, with insufficient understanding of exactly how data can help achieve change, under what circumstances and with what consequences.

New resources will be required to build capacity, most notably in informatics and information technology. Most importantly, performance data will have to be made even more useful to target audiences with clear, measurable and achievable goals for improvement.


• Blendon R.J., Schoen C., Donelan K., Osborn R., DesRoches C.M., Scoles K., Davis K., Binns K., Zapert K., "Physicians' View on Quality of Care: A Five-Country Comparison", Health Affairs, May-June, 2001.

 Emanuel E.J., Emanuel L.L., "What is Accountability in Health Care?", Annals of Internal Medicine, January, 1996.

 Marshall M., Shekelle P., Leatherman S., Brook R., "What Do We Expect to Gain from the Public Release of Performance Data? A Review of the Evidence", Journal American Medical Association 2000, Vol. 283, No.14.

©OECD Observer No 229, November 2001 

Economic data

GDP growth: -9.8% Q2/Q1 2020 2020
Consumer price inflation: 1.2% Aug 2020 annual
Trade (G20): -17.7% exp, -16.7% imp, Q2/Q1 2020
Unemployment: 7.4% Aug 2020
Last update: 13 Oct 2020

OECD Observer Newsletter

Stay up-to-date with the latest news from the OECD by signing up for our e-newsletter :

Twitter feed

Digital Editions

Don't miss

Most Popular Articles

NOTE: All signed articles in the OECD Observer express the opinions of the authors
and do not necessarily represent the official views of OECD member countries.

All rights reserved. OECD 2020