A response to two recent reports citing apparent reductions in productivity

November 2008

Health economics is an exceedingly complex area, underpinned by an increasingly robust and sophisticated international research literature. It is also an area driven by politics (and Politics) with huge but diverse vested interests amongst funders, providers, managers, employees (especially doctors) and tax payers. There are significant differences between the concepts of efficiency, value for money, productivity and productivity growth that have implications both for methods of measurement and policy relevance of the resulting indices.

(“Efficiency” is measured as the ratio of output produced with given inputs relative to the maximum feasible output. “Value for money” reflects the value individuals / society place on output relative to the costs of production. This often corresponds to a cost-benefit analysis. “Productivity” is the ratio of a measure of total output to a measure of total inputs. “Productivity growth” is the change in output relative to the change in inputs. It is often interpreted as reflecting the effect of technical change on production. Robust measurement requires precise definition of the concept to be measured.)

Effective employment of these measures in pursuit of policy objectives requires selection of the appropriate measure for the issue at hand.

Research evidence predicts short term negative growth rates in the wake of significant technological changes. The literature specifically on the impact of information technology on productivity also points to an important role of organisational changes in realising potential benefits from new technology. Basu et al. (2003) suggest that these changes can lead to declining productivity in the short run due to disruption of production processes. There is no doubt that the health system is undergoing significant and continuous change, not least in the use of technology, and anyone with personal knowledge of the impact of changing IT can vouch for the short term disruption and dislocation that such technology brings.

Of more consequence for the health sector is the evidence for diminishing returns as increased activity allows more complex and expensive treatment. Prevention of ill health or further deterioration is notoriously difficult to demonstrate in the short term, and health service targets and crude outputs can act as perverse and unintended consequences. This was particularly so with the crude use of waiting time targets. Though activity rates have demonstrably been increasing more rapidly in recent years, there is some evidence that the average age of patients treated in hospitals has risen. Data that identify the characteristics of patients is needed to identify the extent to which changes in health productivity are affected by diminishing returns – and society has some tough choices regarding health policy affecting care of the elderly. Objectively, concentration of resources and power almost exclusively in the hands of high tech / tertiary health care is irrational compared with spending on reducing poverty, and the consequent inequalities in housing, nutrition, education and quality of life; however, that’s another debate!

A NZ Ministry of Health report (2006) http://www.treasury.govt.nz/publications/research-policy/ppp/2006/06-09/07.htm commenting on a 2004 paper measuring NHS productivity (which was remarkably similar in approach to the Roundtable paper) was very critical of the simplistic productivity measures used. Similarly, a major criticism of that paper is that no adjustments for changes in quality have been incorporated. From the international literature, specifically the NHS, it has been acknowledged that this type of crude analysis (outputs divided by inputs) is both flawed and unhelpful. In the UK, The Department of Health has since introduced new measures incorporating adjustments for quality, and have invested in the research to most effectively define new data and performance indicators to allow valid comparisons year on year. Many compound output measures, adjusted for quality, are now used. These include: survival rates, measured increases in patient health due to intervention, changes life expectancy, deaths in hospital, incidence of hospital acquired infections, waiting times, measures of patient satisfaction, and the use of value weight instead of cost weight measures.

A simple example of the use of a value weight is the calculation of health gain (and consequent increased productivity) attributable to introduction of statins to control cholesterol, compared to the cost of prescription of statins. When value weights based on added life years as a direct result of statin therapy were calculated from estimates of the number of life years gained by patients taking the drugs, the value weight was found to be over four times greater than the unit cost of the drug, increasing the measurable overall output of the NHS.

Health economists have long recommended that health care outcomes should be adjusted to reflect the higher value as economies grow and healthy individuals earn more than previously. When adjustments were made for both increases in quality and in whole system growth (2005) measurements of productivity by the NHS changed from negative to positive.

Two major costs impact on health providers: the costs of a scarce and highly skilled workforce, and the costs of pharmaceuticals. Both are predicted to continue to outstrip the general inflation rates due to the laws of supply and demand on the one hand, and to compound inflation effects and international patent protections on the other. This is a key reason why comparators using retail price index as measure of inflation are meaningless.

The work on quality of health output was based on research carried out by the Department of Health and by the Centre for Health Economics at the University of York and the National Institute of Economic and Social Research (Department of Health 2005, University of York and National Institute for Economic and Social Research 2005). http://www.york.ac.uk/inst/che/pdf/rp6.pdf

Fundamentally, drastic improvements in data are required: these must cover outputs, inputs and assets

Robust data are urgently needed for the development of valid measures of output, productivity and quality change in the Health Sector. Learning from similar work overseas, it becomes apparent that the size of the budgets spent on health justifies the increased expenditure required to be able to do meaningful performance measurement.

Outputs

The main aim of a health system is the improvement of the health of the population.

Therefore, any measure of health system performance, output and productivity should include measures of the effect of the whole system on population health. This requires the construction of productivity indices, depending in turn on data on changes in health status attributable to interventions. Development and systematic use of such standardised measures of health status to improve the effective management of the health sector and to provide the fundamental data needed to properly reflect any changes in productivity is required.

Inputs

Comprehensive data on inputs to health services is needed in order to explain changes in outputs and to measure productivity. Accurate data covering values of expenditures on inputs, volume measures and price deflators to convert values to volume measures when the latter are not available are essential. Also required, are data on the extent to which the quality of these inputs are changing through time. When volume measures are available, under the assumption that payments to labour equal marginal products, weighting diverse inputs by their shares in wage bills can be employed to adjust for quality change. Where either skill mix or qualifications change markedly, these changes must be factored in – dependence on simplistic FTE or headcounts is just not adequate. In the context of New Zealand, as the recent OECD report showed, the effects of inward and outward health workforce migration are profound and potentially serious. It is imperative therefore, that accurate data on the composition of the workforce is available for planning purposes.

Assets

In order to quality adjust capital input it would be useful to have separate investment data on types of equipment that have seen rapid technological change and change in unit cost (e.g. MRI scanners). Alternatively it would be useful to have the value of the stock of these assets, numbers of items and age profiles of the stock. Unless it is possible to access information on stocks and value, it may not be possible to adequately deal with questions of productivity growth and technical change associated with investment in new equipment.

In conclusion, accurate and objective measurement of health sector productivity is fundamentally different to industrial process production methodology. New and suitable measures providing adequate data are required to inform the productivity debate. To be fair, both papers were highly critical of the quality of data available and of the assumptions the authors were forced into adopting in the course of their work: and both urged caution with interpretation. However, until there is considerable improvement in the quality of the data, and of the analysis, it is unfair, misleading and unhelpful to allow such biased reporting of simplistic material, leaving the New Zealand public with the inaccurate impression that their tax dollars were being wasted on a poorer performing and better paid work force.

Specific comment on each of the 2 productivity papers

Both papers reflect the difficulties and complexities of designing robust and meaningful metrics by which to performance manage and contain the rising costs associated with healthcare. Both also highlight the lack of robust data on which to base comparisons, or workforce planning. Until health workforce / management information systems catch up with data quality in other areas of health, policy makers and planners will be unable to provide reports of sufficient depth and accuracy to use as levers for identifying and spreading best (value) practice. Adding in quality metrics, and developing the sensitivity to be able to track the impact of changes will also be vital to ensure the best use of tax dollars as patient expectations and health costs globally and locally continue to rise.

Productivity Performance of NZ public hospitals, 1998-2006 by Mani Maniparathy , funded by NZ Business Roundtable.

I found this to be an extraordinarily partisan report, which was very selective in the conclusions reported even from its own (limited) data. Although it did acknowledge many of the limitations, not surprisingly, the press grabbed hold of the headline “doctors and nurses are less productive despite being paid more”.

The main criticism of this paper is that costs have risen across the board in health care, faster than the CPI. Although the paper acknowledges that there is controversy in applying this (rather than PPI) as a comparator, it is disingenuous to ignore the international effects of health care pricing, or the fact that the ongoing changes in medical and nursing practice means that expectations have risen, that more is being done, that people survive longer but need more complex care, and that changing population demographics plus new technology means that the crude output measures attempted are just not able to compare like with like over this time period.

Methodologically, there is not enough detail given to be able to comment on the assumptions used in the models, particularly the use of proxy measures of output (DRG only) and CPI versus PPI.

It is interesting to note that the slopes representing rising costs for all the graphs run parallel – which given the rates of change in inflation, and wage rises would not be expected. Looking for example at the figures for the cost of blood, and for patient transfer, these both proportionately rose far more than staff costs. FTE (especially given the difficulties comparing these) hide differential costs of outsourced and part time staff. Although up to 80% of all costs in the health sector are said to be staff related, similar changes over time in detailed figures for pharmacy, utilities, capital projects, IT and training, medical equipment and supplies would have been instructive. Total costs could justifiably be compared with the rising costs of staffing budgets using the same crude patient outputs, and good performers best practice discovered and shared. Headcount versus FTE is a crucial distinction – there are extra costs in employing part timers (increasing headcount) FTE are often posts that are not filled in practice – with delays in recruitment, secondments, sickness etc not accounted for in the figures. Figures show widely divergent dotted lines (reflecting the levels of uncertainty).

They also show a steady rise in the number (actual volume) of admitted patients, showing a 22% rise in weighted outputs. The paper reports an overall real average personnel costs increase 16% over the five years between 2000 and 2006. One could just as easily claim an increase in output per cost, relative to inflation! The graph (fig 3) showing doctor productivity falling slower than nurse productivity is meaningless: one hip operation produced by one surgeon requires nurses in theatre, recovery and on the ward after care for some days: the picture shown is therefore completely dependent on the weighting adjustments being accurate – considering doctor costs rose faster than nursing costs, (fig 4) it is hard to understand why nursing productivity apparently fell. Organisational change, the changes to physical infrastructure, and many other rate limiting elements have not been accounted for in the analysis. The increase in actual output has been achieved partly by factors such as changes in the ratio of in to outpatient days, reduction of length of stay. The figures are not provided in sufficient detail to show the actual number of hours worked per doctor of nurse, but one driver for increasing the headcounts was to achieve a reduction in the number of hours overtime, and to reduce the number of outsourced and agency staff required.

Reduction in hours worked per staff member may be less “efficient”, but is crucial if NZ is to retain these skilled staff, and if patient safety is to be assured. It is instructive to reflect that the increases in senior doctor costs in particular are raised disproportionately by dependence on outsourcing – a consequence of difficulties in recruitment and selection. More rigorous analysis, and considerably better data is required to determine the best fit of inputs and outputs. The law of diminishing returns may apply: for example increases in surgery can be limited by the availability of beds. Other metrics such as % bed occupancy, or quality indicators are equally valid indicators of effective management. It is not sustainable or safe to continue to ask staff to work excessive hours to meet basic need. The paper does not report outpatient activity, which has increased significantly more than inpatient activity. Its omission is highly significant. (From the Ministry paper below, these appear to constitute 60% of activity).

Ministry paper: Productivity & efficiency in the delivery of public hospital services in NZ: initial results from two approaches to measurement, MoH June 2007

This was a much more comprehensive and thoughtfully produced report, but again, the headlines did not reflect the detail reported. One key bullet point would justify a headline “more expensive and senior staff equals improved productivity!” The statements: “Significant wage rises led to increase in cost per output” and “productivity rose 04/05 and 05/06” are incompatible, given the MECA agreement negotiated in 2004 (bullet points 6&7 page 1).

One wonders whether the 3 DHBs chosen while comparable with each other, are representative of DHB’s generally, particularly large urban ones.

Key bullet points:

• Outpatient activity and certain inpatient stays were accounted for differently (p8) – rather than being given a straight forward weighting, numbers were multiplied by price: the impact of all other inflation pressures in the system (including costs of wages) therefore impacted. Outpatient activity accounts for up to 60% of activity.
• The paper stated that outsourcing may lead to overstated costs – for some DHB’s again from the TD approach, this was 29% for doctors in Southland.
• “We decided not to include certain price adjusters and premia because they didn’t appear to be appropriately aligned with what we were measuring” (p14) net effect being to reduce outputs! (plus acknowledged severe limitations in the data)
• Given the small numbers of staff in some specialities, the variability in salaries, the changes in measuring FTE prior to 2006, and the large variation in rates of outsourcing and use of nursing assistants, I remain to be convinced that the pilot as a way of comparing costs and productivity across different organisations was a success.
• On such crude data, quoting means (of three!) which have ranges of over 20% is pretty meaningless.
• Is it safe to base an analysis on planned / prior purchasing agreements? If you perform your allocation before the end of the accounting period, what happens? Obviously, acute assessments and emergency admissions are less predictable that elective surgery.
• What has been the effect over the time period of any move from the public to private hospitals of high volume, low risk elective surgery? In the UK this was used to manage waiting lists, and had the effect of increasing the complexity and fragility of the patients left to be dealt with in the public system, with the more profitable, easier surgery creamed off. Are comparable figures available for the private hospitals regarding productivity of their staff?

Léonie Walker
Researcher, NZNO