NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.
Raine R, Fitzpatrick R, Barratt H, et al. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Southampton (UK): NIHR Journals Library; 2016 May. (Health Services and Delivery Research, No. 4.16.) doi: 10.3310/hsdr04160-55
Elizabeth Gibbons , Nick Black , Lesley Fallowfield , Robin Newhouse , and Ray Fitzpatrick .
Elizabeth Gibbons , 1 Nick Black , 2 Lesley Fallowfield , 3 Robin Newhouse , 4 and Ray Fitzpatrick 1 .
1 Health Services Research Unit, Nuffield Department of Population Health, University of Oxford, Oxford, UK
2 Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, UK
3 Sussex Health Outcomes Research and Education in Cancer (SHORE-C), University of Sussex, Brighton, UK
4 Indiana University School of Nursing, Indianapolis, IN, UKA growing consensus has emerged about patient-reported outcome measures (PROMs) as an important tool in the evaluation of services. There is little dispute regarding the range of relevant dimensions of health status, the types of measures available to capture what matters to patients, their required measurement properties and reporting standards when PROMs are used in studies. Their use on a large scale in the NHS national PROMs programme has produced valuable lessons about elective surgery but also about PROMs as tool.
There are outstanding issues if PROMs are to be further applied in the evaluation of services. Patients need to be actively engaged. Health professionals need to see the merits of this approach to patient care and outcome assessment. PROMs need to be integrated into health records. Some recent initiatives have attempted to address these three needs, and further development and testing of such initiatives is needed.
A growing consensus has emerged about patient-reported outcome measures (PROMs) as important tool in the evaluation of services. There is little dispute regarding the range of relevant dimensions of health status, the types of measures available to capture what matters to patients, their required measurement properties and reporting standards when PROMs are used in studies. Their use on a large scale in the NHS national PROMs programme has produced valuable lessons about elective surgery but also about PROMs as tool.
Respondent engagement is fundamental to ensure adequate response rates to be able use outcomes for the evaluation of services. Evidence exists that patients with poorer health status at baseline impact on response rates which may produce misleading evidence of the effectiveness of services. Different mechanisms for the delivery and capture of PROMs are being developed such as electronic data collection and use of other media. This could complement existing paper-based methods and has the potential to reduce respondent burden of completion.
Specific populations pose particular challenges in engagement such as those with developmental limitations (children) and those with cognitive impairment such as patients with dementia or learning difficulties. In addition, the collection of baseline measurement in patients experiencing acute or emergency health problems poses difficulties.
Missing or incomplete data present analytical problems and, despite a range of statistical procedures used, performance of services could be influenced by methods adopted.
Evidence suggesting there is diversity in levels of staff engagement from enthusiasm to scepticism. Difficulties are apparent in relation to understanding and interpretation of the data, as well as the challenges of incorporating data into existing systems and processes of care. This results in little impact of PROMs data used for service improvement.
Further evolution in PROMs is still needed so that they focus as much on practical, feasible and clinically actionable content as on the psychometrically validated content emphasised to date. Development and testing of feasible software, platforms and electronic health records needs to be developed to support PROMs as a routine feature of both clinical care and evaluative research. Training of staff to strengthen understanding of PROMs and interpreting will be essential. Most outstanding of all will be the need to establish benefits to patients. There is much speculation about PROMs supporting and informing patients’ choice of health-care options, health and consulting behaviour and self-management, but there are very few studies to demonstrate such benefits.
Health services are under constant pressure to be more patient centred and positively valued by patients and the public. Evaluative research reflects that pressure by increasingly assessing outcomes as perceived by patients. Patient-reported outcome measures (PROMs) are a diverse array of questionnaires and related techniques intended to obtain patients’ own views of their health status and benefits experienced from receiving health services. Because of their focus on accurately capturing the patient’s perspective on outcomes, they have been seen for some time as having enormous and distinctive potential to transform the assessment of the performance of services. 1 A recent overview described measures of health status (effectively PROMs) as ‘. . . the most important scientific development in the last 50 years in the field of health services research’. 2 This essay provides an overview of the progress that has been made in the measurement of patients’ views of health status and outcomes, and identifies some of the key challenges to be faced in their further use to evaluate services.
Patient-reported outcome measures have been considered as solutions to a diverse range of problems. First developed over 40 years ago, they were early on considered as invaluable outcome measures in health services research, for example in the widely cited randomised controlled trial (RCT) of alternative mechanisms of reimbursement, the RAND Health Insurance Experiment. 3 Recent examples of the use of PROMs as outcomes in large multicentre health services research include their use to judge the benefits of personally controlled health budgets, 4 and as outcomes of telehealth services. 5 It is in the adjacent field of clinical trials that PROMs have been most widely applied, often as secondary outcomes to weigh against traditional indicators such as mortality but occasionally as the primary end point. A recent survey of nearly 100,000 clinical trials published between 2007 and 2013 found that a PROM was used in 27% of trials. 6 A specific subset of such trials includes cost–utility analysis, estimating utility or net value of a treatment by means of a particular type of PROM described Types of patient-reported outcome measure as utility measure.
Another use of the PROM is to provide alternative expressions of health need in the population. A classic example is the use of a measure, a version of the Sickness Impact Profile, to assess the epidemiology of disability in London. 7 A related application is their use as prognostic variables and impressive evidence exists for this use in areas such as cancer. 8
Patient-reported outcome measures are beginning to be used as instruments to contribute to local quality improvement developments; for example, Nilsson et al. 9 recently reviewed the use of PROMs in Swedish disease registers and found PROMs commonly used this way. There is growing interest in whether or not they have a role in improving individual patient care, for example by strengthening shared decision-making or improving the detection and management of patients’ problems. Trials to date have had mixed results. 10 , 11 As will be argued later, their acceptance in routine patient care may prove to be crucial to furthering their use in evaluative research.
The example of cancer provides compelling evidence of why validated PROMs are needed to evaluate the performance of services. Generally, health professionals show poor understanding of how cancer affects quality of life, both in regular care and in evaluative research. In clinical trials for ovarian cancer, patients reported more symptoms and reduced quality of life compared with clinician-completed toxicity scales. 12 Vera-Badillo et al. 13 reviewed a series of breast cancer trials and found both bias and under-reporting of toxicities. Di Maio et al. 14 compared the reports by patients and physicians of six specific toxicities reported by patients and physicians (anorexia, nausea, vomiting, constipation, diarrhoea and hair loss) in three RCTS for cancer. Physicians consistently under-recognised and under-reported toxicities. Such results mean not only that the benefit-to-harm ratio of treatments is often inaccurate but also that the higher burdens on patients experiencing these symptoms can lead to non-adherence or dose reductions limiting the expected clinical benefits.
Some recent developments have raised the visibility of PROMs as a distinctive lens through which to assess the quality of health services. In England, a national PROMS programme was introduced in 2009 mandating the use of PROMs for NHS patients in relation to selected elective surgical procedures. This is described further in A case study: patient-reported outcome measures and NHS elective surgery because of its very size, ambition and potential lessons.
In the USA, major impetus for PROMs has stemmed from the establishment with major funding by Congress of the Patient-Centered Outcomes Research Institute (PCORI) designed to answer key uncertainties in health-care delivery by the conduct of patient-centred comparative effectiveness research. A key feature of the work of PCORI is that patients and patients’ values should be central to every stage of health research. 15 Not surprisingly, the application and improved methods for patient-reported outcomes are therefore a priority (www.pcori.org/events/2013/patient-reported-outcomes-pro-infrastructure-workshop).
There is substantial and growing consensus about PROMs. This essay focuses on four key areas of agreement: relevant dimensions, required measurement properties, types of measure and reporting standards.
A key feature of PROMs is that they assess health in the broadest sense. Individual instruments vary in content but the domains identified in Box 4.1 16 cover most aspects of health status addressed by PROMs. In their most common form, measures of dimensions of health status, such as are shown in Box 4.1, are assessed before and after an intervention, and, strictly, it is the change over time that renders the expression of an outcome that, in principle, may be attributed to an intervention.
Dimensions of health in PROMs Reproduced with permission from Fitzpatrick et al. Contains information licensed under the Non-Commercial Government Licence v1.0’ from this link: http://www.journalslibrary.nihr.ac.uk/rights_and_permissions.
There is broad agreement about the measurement properties required of a PROM and techniques used to assess such properties. Reliability is the extent to which a PROM is free from measurement error and reproducible. Face and content validity concern basic judgements about whether or not items of a questionnaire appear to address the experiences relevant to respondents. In practice, this is commonly judged from evidence of the extent to which patients have fully contributed to the derivation and approval of content of an instrument. Construct validity is evaluated to explore the hypothetical constructs it purports to measure and ensure that there is a statistical relationship between measures of similar constructs. The measure should avoid ceiling or floor effects demonstrating precision of measurement. Consideration of the ability of the measure to detect change and ensure responsiveness is particularly important for PROMs, most usefully tested against the respondent’s own retrospective judgement of change. When used to determine the ‘value’ of treatments from an international perspective, cultural sensitivity and appropriate forwards and backwards field-testing of translations are also vital to demonstrate.
These properties are familiar from other fields of measurement in science. They are tested by a range of methods familiar from classical psychometrics as well as more recently applied techniques such as Rasch Analysis and Item Response Theory. 17
However, as PROMs are almost invariably used with patients receiving health care for real presenting problems, it is as important to consider the acceptability of PROMs along with their technical measurement properties in order to minimise burden. Minimal burden should also strengthen response rates and hence the generalisability of results.
There is broad agreement about the range of types of PROM available and their respective merits.
Generic PROMs are more commonly multidimensional and applicable across a range of conditions, and enable comparison within and between different health conditions, populations and interventions. The SF-36 (Short Form Questionnaire-36 items) is probably the most widely used such measure.
Preference-based measures, such as the European Quality of Life-5 Dimensions (EQ-5D) (www.euroqol.org/), provide a descriptive profile and index of a person’s health status in the form of a utility derived from public preferences for different health states and weighted accordingly. The distinctive role of such measures is therefore used in economic evaluations such as cost–utility analyses. They are a specific subtype of generic measure.
Disease- and or procedure-specific PROMs aim to represent particular aspects of health status relevant, for example, to patients living with diabetes or those undergoing a specific procedure, such as joint replacement. The distinctive merit of such measures is to be maximally sensitive to the particular challenges posed by a given disease and, therefore, maximally sensitive to benefits of an intervention for the condition.
Individualised measures of health status elicit patients’ personal goals and concerns regarding their health, thereby avoiding the standardised format of the other measures just described.
A common strategy in the evaluation of a specific intervention for a limited range of conditions is to recommend that an evaluation include a generic and a disease-specific measure to address the widest spectrum of anticipated and unanticipated outcomes.
The field has evolved to the extent that a range of guidelines has now been developed relevant to many aspects of PROMs and their use. Thus, several influential guidelines have emerged regarding appropriate methods and standards for the development of a PROM. 18 , 19 Other guidance focuses on how PROMs should be reported. 20 Guidance also exists on how to carry out systematic reviews of the measurement properties of PROMs. 21 Although invaluable for the development of the field, implicit in much of such guidance is a focus on clinical trials for single treatments. Much routine assessment of services needs to be based on more complex interventions delivered to heterogeneous populations, for which such guidance may be less helpful.
A number of developments have occurred that may enhance the role of PROMs in health services research. Thus, studies have begun to show that shorter versions of PROMs may have equivalent measurement properties, a development that may significantly decrease burden and increase response rates. 22 Caution must be used in methods of shortening to avoid losing key content. 23 Juniper et al. 24 proposed taking account of the frequency with which problems are reported in different items, combined with patients’ ratings of their importance, to select key items.
There may be circumstances when, for example, disease-specific PROMs have been used in the valuation of an intervention but there has been no preference-based measure, making cost–utility analyses difficult. A range of techniques have now been developed that map utility values across to non-preference-based measures; the advantage is that not only can utility values be calculated but potential additional precision of the disease-specific measure is retained. 25
Methods for interpreting scores from PROMs have advanced. The most important scores generated by PROMs are usually the changes over time from before to after an intervention, but numerical change scores of non-intuitive scales may be difficult to interpret. Traditional solutions focused purely on statistical solutions to establish minimally important differences. However, an alternative is to identify the smallest change that would be appreciated especially by patients, identified by relating change scores to patients’ retrospective judgements of how they value change. This approach, referred to as ‘anchor-based’, has the advantage of determining the meaning of PROMs from the patient’s perspective, which can still be cross-checked with more statistical approaches. 26
The content of PROMs relevant to evaluative research is also being expanded to include related valued goals and concerns of the patient. Although some may challenge the extension of PROMs beyond health status, a range of related constructs may be just as important to the patient and closely related in potential causal chains between intervention and final health outcome, for example self-efficacy and patient activation. Additionally, the boundary between PROMs and patient experience may also need to be relaxed; when, for example, patients are capable of giving meaningful answers to validated questionnaires retrospectively judging the outcomes and benefits of treatments. 27 – 29 An ongoing problem involves the phenomenon of response-shift bias, which occurs when a patient with a deteriorating condition nevertheless reports an unchanged quality of life owing to successful adaptation and a shift in the thresholds they themselves use to describe the severity or impact of a problem.
Since 2009, the providers of care for NHS patients have been required to collect data before and 3 or 6 months after surgery with a condition-specific and a generic PROM for four elective procedures: hip replacements, knee replacements, groin hernia repair and varicose veins surgery. Importantly, the information generated is publicly available (www.hscic.gov.uk/proms) and guidance is provided to help the public navigate and interpret data for individual hospital trusts. The database has provided a uniquely rich source for evaluative research using the PROM.
Some overall patterns are striking; for example, the evidence for the effectiveness of joint replacement (especially in terms of condition-specific measures) and the relative lack of evidence of provider trusts being consistently poor outlier performers in relation to appropriately adjusted PROMs are reassuring for the NHS. There was somewhat more evidence of individual surgeons performing markedly poorer, according to PROMs, compared with the much lower variation according to 90-day post-surgical mortality. 30 The national PROMs programme provide results that challenge conventional thinking. There is no support for the view that either individual surgeons or whole trusts with larger volumes of surgery have better results. 31
The programme has provided important methodological insights, showing how response rates and choice of PROM may influence the comparative performance of trusts. As discussed below, these results illustrate the outstanding challenges for future use of PROMs.
Patient-reported outcome measures are unusual compared with almost all other health indicators, in that they require active input from the patient or respondent. For example, in the NHS national PROMs programme, whereas pre-operative recruitment rates to PROMs for patients receiving joint replacement surgery have been 68%, the rate is markedly lower, 41%, in patients receiving varicose vein repair surgery, although this is largely attributable to eligible patients not being invited. 32 In a sample of patients recruited to complete PROMs for any of six long-term conditions recruited via general practices, the recruitment rate was lower again, at 38%. 33 There is some evidence that response rates have deteriorated over time; Hazell et al. 34 found marked deteriorations in response rate to an identical survey to patients about their asthma, from 71% in 1993 to 47% in 2004.
Clearly, a number of factors may influence the response rate. In the examples just cited, patients were more likely to respond to surveys about receiving a specific surgical intervention than to primary care surveys about long-term conditions where there was no link to receiving treatment. The concern is that the lower the response rate, the harder it is to use the outcomes obtained to evaluate services.
However, the greater concern is if there is evidence of response bias: if important characteristics of either patients or services are associated with differential response rate. There is evidence that this may occur for PROMs. In the national PROMs programme, Hutchings et al. 35 found that the poorer the health status in terms of comorbidities prior to surgery, the poorer the response rate to PROMs questionnaires mailed out after surgery. Similarly, in the primary care study of PROMs for long-term conditions referred to above, when patients were asked to return a follow-up PROM 1 year after the baseline, the response rate was lower in those with poorer baseline health status. 36 It is clear that such biases may produce misleading evidence of the effectiveness of services.
There are more subtle forms of problem with PROMs if questionnaires are returned incomplete, for example if more sensitive or personal questions are not completed or more difficult items are omitted. Usually the development phase of PROMs reduces such risks. There is a range of statistical techniques for addressing missing or incomplete data from longitudinal data sets such as PROMs, most commonly the use of imputation methods derived from the information about respondents that has been obtained. Analysing missing data from the national PROMs programme, Gomes et al. 37 found that inferences about the performance of services could be influenced by the assumptions and methods made to make imputations to address missing data.
To address the problem of non-response, most effort has gone into the design of PROMs and mechanisms of delivery (e.g. the use of the internet and other modern media), partly because these are practical solutions that can be implemented. The scope for innovative technology to enhance respondent engagement is significant, although, currently, for the majority of PROMs, evaluative research still relies on traditional survey methods, particularly the mailed questionnaire. Although beyond the scope of this essay, recent innovations in the core format of PROMs may also strengthen their acceptability and power to engage the respondent. The innovation is the use of computer adaptive testing to tailor questionnaire items to the individual, with responses to initially administered items determining the choice of subsequent items. The total number of items required to be completed is significantly reduced. This has been the subject of a major National Institutes of Health-funded initiative in the USA, Patient Reported Outcome Measures System PROMIS (www.nihpromis.org/), to produce questionnaire items that can populate CAT systems. Although an exciting initiative in the science of PROMs, particularly in terms of their use in assessing individuals’ health, there is little evidence to date of their use in evaluative research.
The other major concern occurs where there are major difficulties in engaging respondents, whether because of major physical, cognitive or developmental limitations or because of social exclusion. One example is the involvement of children. PROMs exist that are designed to be relevant to the perceptions of children. 38 Additional care may be required, in terms of appropriate interviewing, determining intellectual capacity, not relying solely on chronological age, and the role of observation and proxy informants. 39
Dementia poses related challenges. There are dementia-specific PROMs, which appear to work satisfactorily for patients with mild to moderate dementia. 40 However, for those with more severe levels of dementia, assessments by a proxy carer are the more plausible option. Nevertheless, proxy ratings should not be treated simplistically or uncritically because their rates may, in turn, be influenced by burden of care and carer burnout, and appropriate adjustments are, therefore, required. 41
In some areas of health care it is impossible to obtain a pre-treatment or baseline assessment for obvious reasons; patients who experience sudden health events such as a stroke or hip fracture will not have any reason to have completed such an assessment. A number of approaches may be adopted for which the patient’s perspective is needed. One approach is simply to give up on pre-event assessment and assess progress and possible impact of interventions after the sudden event and as soon as it is feasible to engage the patient. 42 Alternative strategies can include inviting the patient retrospectively to assess their pre-event health or to use appropriately matched population-based data. Neither of these two strategies is straightforward.
There is only limited evidence of the impact of social exclusion in relation to PROMs. The problem is, however, well highlighted in a study by Jahagirdar et al., 43 who found that excluded groups, such as those with learning difficulties or low literacy, were less likely to be involved in the development of PROMs for chronic obstructive pulmonary disease. The commonest pragmatic solution to social exclusion more generally is to weight evidence for under-represented groups in surveys. A recent review of the sparse evidence argues that multiple and flexible approaches are needed properly to ensure that socially excluded groups’ views are not overlooked 44 (www.pssru.ac.uk/archive/pdf/4390.pdf). The importance of social inequalities and social exclusion in health services research is further discussed in Essay 5 in this volume.
If the role of PROMs is to be expanded, it is likely to require greater engagement of clinicians so that collection of PROMs becomes more a part of routine care. Reference has already been made to evidence from trials introducing PROMs into individual patient care and showing mixed evidence of impact on management decisions, patient experience of care and ultimate health outcomes. This variable impact may, in turn, be attributable to health professionals’ attitudes, beliefs and experiences regarding PROMs; these have been the subject of some research. A recent study of surgeons’ views of PROMs found considerable diversity with a range from enthusiast to sceptic. 45 In this and other studies, a number of concerns have been expressed. Cognitive problems include the view that PROMs data are difficult to interpret and relate to management decisions. 46 , 47 Other concerns focus on logistics, time constraints and difficulties of incorporating PROMs into clinical routines. 46 – 48 These studies generally stress clinicians’ expressed need for greater training to incorporate PROMs into practice. 49 At worst, some studies suggest concerns that PROMs actually may cause harm if evidence from them is misunderstood or misused by third-party audiences such as managers, commissioners or politicians. 50 , 51
A recent survey obtained the views of relevant experts from the USA, England and the Netherlands about prospects for future use and impact of PROMs. 52 There was a clear consensus that for PROMs to have their fullest impact in assessing the performance of health services, there needed to be greater integration of information systems for routine patient care and for system performance measurement. Currently in all three countries, information systems for the two different functions are effectively independent. The experts identified a number of barriers that would need to be overcome to integrate these two worlds of activity, including lack of trust from participants and insufficient belief in the value of PROMs in patient care.
A number of reports are beginning to appear broadly supportive of the conclusions of Van der Wees et al. 52 and also express optimism that the integration of information from PROMS for patient care and system performance can be achieved. In the USA, Wu et al. 53 describe a number of health record systems already available and in use to support the two functions. They identify three developments favourable to this integration: the positive trend towards patient-centeredness and electronic applications for PROMs, the growth of electronic health records and trend towards comparative effectiveness research that is patient oriented. More recently, Jensen et al. 48 reported a number of encouraging case studies in the USA in which PROMs served both patient care and system evaluation. They effectively identify the same three generally supportive developments as Wu et al. 53
Warrington et al. 54 provide a positive account of a NHS setting in which PROMs have been successfully implemented to provide long-term follow-up of cancer survivors. They acknowledge that in a systematic review of their field of cancer care, Nama et al. 55 could find no high-quality evaluations to demonstrate clearly the benefits of such systems. This lack of evidence remains a major challenge.
Patient-reported outcome measures have an important role in evaluating services in terms of outcomes that matter to patients. They will continue to play a role as primary or secondary end points in bespoke research trials and evaluative studies in which the patient is recruited to participate as research subject. In addition, there is likely to be a larger role of patients’ routinely contributing information about their health and outcomes of their care as health-care systems adapt to take advantage of developments in informatics and PROMs. In anticipation of such developments, it has been argued that further evolution in PROMs is still needed so that they focus as much on practical, feasible and clinically actionable content as on the psychometrically validated content emphasised to date. 56 Further development and testing is needed of feasible software, platforms and electronic health records to support PROMs as a routine feature of both clinical care and evaluative research. Major issues of trust, confidentiality and sustainability remain to be addressed. As already emphasised, training to strengthen understanding will be essential. Evaluative studies will be needed to test the overall viability of electronic health-based health records that include PROMs. Most outstanding of all will be the need to establish benefits to patients. There is much speculation of PROMs supporting and informing patients’ choice of health-care options, health and consulting behaviour and self-management, but there are very few studies to demonstrate such benefits.
Elizabeth Gibbons (Senior Research Scientist, Health Services Research) and Ray Fitzpatrick (Professor, Public Health) wrote the first draft of the essay.
Nick Black (Professor, Health Services Research), Lesley Fallowfield (Professor, Psycho-oncology) and Robin Newhouse (Professor, Nursing) commented on and contributed to the draft.
Elwood P. Shattuck lecture – outcomes management. A technology of patient experience . N Engl J Med 1988;318:1549–56. [PubMed : 3367968 ]
Brook R, Vaiana M. Using the knowledge base of health services research to redefine health care systems . J Gen Intern Med 2015;30:1547–56. 10.1007/s11606-015-3298-2. [PMC free article : PMC4579238 ] [PubMed : 25840780 ] [CrossRef]
Newhouse J. A summary of the RAND Health Insurance study . Ann N Y Acad Sci 1982;387:111–14. 10.1111/j.1749-6632.1982.tb17166.x. [PubMed : 6953855 ] [CrossRef]
Jones K, Forder J, Caiels J, Welch E, Glendinning C, Windle K. Personalization in the health care system: do personal health budgets have an impact on outcomes and cost? J Health Serv Res Policy 2013;18(Suppl. 2):59–67. 10.1177/1355819613503152. [PubMed : 24048694 ] [CrossRef]
Cartwright M, Hirani SP, Rixon L, Beynon M, Doll H, Bower P, et al. Whole Systems Demonstrator Evaluation Team. Effect of telehealth on quality of life and psychological outcomes over 12 months (Whole Systems Demonstrator telehealth questionnaire study): nested study of patient reported outcomes in a pragmatic, cluster randomised controlled trial . BMJ 2013;346:f653. 10.1136/bmj.f653. [PMC free article : PMC3582704 ] [PubMed : 23444424 ] [CrossRef]
Vodicka E, Kimb K, Devinea E, Gnanasakthy A, Scoggins J, Patrick D. Inclusion of patient-reported outcome measures in registered clinical trials: evidence from ClinicalTrials.gov (2007–2013) . Contemp Clin Trials 2015;43:1–9. 10.1016/j.cct.2015.04.004. [PubMed : 25896116 ] [CrossRef]
Patrick D, Peach H. Disablement in the Community . Oxford: Oxford University Press; 1989.Gotay CC, Kawamoto CT, Bottomley A, Efficace F. The prognostic significance of patient-reported outcomes in cancer clinical trials . J Clin Oncol 2008;26:1355–63. 10.1200/JCO.2007.13.3439. [PubMed : 18227528 ] [CrossRef]
Nilsson E, Orwelius L, Kristenson M. Patient-reported outcomes in the Swedish National Quality Registers . J Int Med 2015;279:141–53. 10.1111/joim.12409. [PubMed : 26306802 ] [CrossRef]
Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: a structured review . J Eval Clin Pract 2006;12:559–68. 10.1111/j.1365-2753.2006.00650.x. [PubMed : 16987118 ] [CrossRef]
Valderas JM, Kotzeva A, Espallargues M, Guyatt G, Ferrans CE, Halyard MY, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature . Qual Life Res 2008;17:179–93. 10.1007/s11136-007-9295-0. [PubMed : 18175207 ] [CrossRef]
Greimel ER, Bjelic-Radisic V, Pfisterer J, Hilpert F, Daghofer F, Pujade-Lauraine E, et al. Toxicity and quality of life outcomes in ovarian cancer patients participating in randomized controlled trials . Support Care Cancer 2011;19:1421–7. 10.1007/s00520-010-0969-8. [PubMed : 20694564 ] [CrossRef]
Vera-Badillo FE, Shapiro R, Ocana A, Amir E, Tannock IF. Bias in reporting of end points of efficacy and toxicity in randomized, clinical trials for women with breast cancer . Ann Oncol 2013;24:1238–44. 10.1093/annonc/mds636. [PubMed : 23303339 ] [CrossRef]
Di Maio M, Gallo C, Leighl NB, Piccirillo MC, Daniele G, Nuzzo F, et al. Symptomatic toxicities experienced during anticancer treatment: agreement between patient and physician reporting in three randomized trials . J Clin Oncol 2015;33:910–15. 10.1200/JCO.2014.57.9334. [PubMed : 25624439 ] [CrossRef]
Newhouse R, Barksdale DJ, Miller JA. The Patient-Centered Outcomes Research Institute: research done differently . Nurs Res 2015;64:72–7. 10.1097/NNR.0000000000000070. [PubMed : 25502063 ] [CrossRef]
Fitzpatrick R, Davey C, Buxton MJ, Jones DR. Evaluating patient-based outcome measures for use in clinical trials: a review . Health Technol Assess 1998;2(14). [PubMed : 9812244 ]
Streiner D, Norman G. Health Measurement Scales . Oxford: Oxford University Press; 1995.Food and Drug Administration Department of Health and Human Sciences. Guidance to Industry Patient Reported Outcome Measures Use in Medical Product Development to Support Labelling Claims . Silver Spring, MD: Department of Health and Human Sciences; 2009. [PMC free article : PMC1629006 ] [PubMed : 17034633 ]
Reeve B, Wyrwich K, Wu A, Velikova G, Terwee C, Snyder C, et al. ISOQOL recommends minimum standards for patient-reported outcome measures used in patient-centered outcomes and comparative effectiveness research . Qual Life Res 2013;22;1889–905. 10.1007/s11136-012-0344-y. [PubMed : 23288613 ] [CrossRef]
Calvert M, Blazeby J, Altman DG, Revicki DA, Moher D, Brundage MD, et al. Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension . J Am Med Assoc 2013;309:814–22. 10.1001/jama.2013.879. [PubMed : 23443445 ] [CrossRef]
Terwee CB, Mokkink LB, Knol DL, Ostelo RW, Bouter LM, de Vet HC. Rating the methodological quality in systematic reviews of studies on measurement properties: a scoring system for the COSMIN checklist . Qual Life Res 2012;21:651–7. 10.1007/s11136-011-9960-1. [PMC free article : PMC3323819 ] [PubMed : 21732199 ] [CrossRef]
Jenkinson C, Clarke C, Gray R, Hewitson P, Ives N, Morley D, et al. Comparing results from long and short form versions of the Parkinson’s disease questionnaire in a longitudinal study . Parkinsonism Relat Disord 2015;21:1312–16. 10.1016/j.parkreldis.2015.09.008. [PubMed : 26360237 ] [CrossRef]
Coste J, Guillemin F, Pouchot J, Fermanian J. Methodological approaches to shortening composite measurement scales . J Clin Epidemiol 1997;50:247–52. 10.1016/S0895-4356(96)00363-0. [PubMed : 9120523 ] [CrossRef]
Juniper EF, Guyatt GH, Streiner DL, King DR. Clinical impact versus factor analysis for quality of life questionnaire construction . J Clin Epidemiol 1997;50:233–8. 10.1016/S0895-4356(96)00377-0. [PubMed : 9120521 ] [CrossRef]
Brazier JE, Yang Y, Tsuchiya A, Rowen DL. A review of studies mapping (or cross walking) non-preference based measures of health to generic preference-based measures . Eur J Health Econ 2010;11:215–25. 10.1007/s10198-009-0168-z. [PubMed : 19585162 ] [CrossRef]
Wyrwich KW, Norquist JM, Lenderking WR, Acaster S. Methods for interpreting change over time in patient-reported outcome measures. Industry Advisory Committee of International Society for Quality of Life Research (ISOQOL) . Qual Life Res 2013;22:475–83. 10.1007/s11136-012-0175-x. [PubMed : 22528240 ] [CrossRef]
Lloyd H, Jenkinson C, Hadi M, Gibbons E, Fitzpatrick R. Patient reports of the outcomes of treatment: a structured review of approaches . Health Qual Life Out 2014;12:5. 10.1186/1477-7525-12-5. [PMC free article : PMC3899626 ] [PubMed : 24422873 ] [CrossRef]
Black N, Varagunam M, Hutchings A. Relationship between patient reported experience (PREMs) and patient reported outcomes (PROMs) in elective surgery . BMJ Qual Saf 2014;23:534–42. 10.1136/bmjqs-2013-002707. [PubMed : 24508681 ] [CrossRef]
Gibbons E, Hewitson P, Morley D, Jenkinson C, Fitzpatrick R. The Outcomes and Experiences Questionnaire: development and validation . Patient Relat Outcome Meas 2015;6:179–89. 10.2147/PROM.S82784. [PMC free article : PMC4509457 ] [PubMed : 26213480 ] [CrossRef]
Varagunam M, Hutchings A, Black N. Relationship between patient-reported outcomes of elective surgery and hospital and consultant volume . Med Care 2015;53:310–16. 10.1097/mlr.0000000000000318. [PubMed : 25654295 ] [CrossRef]
Varagunam M, Hutchings A, Black N. Do patient-reported outcomes offer a more sensitive method for comparing the outcomes of consultants than mortality? A multilevel analysis of routine data . BMJ Qual Saf 2015;24:195–202. 10.1136/bmjqs-2014-003551. [PubMed : 25368319 ] [CrossRef]
Hutchings A, Neuburger J, van der Meulen J, Black N. Estimating recruitment rates for routine use of patient reported outcome measures and the impact on provider comparisons . BMC Health Serv Res 2014;14:66. 10.1186/1472-6963-14-66. [PMC free article : PMC3923248 ] [PubMed : 24511984 ] [CrossRef]
Peters M, Crocker H, Jenkinson C, Doll H, Fitzpatrick R. The routine collection of patient-reported outcome measures (PROMs) for long-term conditions in primary care: a cohort survey . BMJ Open 2014;4:e003968. 10.1136/bmjopen-2013-003968. [PMC free article : PMC3931992 ] [PubMed : 24561495 ] [CrossRef]
Hazell ML, Morris JA, Linehan MF, Frank PI, Frank TL. Factors influencing the response to postal questionnaire surveys about respiratory symptoms . Prim Care Respir J 2009;18:165–70. 10.3132/pcrj.2009.00001. [PMC free article : PMC6619256 ] [PubMed : 19104738 ] [CrossRef]
Hutchings A, Neuburger J, Grosse Frie K, Black N, van der Meulen J. Factors associated with non-response in routine use of patient reported outcome measures after elective surgery in England . Health Qual Life Outcomes 2012;10:34. 10.1186/1477-7525-10-34. [PMC free article : PMC3344679 ] [PubMed : 22462512 ] [CrossRef]
Peters M, Crocker H, Dummett S, Jenkinson C, Doll H, Fitzpatrick R. Change in health status in long-term conditions over a one year period: a cohort survey using patient-reported outcome measures . Health Qual Life Outcomes 2014;12:123. 10.1186/s12955-014-0123-2. [PMC free article : PMC4243951 ] [PubMed : 25113415 ] [CrossRef]
Gomes M, Gutacker N, Bojke C, Street A. Addressing missing data in patient-reported outcome measures (PROMS): implications for the use of proms for comparing provider performance [published online ahead of print 5 March 2015] . Health Econ 2015. 10.1002/hec.3173. [PMC free article : PMC4973682 ] [PubMed : 25740592 ] [CrossRef]
Janssens A, Rogers M, Thompson Coon J, Allen K, Green C, Jenkinson C, et al. A systematic review of generic multidimensional patient-reported outcome measures for children, part I: descriptive characteristics . Value Health 2015;18:315–33. 10.1016/j.jval.2014.12.006. [PubMed : 25773568 ] [CrossRef]
Matza L, Patrick D, Riley A, Alexander J, Rajmi L, Pleil A, et al. Pediatric patient-reported outcome instruments for research to support medical product labeling: report of the ISPORPRO good research practices for the Assessment of Children and Adolescents Task Force . Value Health 2013;16:461–79. 10.1016/j.jval.2013.04.004. [PubMed : 23796280 ] [CrossRef]
Aguirre E, Kang S, Hoare Z, Edwards RT, Orrell M. How does the EQ-5D perform when measuring quality of life in dementia against two other dementia-specific outcome measures? Qual Life Res 2016;25:45–9. 10.1007/s11136-015-1065-9. [PubMed : 26163146 ] [CrossRef]
Grske J, Meyer S, Wolf-Ostermann K. Quality of life ratings in dementia care? A cross-sectional study to identify factors associated with proxy-ratings . Health Qual Life Outcomes 2014;12:177. 10.1186/s12955-014-0177-1. [PMC free article : PMC4276020 ] [PubMed : 25495548 ] [CrossRef]
Parsons N, Griffin X, Achten J, Costa M. Outcome assessment after hip fracture: is EQ-5D the answer? Bone Joint Res 2014;3:69–75. 10.1302/2046-3758.33.2000250. [PMC free article : PMC3963508 ] [PubMed : 24648420 ] [CrossRef]
Jahagirdar D, Kroll T, Ritchie K, Wyke S. Patient-reported outcome measures for chronic obstructive pulmonary disease: the exclusion of people with low literacy skills and learning disabilities . Patient 2013;6:11–21. 10.1007/s40271-013-0004-5. [PMC free article : PMC3585908 ] [PubMed : 23417577 ] [CrossRef]
Beadle-Brown J, Ryan S, Windle K, Holder J, Turnpenny A, Smith N, et al. Engagement of People with Long Term Conditions in Health and Social Care Research: Barriers and Facilitators to Capturing the Views of Seldom-Heard Populations . Quality and Outcomes of Person-centred Care Policy Research Unit; 2012. URL: www .pssru.ac.uk/archive/pdf/4390.pdf (accessed 10 April 2016).
Boyce M, Browne J, Greenhalgh J. Surgeon’s experiences of receiving peer benchmarked feedback using patient-reported outcome measures: a qualitative study . Implement Sci 2014;9:84. 10.1186/1748-5908-9-84. [PMC free article : PMC4227108 ] [PubMed : 24972784 ] [CrossRef]
Bausewein C, Simon S, Benalia H, Downing J, Mwangi-Powell F, Daveson B, et al. Implementing patient reported outcome measures (PROMs) in palliative care – users’ cry for help . Health Qual Life Outcomes 2011;9:27. 10.1186/1477-7525-9-27. [PMC free article : PMC3112059 ] [PubMed : 21507232 ] [CrossRef]
Gilbert A, Sebag-Montefiore D, Davidson S, Velikova G. Use of patient-reported outcomes to measure symptoms and health related quality of life in the clinic . Gynecol Oncol 2015;136:429–39. 10.1016/j.ygyno.2014.11.071. [PubMed : 25448486 ] [CrossRef]
Jensen RE, Rothrock NE, DeWitt EM, Spiegel B, Tucker CA, Crane HM, et al. The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care . Med Care 2015;53:153–9. 10.1097/MLR.0000000000000289. [PMC free article : PMC4801509 ] [PubMed : 25588135 ] [CrossRef]
Santana M, Haverman L, Absolom K, Takeuchi E, Feeny D, Grootenhuis M, et al. Training clinicians in how to use patient-reported outcome measures in routine clinical practice . Qual Life Res 2015;24:1707–18. 10.1007/s11136-014-0903-5. [PubMed : 25589231 ] [CrossRef]
Wolpert M. Uses and abuses of patient reported outcome measures (PROMs): potential iatrogenic impact of PROMs implementation and how it can be mitigated . Adm Policy Ment Health 2014;41:141–5. 10.1007/s10488-013-0509-1. [PMC free article : PMC3909250 ] [PubMed : 23867978 ] [CrossRef]
Hildon Z, Neuburger J, Allwood D, van der Meulen J, Black N. Clinicians’ and patients’ views of metrics of change derived from patient reported outcome measures (PROMs) for comparing providers’ performance of surgery . BMC Health Serv Res 2012;12:171. 10.1186/1472-6963-12-171. [PMC free article : PMC3426480 ] [PubMed : 22721422 ] [CrossRef]
Van der Wees P, Nijhuis Van der Sanden M, Ayanian G, Black N, Westert G, Schneider E. Integrating the use of patient-reported outcomes for both clinical practice and performance measurement: views of experts from 3 countries . Milbank Q 2014;92:754–75. 10.1111/1468-0009.12091. [PMC free article : PMC4266175 ] [PubMed : 25492603 ] [CrossRef]
Wu AW, Kharrazi H, Boulware LE, Snyder CF. Measure once, cut twice – adding patient-reported outcome measures to the electronic health record for comparative effectiveness research . J Clin Epidemiol 2013;66(Suppl. 8):12–20. 10.1016/j.jclinepi.2013.04.005. [PMC free article : PMC3779680 ] [PubMed : 23849145 ] [CrossRef]
Warrington L, Absolom K, Velikova G. Integrated care pathways for cancer survivors – a role for patient-reported outcome measures and health informatics . Acta Oncol 2015;54:600–8. 10.3109/0284186X.2014.995778. [PubMed : 25751758 ] [CrossRef]
Nama V, Nordin A, Bryant A. Patient-reported outcome measures for follow-up after gynaecological cancer treatment . Cochrane Database Syst Rev 2013;11:CD010299. 10.1002/14651858.cd010299.pub2. [PMC free article : PMC6457831 ] [PubMed : 24249483 ] [CrossRef]
Kroenke K, Monahan P, Kean J. Pragmatic characteristics of patient-reported outcome measures are important for use in clinical practice . J Clin Epidemiol 2015;68:1085–92. 10.1016/j.jclinepi.2015.03.023. [PMC free article : PMC4540688 ] [PubMed : 25962972 ] [CrossRef]
computer adaptive testing
Patient-Centered Outcomes Research Institute
patient-reported outcome measure
Patient Reported Outcomes Measurement Information System
randomised controlled trial
Declared competing interests of authors: Robin Newhouse reports personal fees from the Patient Centered Outcomes Research Institute and being chairperson and on the methodology committee, being a board member for AcademyHealth, and grants from Agency for Health Care Research and Quality outside the submitted work.
This essay should be referenced as follows: Gibbons E, Black N, Fallowfield L, Newhouse R, Fitzpatrick R. Patient-reported outcome measures and the evaluation of services. In Raine R, Fitzpatrick R, Barratt H, Bevan G, Black N, Boaden R, et al. Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Health Serv Deliv Res 2016;4(16). pp. 55–68.
Copyright © Queen’s Printer and Controller of HMSO 2016. This work was produced by Raine et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.
