Jump to ContentJump to Main Navigation
Where's the Evidence?Debates in Modern Medicine$

William A. Silverman

Print publication date: 1999

Print ISBN-13: 9780192630889

Published to Oxford Scholarship Online: September 2009

DOI: 10.1093/acprof:oso/9780192630889.001.0001

Show Summary Details
Page of

PRINTED FROM OXFORD SCHOLARSHIP ONLINE (www.oxfordscholarship.com). (c) Copyright Oxford University Press, 2019. All Rights Reserved. An individual user may print out a PDF of a single chapter of a monograph in OSO for personal use.  Subscriber: null; date: 17 September 2019

A ‘win’ in medical Russian roulette

A ‘win’ in medical Russian roulette

Chapter:
(p.196) 45 A ‘win’ in medical Russian roulette
Source:
Where's the Evidence?
Author(s):

William A. Silverman

Publisher:
Oxford University Press
DOI:10.1093/acprof:oso/9780192630889.003.0045

Abstract and Keywords

This chapter presents a 1997 commentary on the extracorporeal membrane oxygenation (ECMO) trial for neonates with respiratory failure. The ECMO experiences shows that there are significant national differences in the factors and dynamics that come into play in the spread of high-cost, high-tech, high-risk modern interventions in medicine. It addresses the question: How can we balance the technical challenge of perfecting hardware for life-support against medicine's humane imperative to avoid needless pain and suffering?

Keywords:   randomized trials, clinical trials, medical practice, extracorporeal membrane oxygenation, neonatal care

Nothing exceeds like excess.

Anon

A large-scale randomised trial of extracorporeal membrane oxygenation (ECMO) for neonates with respiratory failure has been carried out in Britain, and early results were reported recently.1 The long-awaited outcome of this study was published a full twenty years after the first successful adaptation of cardiopulmonary bypass for infants was described in the US in 1976.2 During these two decades there were unrelenting arguments about whether or not there was enough evidence to justify wide use of the aggressive new technique.

The ECMO experience is a reminder that there are significant national differences in the factors and dynamics which come into play in the spread of high-cost, high-risk, high-tech modern interventions in medicine. The protracted period of controversy also calls attention to the kind of problems that turn up when the success of a new intervention depends on the skill of an individual doctor or on the smooth co-ordination of a highly-trained and experienced team of doctors, nurses, and technicians. The hard-won success of the experts at the end of a difficult learning period (their ‘courage to fail’) leads to a predicament that might be called ‘technological entrapment:’ Favourable local experience leads to intensely-held subjective convictions; any questions raised about the interpretation of results are taken as a personal affront. In the development of ECMO (and other ‘desperation technologies [that] attract unusually positive assessment since they offer hope where there was no hope’3), it is easy to lose sight of an overriding question: How can we balance the technical challenge of perfecting hardware for life-support, as against medicine’s humane imperative to avoid needless pain and suffering?

(p.197) Following the introduction of the new technique in the US in the late 1970s, the survival rate of ECMO-treated infants improved dramatically, in association with increased practice in use of the procedure for desperately-ill full-term neonates. Activists argued that the effectiveness of ECMO was self-evident. A registry of ECMO-treated infants reported that 393 infants received this heroic treatment by the end of 1985; at the end of 1989 the number treated climbed to 3 577 patients—83% of infants reported to the registry survived.4 But sceptics pointed to the difficulties of linking cause and magnitude of effect without concurrent controls at a time when overall neonatal survival rates were rising; and, more specifically, when survival with conventional ventilator treatment of respiratory failure was also increasing.5 7

The first randomised trial of ECMO was reported in 1984,8 nine years after the procedure was introduced; five years later a second trial in the US was reported.9 The favourable results in both of these exercises were questioned because too few patients were enrolled to provide reliable estimates of effect-size; and questions were also raised about the interpretation of the adaptive study-designs used (prior-outcome-dependent treatment allocation). A standard design was used in the third trial,10 but, again, relatively few patients were enrolled, and the favourable results of that study were reported only in form of an abstract. Additionally, concerns were raised about numerous early and late adverse complications of the complex treatment, and about the inadequate number of concurrent conventionally-managed controls for follow-up.11

These cranky doubts about the ECMO programmes had little effect on slowing the spread of the technique in the US. On the contrary, there was external pressure to proceed forthwith. Four months before the results of the second randomised trial were published in a medical journal, the researchers were accused publicly of unethical conduct by denying ‘life-saving therapy’ (ECMO) to enrolled infants allocated to conventional management.12

In 1990, the Committee on Fetus and Newborn of the American Academy of Pediatrics acknowledged that ‘new ECMO centers are evolving on the basis of current enthusiasm and without a thorough appreciation of the complexity, intensity, potential hazards, and uncertainties of this form of therapy.’13 Curiously enough, the Committee recommended that the technique should be carried out only at major centres with ‘active research programs,’ and yet, nothing was said about the need for additional large-scale controlled trials. In the same year, the evidence then extant and the ethical issues involved in further clinical research of the topic were reviewed:14 The authors listed a number of features in the ECMO experience that made it difficult, in their opinion, to evaluate the technique prospectively in controlled trials. They concluded that the experience with the development of this heroic treatment demonstrated the weaknesses of formal comparative trials as the (p.198) ultimate basis for the resolution of medical disagreements. And they posited several values that would need to be sacrificed to attain greater certainty offered in randomised trials; among these is respect for the right of physicians to offer what they believe to be the best available treatment for their patients.

What was largely ignored in the debates about the use of ECMO in the US, was the role of market forces in the diffusion of this procedure which generated both profit and prestige at a time when competition between hospitals was heating up in that country. Ann Lennarson Greer, a sociologist, likened the hectic activity to an ‘arms race’.3 She pointed out that the need to transfer patients to another institution is often perceived as a threat by local doctors. ‘… [I]t is possible to imagine,’ she noted, ‘that neonatologists at a non-ECMO center might fear loss of referrals to a NICU [neonatal intensive care unit] having ECMO backup,’ this leads to pressure for the establishment of local centres, each hoping to attract more patients. Moreover, the proliferation of the specialised centres in the US is not solely in the hands of doctors. ‘Under market competition,’ Greer noted, ‘there is increased pressure [from managers] to direct funds to areas that enhance the hospital’s visibility, market share, or potential to tap new markets.’ Valerie Miké, a biostatistician, also questioned the quality of evidence used to justify the rapid diffusion of ECMO in the US.15 She and her coworkers observed that the push for highly-visible development may also come from booster efforts of local government. For example, they came across a magazine advertisement in May 1990 paid for by the Board of City Development of an American city: ‘Our ability to save struggling newborns with an ECMO Unit,’ the ad boasted, ‘makes [our city] one of the healthiest medical communities in the nation.’

Given the permissive views about rules of evidence in the development of desperation technologies and the operation of competitive market forces in the US, it is not surprising that ECMO has flourished in that country without further large-scale rigorous evaluation. Miké’s group observed that ‘of the more than 6 000 treated patients treated by mid-1992, only 40—less than 1%—were evaluated in a randomized trial.’

The spread of neonatal ECMO in Britain was, by contrast, very restrained. (In 1994–95, provision of ECMO centres was more than two and a half times more plentiful in the US as compared with the UK. In the US [1994], there were 75 ECMO centres16 / 3 979 000 births17 = ca 19 centres per 1 million births; compared with the UK’s 5 centres [England, Scotland, and Wales] / 708 189 births, equivalent to 7 centres per 1 million births, A. Johnson, personal communication.) The UK Collaborative ECMO trial group explained1 that the new technology was first used in Britain in 1989, but concerns about long-term disability, high costs, and uncertainty about effectiveness acted to slow acceptance. These fears led to an agreement to limit use only within a national (p.199) randomised trial, and to define the primary outcome-of-interest as either death or severe disability (the latter to be assessed at the age of 1-year). Recruitment began on January 1, 1993 and was halted on November 23, 1995 when a decision was made that the trial provided a clear answer after the enrolment of 180 infants.

Although all survivors were not yet reviewed at age-1-year, the preliminary data suggested that rate of death or severe disability in the ECMO-treated group will be about one-half that in conventionally-managed babies (relative risk of outcome 0.54 [95% CI 0.36–0.80]). Since the preliminary rate of impairment among survivors was similar in the two groups (ECMO 11/43 [26%] vs conventional management 7/24 [29%]) it may turn out that the actual number of impaired survivors will be greater after ECMO treatment because of lowered mortality.

Roger Soil, an American neonatologist,18 has praised the UK’s ECMO centres for taking a more cautious approach than their US counterparts in adopting the controversial procedure. This is an encouraging sign, because it will be unfortunate if the British trial is condemned as an unethical effort to achieve little more than the ‘greater certainty’ denigrated in the US, six years earlier.14 And it will be equally deplorable if the ECMO story is used to disparage a stern and time-honoured warning: Delay in mounting large-scale trials of a new treatment is a dangerous game. The medical profession would do well to remember that even in the admittedly hazardous game of Russian roulette with a six-shooter, in a long series of plays, 5-out-of-6 spins of the gun’s cylinder are expected to end in a ‘win’ for a gullible player—but when the 1-in-6 loss occurs, it is devastating.

Have we already forgotten the lesson taught by the hapless use of DES (diethylstilbestrol) over a period of 20 years, to improve pregnancy outcome?19 That unhappy episode demonstrated just how catastrophic a loss can be when ‘greater certainty’ is eschewed out of respect for the prerogative of physicians to offer what they believe to be the best available treatment.

If there are doubts that such mishaps (the medical analogues of ‘friendly fire’ disasters in conventional warfare) can and do still occur, the uncritical optimism was dispelled by some recent gloomy news. The safety of pulmonary artery (PA) catheterisation for monitoring haemodynamic events in seriously-ill patients has been questioned. (More than 1-million PA catheters have been sold each year in the US, since the specially-designed balloon-tipped devices were introduced in 1970.) The technique, it turns out, was never subjected to rigorous tests. For example, an RCT, attempted in 1991, was terminated because most doctors were so convinced the procedure was safe they refused to allow their patients to be randomised. Now, after 26 years of an enormous world-wide experience, a large prospective observational cohort study (of 5 735 critically ill patients receiving care in an intensive care unit in (p.200) five US teaching hospitals between 1989 and 1994) has found a measurable increase in mortality associated with PA catheterisation.21 And there is now a clamour for a proper randomised trial of the procedure.22

The more things change, sadly, the more they are the same.

Reply by John D. Lantos:

(1997)

Disabuse: To present your neighbour with another and better error than one which he has deemed it advantageous to embrace.

Ambrose Bierce’s Devil’s Dictionary

Silverman sets about to disabuse readers of misperceptions they may have gained from reading an essay that Joel Frader and I wrote in 1990.1 In particular, Silverman worries that we might condemn as unethical efforts to achieve greater certainty than was available through careful analysis of retrospective studies and small prospective studies. He is also concerned that we did not adequately consider the role of market forces in the diffusion of ECMO in the United States.

By 1993, when the UK Collaborative ECMO trial was initiated, it had been seventeen years since the first successful use of ECMO in neonates.2 Analysis of three small randomised trials had concluded that ECMO was superior to maximal ventilatory support for infants with meconium aspiration and persistent pulmonary hypertension, congenital diaphragmatic hernia or pneumonia and sepsis when those infants met rigorous criteria for severe respiratory failure. More than 7 500 babies had been treated with ECMO in 75 programmes in the United States and 17 programmes in other countries.3 Nevertheless, physicians in the UK were uncertain whether the data on the efficacy of ECMO were convincing, and set about to do another randomised clinical trial.

Whenever we consider a randomised trial, we must ask whether we are in a state of genuine uncertainty about the relative merits of the two arms of the trial. If we are not, then it is unethical to randomise patients. Instead, we have an obligation to offer the therapy which is more beneficial. If we are genuinely uncertain, than a randomised trial is not only permissible but may be obligatory as a way of maximising the chances that our patients get the best available treatment. Randomisation, though always problematic, may thus sometimes be the best approach for both the treatment of patients and the evaluation of an innovative therapy. But not always.

Whenever anybody wants to make an argument for greater use of randomised controlled trials, they cite the diethylstilbestrol (DES) (p.201) debacle. This is a bit misleading. A randomised controlled trial of DES done at the University of Chicago showed that DES was ineffective in preventing premature labour,4 and this was part of the reason why the use of DES declined. The trial did not include vaginal adenocarcinoma in the offspring of treated women as an endpoint. Most trials do not examine outcomes in the offspring of treated patients. The association between DES and vaginal adenocarcinoma was discovered using a retrospective case control study design.5 Even randomised trials have limitations. We live in a world where knowledge is always imperfect, where innovation is always associated with peril. One of those perils is the overuse of randomised trials.

In his book on the evaluation of thrombolytic therapies, Brody6 gives a striking example of the unethical use of randomised trials. Long after intracoronary streptokinase had been shown to be efficacious in reducing mortality from acute myocardial infarction, studies continued to be done that randomised patients to a new intervention or a placebo. Brody argues that such trials for thrombolytic therapy led to the avoidable and unnecessary deaths of 497 people. These 497 people were those who were given placebos in trials of intravenous thrombolytics after intracoronary streptokinase had been proven effective and had been approved by the FDA. Brody suggests that, since proven treatment was available, placebo controlled trials were unethical. Each of these studies was subsequently published in peer-reviewed journals without apparent concern about the ethics of study design. We seem to have become so enamoured of randomised trials that we can no longer see some of the problems and conflicts that they raise.

More certainty is always better than less certainty but at some point, we need to decide that we are certain enough. The UK ECMO trial makes us more certain that ECMO works, but are we certain enough? One way to think about this would be to ask whether another study to confirm the results of the UK Collaborative ECMO trial would be ethical at this point. After all, the scientific method would seem to demand that experiments be repeated to make certain that the results are valid. By this argument, we should do another study. But most doctors, I imagine, would be reluctant to participate in another trial. While not completely certain, they are certain enough. In the United States, most doctors were certain enough before the UK trial.

There will always be tensions in the development of new technologies. Market forces in the United States created an environment in which profits could be made by successfully marketing hospitals and health care systems. In a culture enamoured of high technology and the appearance of progress, successful marketing often demanded that hospitals tout the ‘latest’ medical developments, even if they were not the best medical developments. But market forces are not always evil. In Britain, by contrast, the lack of market forces and the administrative structures (p.202) for the evaluation of new therapies may delay the introduction of beneficial therapies. Market forces sometimes demand that unnecessary trials be conducted. Brody demonstrates that a number of unnecessary or unethical trials of thrombolytic therapies were conducted in both the United States and Europe primarily because different drug companies wanted to prove that their product was efficacious.

In 1990, Frader and I1 argued that randomised controlled trials involved a number of trade-offs, and that, although they are epistemo-logically pleasing in a way that other study designs are not, they may not always be necessary, desirable, or practical to conduct. At one level, our argument was simply a practical one. We can never evaluate every innovation using randomised controlled trials because, if we tried, we would always be doing trials. Innovation is a constant and intrinsic aspect of modern medicine. Most innovations are minor variations on a theme, but even they change the mix of available therapies.

At some point, we have to make a judgement about whether or not enough is known about a treatment to deem it better, worse, or about the same as another treatment. When is a therapy ever evaluated enough? When can we say, for sure, that the definitive study has been done and that further study will add no new knowledge? The answer, clearly, is never. There are always innovations, new indications, new drugs or devices that can be given in new combinations. In an epistemologically perfect world, each combination of interventions should be rigorously evaluated against each other possible combination of interventions, Every new drug or device would change the mix, so that, with each advance, testing would need to start anew. Only then would we know, without dogma or bias, what was or was not effective.

We would also be paralysed. Innovation would stop. The evaluation of innovation would become the enemy of innovation.

A deeply-held dogma is a terrible thing We would all prefer to be open-minded than to be dogmatic, to consider the evidence fairly, to be rational rather than biased. And yet, it is often difficult to disentangle dogma from conviction, belief from bias, and faith from knowledge.

Silverman’s editorial about ECMO states that early in the days of ECMO, ‘activists’ argued that the effectiveness of ECMO was ‘self-evident.’ That oversimplifies the story a bit. ECMO investigators were rigorous in their evaluation of ECMO and their publication of results. They published their eligibility criteria, their outcomes, and their complications. They refined their indications and their techniques based on collected experience, and they conducted two randomised controlled trials. Their only sin was that they became convinced of the benefits of ECMO before some statisticians were convinced, and so, felt that it would be unethical to continue to randomise patients to an inferior therapy. The question is not whether ECMO has been evaluated but whether it had been evaluated enough.

(p.203) The design of the British trial drew heavily on this accumulated experience. Techniques, eligibility criteria, and contraindications were all based on the previous world-wide experience. And the results were, predictably, the same as the results of other countries and other centres. In the UK Collaborative ECMO trial, 32% of babies in the ECMO arm died, compared with 59% of babies in the conventional arm. There were 24 ‘excess’ deaths in the conventional arm.7 I’m not sure how I would feel if I was one of the parents whose baby received conventional therapy and died. Of more concern, it is not clear, from the study report, what happened to babies whose parents did not want their babies to be in the study. Presumably, they received ‘conventional’ therapy, and presumably, the only way they could have gained access to ECMO would have been to agree to be randomised.

I would have suggested the opposite, that all the evidence to date suggested that ECMO was more effective than ‘conventional’ therapy and so, ECMO should have been the default treatment of choice. As the trial was designed, there was an implicit element of coercion in this study design.

I think that the data that were available in the early 1990s on the benefits for ECMO were convincing. I have concern about informed consent and possible coercion in the UK study. For these reasons, if I was on an ethics review panel, I would not have approved the trial.

Notes:

(1.) UK Collaborative ECMO Trial Group, 1996.

(2.) Bartlett et al., 1976.

(3.) Greer, 1993.

(4.) Neonatal Extracorporeal Life Support Registry, 1990.

(8.) Bartlett et al., 1985.

(9.) O’Rourke et al., 1989.

(10.) Bifano et al., 1992.

(11.) Elliott, 1991.

(12.) Knox, 1989.

(13.) Fetus and Newborn Committee, 1990.

(14.) Lantos and Frader, 1990.

(15.) Miké et al., 1993.

(16.) Kanto, 1994

(17.) Guyer et al., 1994.

(18.) Johnson, 1996.

(19.) Soll, 1996.

(21.) Connors et al., 1996.

(22.) Sandham et al., 1996.

(1.) Lantos and Frader, 1990.

(2.) Bartlett et al., 1976.

(3.) Kanto, 1994.

(4.) Dieckman et al., 1953.

(5.) Herbst et al., 1971.

(6.) Brody, 1996.

(7.) UK Collaborative ECMO Trial Group, 1996.