Review Article

Understanding bias in the medical literature: With reflections on metastasectomy

Fergus Macbeth, Angela Webster

Received: 13 July 2020; Accepted: 20 July 2020; Published: 23 Sept. 2020

Copyright: © 2020. The Author(s). Licensee: AOSIS.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Abstract

Background: Despite the effect of 25 years of evidence-based medicine, the problem of bias is still prevalent in clinical research and its publication and in clinical practice. Its effect can lead to flawed research, misleading publications and ultimately patient harm.

Aim: To draw attention to the commonest types of bias and how they influence clinical research, thinking and practice.

Methods: This is not a systematic review but draws on the authors’ personal experience as clinical researchers, teachers, systematic reviewers and as arbiters of conflicts of interest for Cochrane. We describe the ones most relevant to oncology and give examples mainly from the literature on pulmonary metastasectomy.

Results: There are two broad kinds of bias: technical bias, seen in the way research is conducted and published, and cognitive bias, the way in which beliefs, previous experience and thinking influence practice. The examples illustrate how common and diverse they are.

Conclusion: These biases are widespread and influential and may actually cause harm. We are all susceptible to them and need to recognise them in ourselves and others and in what we read.

Keywords: oncology; research publications; technical bias; cognitive bias; pulmonary metastasectomy.

Introduction

The evidence-based medicine movement has been with us for over 25 years. Although it has led to significant changes in the way research into new oncology interventions is carried out and its findings analysed, presented and reviewed, there is still widespread misunderstanding about how pervasive and influential the problem of bias is in its reporting and discussion. There are two broad types of bias: technical bias and cognitive bias. Although there are well-established tools for assessing bias in clinical research publications (such as the Cochrane risk of bias tool),¹ they mainly address technical biases and do not consider cognitive ones. In discussing these, we will give examples mainly from the research and publications on liver and lung metastasectomy.

The removal of lung metastases has been practised since the 1970s. It started in the management of osteosarcoma and soft tissue sarcoma, mainly because in those conditions the lung was often the first and only site of metastasis, the patients were usually young and systemic treatments were not very effective. But the intervention has never been tested in a randomised trial, and its clinical effectiveness in prolonging life, though widely believed, has never been confirmed. Over the past 20 years, it has been increasingly used in the management of carcinomas, initially colorectal cancer (CRC) and more recently in others such as lung and breast.

Liver metastasectomy has been part of the management of CRC for many years despite there never having been a randomised trial to show an effect on overall survival (OS). In fact, leading surgeons have declared that such a trial was unnecessary and indeed unethical because the benefit was so obvious.²

The concept of ‘oligometastatic’ disease has appeared and is now widely discussed even though it has no discernible basis in tumour biology or agreed definition.³ Now that minimal access surgical techniques, stereotactic radiotherapy (SABR) and image-guided thermal ablation (IGTA) are increasingly available, removing or ablating lung metastases has become more widespread and perhaps less risky.

Technical biases

Observational studies of interventions

Almost all oncology journals contain a majority of articles reporting observational studies of interventions – Phase I and II pharmacological trials, retrospective and prospective case series, case-control studies and clinical audits. Even the increasingly fashionable ‘big data’ studies are in this category. These are all prone to the following biases:

• Selection bias: This is perhaps the commonest and most influential bias. The patients receiving the intervention being studied are always selected because they have certain clinical characteristics that are thought to make them suitable. Those characteristics may be obvious and well recorded but may be subjective and poorly (or not at all) documented. There may also be unknown characteristics that have prognostic relevance. Any historical or contemporary population used as a comparator when reporting the results is very unlikely to have the same distribution of those characteristics and so will almost certainly have a different prognosis for the key outcomes. Only a robust randomisation process can ensure that these characteristics are distributed equally between two groups being compared.
• Secular trends: When using historical controls or referring to the outcomes in previous studies for comparison, there may well be bias as a result of changes in clinical practice in the time between the observation of the study and control populations. These changes may affect the results of the observed outcome for reasons that have nothing to do with the intervention of interest. Similar problems occur when a comparator population is drawn from a study in a different country or healthcare system where clinical practice or general population characteristics may not be similar and affect the outcome.
• Immortal time bias: This is a phenomenon in which the intervention group appears to have better outcomes through a design flaw, because the intervention group experiences a period of time where they cannot develop the outcome of interest (they are ‘immortal’). In observational studies, the follow-up period starts after a specific event or intervention, and the subjects will inevitably have lived for a certain time before they start being followed, without experiencing the outcome of interest (for instance, disease progression or death). Any potential participants who developed the outcome after the intervention but before entering the study would not be included. The comparator population will not necessarily experience that initial outcome-free time interval and a proportion of them will go on to experience the outcome in that time (i.e. have their disease progress or die). So, the proportion experiencing the outcome over a given time of observation will appear worse in the comparator group than in the intervention group, through this immortal time bias artifact.⁴

All these biases are seen in the observational studies of pulmonary metastasectomy for patients with CRC. Patients are selected for the intervention on the basis of what are well described prognostic factors such as the time from initial diagnosis and treatment to the appearance of lung metastases, the number of metastases, the absence of metastases at other sites and carcinoma embryonic antigen (CEA) level, as well as more general indicators such as performance status and fitness for surgery. The comparator population, whether historical or contemporary, is very unlikely to have a similar distribution of these important factors and will almost certainly have a worse prognosis. This can be seen clearly in a recent report from Korea in which 105 patients with CRC lung metastases were studied retrospectively.⁵ Those who underwent lung metastasectomy had significantly better survival than those who did not, but clearly had a better prognosis. They had significantly fewer metastases, which were more likely to be unilateral and in a single lobe and they also had a shorter disease-free interval and a lower CEA level – all well-defined prognostic factors. So, drawing the conclusion that the metastasectomy could be the main reason for the longer survival is misleading.

The systemic treatment of metastatic CRC has improved significantly in the past 20 years with better survival outcomes. So, any comparison of the survival of recent patients with historical series is likely to be subject to bias from this secular trend.

Randomised controlled trials

Randomised controlled trials (RCTs) are rightly considered to be superior to observational studies as sources of evidence for the effectiveness of interventions. This is because the process of randomisation (if properly done) should eliminate both selection and immortal time bias. Any differences between the treatment groups for known and unknown prognostic factors should be randomly distributed between the arms of the trial and the only differences affecting the outcomes should be the various treatment options to which the groups were randomised. Unfortunately, RCTs are rarely perfect and can often show bias of various kinds. A careful reading of the Methods section and inspection of the flow chart are sometimes needed to spot these. There are also well-developed and validated tools for analysing the risk of bias in RCTs – such as the Cochrane Risk of Bias 2 tool.⁶

The most important biases are as follows:

• Design bias: The RCT may be designed in a way through its choice of primary outcome, use of ancillary treatment or timing of assessments, which favours one treatment rather than another. Also, the comparator treatment may not be the current standard or ‘best’ practice, for instance, a drug regimen known to be less effective.
• Randomisation process: Although a trial may claim to be randomised, there may be problems with the actual process of randomisation such that factors other than pure chance may affect which intervention the subject is allocated to. Ideally, randomisation should be done by a remote, computerised system, and this is the way most multicentre RCTs are run now. But simpler methods, such as the use of pre-prepared, sealed envelopes, are acceptable provided the process cannot be corrupted. The important thing is that the clinician seeing the patient and registering him or her for the trial does not know in advance what treatment the patient will be allocated to. If they do somehow know, they may decide for themselves whether or not the next allocated treatment is appropriate for that subject and whether they should enter the trial. This allocation bias can occur if simple techniques such as using the patient’s date of birth or hospital number for the randomisation process are used or if the envelope is not truly opaque and the allocation can be seen under strong light. Allocation bias can result in the patient groups being dissimilar for important prognostic factors and should be cross-checked if the various groups are not very similar in total number and the distribution of key characteristics. With small RCTs, the play of chance may lead to an uneven distribution of important prognostic factors, and this can be minimised by ‘stratification’ or ‘minimisation’ but this depends on the most important prognostic factors being selected, measured and known.
• Differential or incomplete follow up: Sometimes, by design or by accident, the different groups in an RCT are followed up differently or incompletely. If that happens, there is a risk that differences in the frequency or timing of the recording of key outcome measures could lead to apparent but misleading differences in those outcomes.
• Crossover: It is not uncommon in RCTs of cancer interventions for the participants in the control arm of the trial eventually to get the intervention of interest. If so, the use of OS as a primary outcome becomes less meaningful, although progression-free survival (PFS) until the time of crossover may still be useful. But it may still be hard to extrapolate any differences to mean that the intervention actually benefits the patient.
• Detection bias (blinding): If an important outcome in a trial requires a subjective assessment (such as reporting on X-ray imaging or a patient interview) those carrying out that assessment could be biased by knowing which treatment the patient had actually received. This can be prevented by ensuring that the assessments are done by independent people who do not know (are blinded to) the treatment allocation.
• Reporting bias: If an RCT investigated several different outcomes and only some of those are actually reported, there may well be bias. It is likely that the results showing a favourable effect will be reported rather than those that did not.
• Publication bias: It is an unfortunate feature of clinical journals that it is in their interest to publish research findings that appear to support novel interventions in preference to those which show no effect. This means that the so-called negative trials may be published in less high-profile journals or not at all.

Surprisingly, now for such a widespread intervention, there have been relatively few RCTs investigating the value of metastasectomy using ‘no treatment’ controls. This may well be because of strong belief in the findings of the inevitably biased observational studies. The few RCTs that have been published are all small and demonstrate some of the above biases.

Some trials have, such as that of Gomez et al., used PFS rather than OS as the primary outcome of interest.⁷ It is hardly surprising that if all detectable diseases are removed or ablated in one arm of the trial but not in the other, then disease progression will be delayed in that arm. But that does not mean that the patients will live any longer or benefit symptomatically, especially if the metastases were asymptomatic.

Only three randomised trials have investigated OS after the removal, irradiation or thermal ablation of metastases. The CLOCC trial investigated the use of IGTA in patients with liver metastases from CRC; the SABR-COMET trial, the use SABR on metastases at a variety of sites; and the PulMiCC trial, the use of surgical metastasectomy in patients with lung metastases from CRC.^8,9,10

Both the CLOCC and SABR-COMET trials were small (119 and 99 patients, respectively) Phase II studies and appeared to show a survival benefit. But in both cases, there was an obvious imbalance in the distribution of one key prognostic factor – the proportion of patients with a solitary metastasis – favouring the intervention arm. In the SABR-COMET trial, the intervention arm also had a higher proportion of patients with breast and prostate cancer, likely to have a better prognosis. These were not the result of any deliberate manipulation of the randomisation process but because they were both small trials with inappropriate stratification.

Overall survival at 30 months was the primary outcome in the CLOCC trial, and there was no difference seen. But there was an apparently dramatic effect on OS (hazard ratio [HR] 0.58, 95% confidence interval [CI] ¼ 0.38 to 0.88) seen on prolonged follow-up when only 11 patients (out of 119) were still alive. The SABR-COMET trial appeared to show an effect on OS but this did not reach statistical significance (HR 0·57, 95% CI 0·30–1·1). Both of these trials have been claimed to show a clear benefit from intervention but careful reading shows that they do not.

The PulMiCC trial was designed to recruit 380 patients but, probably because of a lack of equipoise in the clinical teams, only managed to include 93 patients before being closed. The two arms were well balanced for prognostic factors, and it showed no effect on OS (HR 0.82 [95% CI 0.43, 1.56]).

It is clearly important for RCTs of interventions to report the adverse effects. The CLOCC trial failed to do that other than to mention one IGTA-related death. We do not know how many patients were harmed or had to remain in hospital as a result of the intervention. The SABR-COMET trial reported adverse events and 29% experienced grade 2 or worse toxicity from SABR, and there were three treatment-related deaths.

Publication bias is seen by the fact that both the CLOCC and SABR-COMET trials were published in relatively high impact oncology journals (JNCI and Lancet, respectively), whereas PulMiCC was turned down by several journals before being published initially in a lower profile, more general open access journal, trials.¹¹

Cognitive biases

These are the biases that inform the way in which we all think, act and write. In his book ‘Thinking Fast and Slow’, the Nobel Prize winning psychologist Daniel Kahneman describes two ways of thinking¹²: System 1, operating quickly and automatically, and System 2, requiring mental effort, concentration and deliberate judgement. Cognitive biases particularly affect System 1 thinking and are well known in the world of psychology, business management and in relation to medical decision-making.¹³ They are less well recognised in their effects on medical research and publishing, and it is perhaps surprising that they should influence a field theoretically dominated by System 2 thinking. Although mainly seen in opinion articles (editorials, reviews, commentaries, letters, etc.), they can also influence the way in which research studies are designed and reported. Many different types of cognitive bias have been described and are summarised in the Cognitive Bias Codex.¹⁴ These biases tend to overlap and reinforce each other but the most relevant ones in this context can be broadly considered as:

Confirmation bias: This is the tendency to look for and find results, reports and opinions that confirm one’s prior beliefs and to ignore or downplay those that do not.

Availability bias: Events that are unusual or recent tend to be recalled more often and have more persuasive power than those that are common, less visible or less memorable. This bias is often seen in case reports and observational studies with inadequate controls, and it also influences the way we all think. The relatively unusual patients who are surviving and apparently disease-free after metastasectomy will be recalled more easily and their outcome will be given more weight than that of the many, now unseen or unremembered, patients who have died.

Optimism bias: This is often seen in the field of project management where complex projects almost always over-run time and budget because of inappropriate optimism in their planning. Health professionals are always looking for new treatments to improve the care of their patients. This is a natural and generally helpful trait. But it can result in bias if new treatments are welcomed overenthusiastically and evidence of their effectiveness not evaluated carefully. If an RCT shows a small but not statistically significant difference in survival favouring a new treatment, this finding will often be described as ‘promising’ or ‘provocative’ implying that the author believes there really is a difference even though it was not proven. The results of clinical trials are often described as ‘positive’ or ‘exciting’ if they show evidence that a new intervention is effective and ‘negative’ or ‘disappointing’ if they do not. These are value-laden descriptions, whereas research findings are neutral and, if valid, should be given equal value whatever they show.

Authority, repetition and publication biases: When statements are made by or attributed to acknowledged authorities and then repeated widely, they gain a validity that may not be justified. Authority bias also results from research being published by authors with an acknowledged reputation or in the highest profile medical journals. There is, in general, a correlation between the quality and importance of research reports and commentaries and the citation index of the journal in which they are published, but this is not guaranteed. There are plenty of examples of flawed research being published in leading journals, often as a result of optimism bias and poor peer review. Expert peer reviewers because of their own biases and conflicts of interest may (unwittingly or deliberately) look more favourably on research reports that support their own beliefs and be more critical of those that do not.

Conflicts of interest: Most journals now expect authors to list the commercial organisations from which they have received personal payment or research funding. The assumption, borne out by good evidence, is that these payments may bias the authors. So it is important that readers are aware of them.¹⁵ The powerful, but usually undeclared, professional and academic conflicts of interest may also influence the way in which research reports, ‘expert’ reviews and commentaries are written.

Cultural bias: We all are products of the society we were brought up in and now live in and of the clinical culture we work in. These all influence us and bias the way in which we write and read clinical research.

Most of these biases are obvious in the way that metastasectomy has been promoted and become widely used in clinical practice despite the lack of objective evidence from RCTs that it improves OS or quality of life, and despite the fact that there is a clear risk of harm. If these various interventions had been new anticancer pharmaceuticals, they would have had to go through extensive testing and rigorous evaluation before being licensed and approved for use nationally and locally. This appears to be unnecessary for interventional procedures, which can become widely used and accepted without formal appraisal for clinical effectiveness.

New developments

With the increased understanding of tumour biology and genomics, there has been an explosion in the number and types of targets for drug therapy in oncology. This has led in turn to an increasing number of new agents that need to be tested in clinical trials. There is pressure to investigate therapies more efficiently and rapidly to reduce delay in translating new evidence into practice for the benefit of patients. This has led to innovation in trial design. Adaptive design allows pre-specified changes to the trial protocol as the trial progresses, without compromising validity or integrity.¹⁶ Multiple adaptations may be used, including modifying interventions, populations enrolled or the randomisation method. Platform trials investigate multiple interventions at the same time, and basket trials investigate one intervention in multiple populations at the same time. Whilst innovative methods and impressively swift trial timelines are positive, adaptive design trials are subject to the same kind of biases as traditional randomised trials. The statistical considerations can be more complex, and the necessity to pre-plan and specify design and analysis can be difficult. Interpretation of results is more challenging, and bias can play a part at each step.

Conclusion

Bias of all kinds is widespread throughout the medical literature. This is probably inevitable, but it probably shapes modern clinical research and practice more than we are aware of or prepared to admit. There are several important lessons to be drawn from this:

• Always be sceptical and open-minded and engage in ‘System 2’ thinking, both when your beliefs or received wisdom are challenged and also when they appear to be confirmed.
• Read the Methods section of any article very carefully and look out for the important technical biases that may be influencing the reported results and conclusions. There will almost always be some, but they may not be obvious.
• Be very cautious if a publication may change your and your colleagues’ practice. Be aware of your own biases and prior beliefs.
• Do not fall victim to confirmation bias.

Acknowledgements

The authors acknowledge the help and useful comments from Professor Tom Treasure.

The authors have declared that no competing interests exist.

F.M. conceived and drafted the article, and A.W. contributed and approved the final version.

This article followed all ethical standards for carrying out research.

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

Data sharing is not applicable to this article as no new data were created or analysed in this study.

The views and opinions expressed in this article are those of the authors and do not necessarily reflect the official policy or position of any affiliated agency of the authors.

References