Jump to ContentJump to Main Navigation
Critical Appraisal of Epidemiological Studies and Clinical Trials$

Mark Elwood

Print publication date: 2007

Print ISBN-13: 9780198529552

Published to Oxford Scholarship Online: September 2009

DOI: 10.1093/acprof:oso/9780198529552.001.0001

Show Summary Details
Page of

PRINTED FROM OXFORD SCHOLARSHIP ONLINE (www.oxfordscholarship.com). (c) Copyright Oxford University Press, 2017. All Rights Reserved. Under the terms of the licence agreement, an individual user may print out a PDF of a single chapter of a monograph in OSO for personal use (for details see http://www.oxfordscholarship.com/page/privacy-policy). Subscriber: null; date: 26 February 2017

Critical appraisal of a retrospective cohort study

Critical appraisal of a retrospective cohort study

(p.433) Chapter 13 Critical appraisal of a retrospective cohort study
Critical Appraisal of Epidemiological Studies and Clinical Trials

J. Mark Elwood

Oxford University Press

Abstract and Keywords

This chapter presents an example of the application of the scheme for critical appraisal: a retrospective cohort study entitled ‘Cancer mortality in workers exposed to chlorophenoxy herbicides and chlorophenols’, published in The Lancet in 1991. Assessed on its own, this study has produced results which are consistent with causality, but are not convincing, as other explanations cannot be excluded. Taken with the background of several other studies which also suggest an increase in this rare condition with this particular exposure, the overall evidence becomes much more persuasive.

Keywords:   cohort studies, causal relationships, cancer mortality, chlorophenoxy herbicides, chlorophenols

This study was published in The Lancet, 26 October 1991, 338, 1027–1032 [1], and is available at www.thelancet.com. The abstract, introduction, and part of the methods section of the paper are reproduced here, with permission from Elsevier and the first author.

Cancer mortality in workers exposed to chlorophenoxy herbicides and chlorophenols

Rodolfo Saracci, Manolis Kogevinas, Pier-Alberto Bertazzi, Bas H. Bueno de Mesquita, David Coggon, Lois M Green, Timo Kauppinen, Kristan A. L’Abbé, Margareta Littorin, Elsebeth Lynge, John D Mathews, Manfred Neuberger, John Osman, Neil Pearce, Regina Winkelman.


Epidemiological studies have revealed an increased risk of cancer, notably soft-tissue sarcomas and non-Hodgkin’s lymphomas, in people occupationally exposed to chlorophenoxy herbicides, including those contaminated by 2,3,7,8-tetrachlorodibenzo-p-dioxin (TCDD). We report here a historical cohort study of mortality in an international register of 18 910 production workers or sprayers from ten countries.

Exposure was reconstructed through questionnaires, factory or spraying records, and job histories. Cause-specific national death rates were used as reference. No excess was observed in all-cause mortality, for all neoplasms, for the most common epithelial cancers, or for lymphomas. A statistically non-significant two-fold excess risk, based on 4 observed deaths, was noted for soft-tissue sarcoma with a standardised mortality ratio (SMR) of 196 and 95% confidence interval (CI) 53–502; this was concentrated as a six-fold statistically significant excess, occurring 10–19 years from first exposure in the cohort as a whole (SMR = 606 [165–1552]) and, for the same time period, as a nine-fold excess among sprayers (SMR = 882 [182–2579]). Risks appeared to be increased for cancers of the testicle, thyroid, other endocrine glands, and nose and nasal cavity, based on small numbers of deaths.

(p.434) The excess of soft-tissue sarcomas among sprayers is compatible with a causal role of chlorophenoxy herbicides but the excess does not seem to be specifically associated with those herbicides probably contaminated by TCDD.


Chlorophenoxy herbicides have been used extensively since the mid-1950s for control of weeds and for removing unwanted brush on non-crop land. In the 1960s an equal mixture of 2,4-dichloro and 2,4,5-trichloro phenoxyacetic acids (agent orange) was heavily used in South Vietnam and Cambodia for defoliation by the US armed forces. Since 1969 in several industrialised countries, production and use of some compounds, and especially of 2,4,5-trichlorophenoxy acetic acid (2,4,5-T) and it derivatives have been reduced or banned. Chlorinated phenols are intermediates in the production of those chlorophenoxy herbicides and are also used directly for wood preservation. Both groups of compounds may be contaminated during the production process with polychlorinated dioxins and furans, including tetrachlorodibenzo-p-dioxin (dioxin, TCDD), which is a widespread contaminant of the general environment.

Studies of cancer risks have revealed excesses for soft-tissue sarcoma and non-Hodgkin’s lymphoma in populations exposed to chlorophenoxy herbicides, chlorinated phenols, dioxins, and furans during manufacture and spraying or after accidents. In 1987 an International Agency for Research on Cancer (IARC) working group concluded that there was ‘limited’ evidence of human carcinogenicity for chlorophenoxy herbicides and chlorinated phenols. A recent paper, focussing on exposure in chemical plants to ‘dioxin’, reported excesses for cancer of the respiratory tract and soft-tissue sarcoma, but could not exclude the contribution of smoking and other exposures in the workplace. We present here the first detailed mortality analysis of a large international cohort of workers (the International Register of Workers Exposed to Phenoxy Herbicides and their Contaminants) set up by the IARC in association with the US National Institute of Environmental Health Sciences (NIEHS). Results for some cohorts in the register have been reported earlier but for different follow-up periods.

Materials and methods

Study population

The register incorporates information on 17 372 male workers, 1537 female workers, and 1 of unknown sex, distributed among twenty cohorts from ten countries. Since publication of the register population minor corrections have led to the exclusion of 62 workers found to be ineligible. Workers from one (p.435) British company which both produced and sprayed herbicides have been separated here into two cohorts (14 and 20).

The register includes workers ever employed in production or spraying, except in the cohorts from Australia, Canada, and New Zealand, in which minimum employment periods of 1 year, 6 months, and 1 month, respectively, were specified. Eligibility of cohorts depended on the completeness at company level of records identifying workers and on the ability to ensure tracing rates of 95% or more. Follow-up for mortality was based either on computerised national record systems or on active follow-up procedures. Additional information on cases of soft-tissue sarcoma and non-Hodgkin’s lymphoma cases was sought from medical records and cancer registries. Denmark, New Zealand, Finland and Sweden provided incidence data from population-based cancer registries. Person-years at risk were calculated from 1955 onwards since only from that year were cancer-specific mortality rates available for all participating countries. Excluded from the analysis were 220 workers who either died or were lost to follow-up before 1955, 105 workers with unknown year of first exposure, 131 with unknown year of birth, 1 with unknown sex, and 63 with other missing information. The 18 390 workers included comprise 16 863 males and 1527 females, and 307 488 person-years at risk were accumulated, with an average follow-up of 17 years. Workers lost to follow-up constituted 5% of the total cohort, and in no individual cohort did this proportion exceed 10%.

Exposure assessment

Questionnaires were constructed for factories producing chlorophenoxy herbicides or chlorinated phenols and for spraying cohorts. These were completed with the assistance of industrial hygienists, workers, and/or factory personnel. Industry and other production records were also used. Job histories were examined when available. Workers were classified as exposed, probably exposed, exposure unknown, or non-exposed.

Exposed workers (n = 13 482) comprise all known to have sprayed chlorophenoxy herbicides and all who had worked in any of the following departments at factories producing chlorophenoxy herbicides or chlorinated phenols: synthesis, finishing, formulation, packing, maintenance/repair, laboratory, chemical effluent/waste, cleaning, shipping/transportation/stores/warehouse, plant supervision, cleaning during accident, and other and unclassified exposure.

Probably exposed workers (n = 416) comprise all workers in cohorts 15 and 18; no job titles were available but it was judged that most workers would have been exposed. Exposed and probably exposed are aggregated for some analyses.

Workers with unknown exposure (n = 541) had no information on exposure status.

(p.436) Non-exposed workers (n = 3951) were those never employed in the parts of factories which produced chlorophenoxy herbicides or chlorinated phenols and who never sprayed chlorophenoxy herbicides came mainly from Australia, Denmark, Netherlands, New Zealand, and UK (224, 1966, 1283, 214, and 180, respectively). Although not exposed to chlorophenoxy herbicides they were exposed to other chemicals, such as dyes and rodenticides.

Workers were also categorised as producers (12,492) and sprayers (5898). Exposed and probably exposed workers were also classified by groups of chemicals produced or sprayed (9377 chlorophenoxy herbicides, 408 chlorinated phenols, and 4113 both) and within the manufacturing cohorts by department (3034 main production, 1522 maintenance and cleaning, 1665 other, 1907 unclassifiable). A substantial number of workers producing chlorophenoxy herbicides may also have exposed to chlorophenols (e.g. p-chloro-o-cresol) which are used as raw materials in the synthesis of chlorophenoxy acids. Exposure to the most toxic dioxin congener (2,3,7,8-TCDD) may occur during production of 2,4,5-TCP and 2,4,5-T, and during spraying of herbicides containing 2,4,5-T or its derivatives. Ten factories (n = 6845) had either not produced 2,4,5-T or had produced very little of it (around 10 tonnes per year during the study period). Workers in these factories were exposed to various chlorophenoxy herbicides, chlorinated phenols, polychlorinated dibenzodioxins, and furans but were probably not exposed to TCDD. Thus within the register population it was possible to differentiate workers probably exposed to TCDD and workers probably not exposed to TCDD.

Statistical analysis

The person-years method was used to derive standardised mortality ratios (SMR) with 95% confidence intervals (CI) based on the Poisson distribution. An excess or deficit of an SMR is regarded as statistically significant at p < 0.05 (two-tail) when the CI does not include 100. The WHO Mortality Data Bank was used to compute national mortality reference rates for sex, age (in 5-year age groups), and calendar period (in 5-year periods, except when such a period coincides with an ICD revision). Duration of exposure was treated as a time-dependent variable in the allocation of person-years at risk. Poisson regression analysis was applied for selected sites.


Coding of underlying cause of death was done nationally. A conversion table has been prepared in IARC to allow the pooling of results over different ICD revisions. About 40% of soft-tissue sarcomas (one of the a priori neoplasms of interest) develop in parenchymal organs and are not coded under ICD 171 (8th and (p.437) 9th revision, ‘malignant neoplasms of connective and other soft tissue’). Histological diagnoses were not generally available for sarcomas not coded as ICD 171. SMRs for soft-tissue sarcoma therefore relate only to sarcomas coded as ICD 171. Information on other sarcomas, identified from medical records and cancer registration, is presented but no statistical analysis was done.

A. Description of the evidence

  1. 1. What was the exposure or intervention?

  2. 2. What was the outcome?

  3. 3. What was the study design?

  4. 4. What was the study population?

  5. 5. What was the main result?

The objective was to assess various possible effects of a specific exposure: chlorophenoxy herbicides. These chemicals have been widely used since the 1950s for controlling unwanted plant growth, and formed a component of the notorious ‘agent orange’ used in the Vietnam War for defoliation. The closely related chlorinated phenol compounds are intermediates in the production of these herbicides, and are also used themselves for wood preservation. Both sets of compounds may be contaminated during production by a range of other compounds, including tetrachlorodibenzo-dioxin (TCDD). This information is given in the introduction to the paper, which also gives more detailed references. This study is based on exposure to these compounds, defined through holding an occupation likely to involve such exposure.

The outcome was mortality, from all causes, with particular attention given to mortality from specified types of cancer.

Being exposure based, this is a cohort study, and is a retrospective (historical) cohort study. The study population was identified through current and past employment records, which were linked to mortality records to give outcome information.

The study population were workers identified through the ‘International Register of Workers Exposed to Phenoxy Herbicides and their Contaminants’, which was set up by an international and a US group. This consisted of 20 separate cohorts representing different employers, workplaces, and countries, involving in total 18 390 workers (16 863 male, 1527 female) from 10 countries. The derivation of the study participants is shown in Ex. 13.1.

Information on where each employee in the register worked and when, and a more detailed job history for most workers, was available. Job histories were categorized by levels of exposure, using the experience of industrial hygienists and workers. The employees were grouped into ‘exposed’workers, who had jobs (p.438)

                      Critical appraisal of a retrospective cohort study

Ex. 13.1. Derivation of the cohort of workers contributing to the analysis of Saracci et al. [1]

regarded as very likely to involve exposure to chlorophenoxy herbicides or chlorinated phenols, ‘probably exposed’ workers, which included all workers in two cohorts where no detailed job information was available, a group of workers with ‘unknown exposure’, where there was no information, and a group of ‘non-exposed’ workers. The last group comprised workers who had been employed in workplaces within the registry, but who had never been employed in the parts of the factories which produced the chemicals, and had never sprayed them; these workers are regarded as being non-exposed to the chemicals in question, although they were exposed to other chemicals. Therefore the study has an external control group, shown by the comparison of observed (p.439) mortality with the mortality expected on the basis of national data, and also an internal comparison, comparing different categories of exposure.

Results for total mortality, major causes of mortality, and cancer mortality are presented. For this review, we will concentrate on the result that is emphasized by the authors, which is mortality from soft-tissue sarcoma. This is emphasized because previous studies had demonstrated excesses of incidence or mortality of soft-tissue sarcoma in relationship to exposure to these chemicals, so that the hypothesis of an increased mortality rate was established a priori, and the study is testing that hypothesis. For many of the other results, the study is a hypothesis generation study, as the findings are unanticipated.

The mortality of the identified cohorts is compared with the expected mortality based on national mortality statistics by sex, age, and calendar period for the country relating to the individual cohort. For each cause of mortality the observed number of deaths is compared with the expected number, the ratio being the standardized mortality ratio (SMR). This is expressed in percentage terms, the null value being 100 and higher values showing an excess mortality. Ninety-five per cent two sided confidence intervals are calculated.

The main result with regard to soft-tissue sarcoma (Ex. 13.2) is that for exposed and probably exposed workers combined; four deaths were observed compared with 2.04 expected, giving an SMR of 196, with 95 per cent confidence limits of 53 to 502. For the non-exposed workers, there were no observed deaths, compared with 0.42 expected; the SMR of zero has confidence limits of 0 to 878. Therefore the main result is an excess mortality rate from soft-tissue sarcoma in workers exposed to chlorophenoxy herbicides or chlorinated phenols. We need to assess whether this association is likely to be due to a causal relationship.

B. Internal validity: consideration of non-causal explanations

6. Are the results likely to be affected by observation bias?

As the key result with regard to soft-tissue sarcomas is dependent on the observation of four deaths, observation bias and observation error are both relevant. If even one or two true occurrences of death from this disease have been missed, or if some of the deaths recorded are erroneous, the results could be greatly affected.

In terms of bias, the essential issue is whether the recognition and recording, which includes the classification of the death using the International Classification of Diseases, was done in an identical manner in these workers as in the general population (with reference to the external control comparison), and (p.440)

                      Critical appraisal of a retrospective cohort study

Ex. 13.2. Main results for deaths from soft-tissue sarcoma: from the retrospective cohort study of Saracci et al. [1]. Results taken from Table III of the paper, SMR = standardized mortality ratio (100 = null result); TCCD = 2,3,7,8-tetra chlorodibenzo-p-dioxin

whether it was done in an identical fashion in groups defined with different levels of exposure (for the internal comparison). Bias compared with the general population could arise if this disease or others like it were thought to be more common in such workers: if so, there could be greater clinical awareness for this condition, specific diagnostic procedures might be more likely to be done, if it did occur it could be more likely to be recorded on the clinical record or on a death certificate, and an ambiguous record might be more likely to be coded to sarcoma. A study of this issue done now would need to take account of such potential biases, as the possible association between herbicide exposure and (p.441) soft-tissue sarcomas is now widely recognized. However, the observation period for mortality in this study largely preceded such interest. The period of observation varies with the different cohorts: the starting dates range from 1955 to 1975, and the cut-off dates range from 1982 to 1988. The earliest suspicion of a link, which can be judged from the extensive list of references given in this paper, appears to be from a case–control study published in 1979. It seems unlikely that this suspicion would have affected clinical practice or death certification until some years after that. The potential bias could be addressed by seeing if the excess mortality was confined to deaths occurring in the later years, when knowledge of the potential hypothesis would be more widespread. An analysis by calendar year of death would of course have to be appropriately adjusted for the associations between year of death, duration of exposure, and year since first exposure; no such analysis is presented here. Therefore while observation bias remains a theoretical possibility, it is unlikely to be a major issue.

The question of the error in the reported mortality from soft-tissue sarcomas is of greater significance. These cancers are not particularly easy to diagnose, and they are not very clearly defined in clinical or pathological terms; therefore some deaths from soft-tissue sarcomas could have been inadequately described on a death certificate and coded to some other category. It is less likely that any of the deaths recorded as being from soft-tissue sarcomas were in fact from something else. This type of inaccuracy, occurring without bias, would produce some misclassification of the outcome, and would weaken the power of the study to detect a real effect; this could mean that the observed result is an underestimate of the true relationship.

7. Are the results likely to be affected by confounding?

In assessing potential confounders, we need to ask the two general questions. Apart from these herbicides, what factors are known to increase or decrease the risk of death from soft-tissue sarcoma? What other exposures are likely to be associated with exposure to these chemicals? The first question is not addressed in the paper, and so further review of the literature is necessary. This is rather unproductive. Apart from the general confounders of age, sex, and country (which are controlled in this study because the expected numbers of deaths are based on analyses specific to those factors), little is known about causal factors for soft-tissue sarcoma. The disease is not known to be related to smoking, which is often a confounding factor in occupational studies. In terms of factors associated with the exposure, the most obvious are other chemical exposures. The study participants have worked in jobs which involve using or manufacturing chemicals, and are very likely to have been exposed to many chemicals. This is discussed in the paper, and the authors indeed conclude (p.442) that their attempt to relate risks to specific agents was ‘only partially successful’ (discussion, paragraph 1). They argue that the most specific exposure to chlorophenoxy herbicides would be in sprayers, rather than production workers. The results show that the excess mortality was confined to sprayers, with an SMR of 297 based on three deaths, whereas the SMR in production workers was not increased, being 97 based on one death.

There is information, based on the job history, with regard to whether exposure was likely to have been to phenoxy compounds, chlorophenols, or both, and whether exposure to TCDD was likely. There were no deaths in those exposed only to chlorophenols, with the excess mortality being seen both in those exposed to phenoxy compounds (three deaths, SMR 200) and in those exposed to both (one death, SMR 208). There was no difference in the SMR by likely exposure to TCDD (Ex. 13.2).

The information with regard to spraying versus production work, and the information on likely individual chemical exposure, suggest that the association is primarily with phenoxy herbicides, rather than with chlorophenols or TCDD. Given the limitations of the classification of exposure and the very small numbers, this conclusion can only be tentative.

Potential confounding factors can also be identified by examining other mortality outcomes. For example, if the exposed workers had increased mortality from cancers related to smoking, it would suggest that they smoked more than the general population. If, in addition, soft-tissue sarcoma were related to smoking, smoking would be a confounder. In fact, neither of these relationships is supported. In this study, the SMR for lung cancer in exposed workers was not increased, being 102, and review of the literature shows that sarcomas are not known to be related to smoking, The pattern of other mortality is difficult to interpret because of the small numbers. Results are presented for 29 different cancer sites, of which 17 have SMRs above 100 and 12 below; there is nothing in the pattern which suggests any other causal agent which could be a confounder in the association with soft-tissue sarcomas.

8. Are the results likely to be affected by chance variation?

The increased SMR in exposed and probably exposed workers is not statistically significant, being 196, with a 95 per cent confidence interval of 53 to 502. In other words, the result is compatible with a mortality rate from soft-tissue sarcomas of approximately half that of the general population, up to a rate of five times. This demonstrates the weakness of cohort studies in dealing with rare outcomes; this study involved nearly 19 000 workers and was a major logistic effort, but for this particular outcome it is a very weak study. The effective (p.443) sample size depends on the four observed deaths. Therefore we need to keep chance variation as a quite likely explanation of the overall result.

The SMR is the ratio of the observed to expected numbers of deaths from a particular cause, as noted in Chapter 6. The statistical assessment is based on the Poisson distribution, which is appropriate for rare events. The variance of the number of events on this distribution is equal to the expected number, so a chi-squared statistic (one degree of freedom) testing the difference in an SMR from the null value of 1 or 100 is given by (Obs – Exp)2/Exp; for the data on sarcoma χ2 = (4 – 2.04)2/2.04 = 1.88 ; P = 0.17 from Appendix Table 15. Confidence limits are much more useful, and formulae for them are given, with a useful discussion, by Checkoway et al. [2]. These formulae will produce the confidence limits published in this paper. The test-based method of calculating confidence limits is not advisable on such small numbers.

C. Internal validity: consideration of positive features of causation

9. Is there a correct time relationship?

The first criterion for the time relationship is that exposure to these chemicals must have occurred before the onset of the sarcoma. Workers were included in the register and classified as exposed from the first year of the study or from the time they began a job falling into that exposure classification; in three of the cohorts minimum employment periods of 1 month, 6 months, and 1 year were specified. Once enrolled, their time experience contributed to the person-years at risk denominator, continuing until death or the end of the follow-up period. The relevant biological effect, causing the sarcoma, could predate death from the sarcoma by several years. Therefore, in those workers who develop a sarcoma, exposure after causation has been added erroneously to the denominator, and this gives an error. However, the number of workers to whom this applies is so small that it would make no difference to the results. Some of the sarcomas may indeed have originated prior to first chemical exposure, or prior to the date of entry into the cohort. These issues are addressed in analyses which look at the time dimensions of the study. If the association is causal, we would not expect death to occur within a few years of first exposure; it would happen after a time interval representing the induction time of the cancer plus the interval from cancer development to death. We might expect risk to rise steadily with time since first exposure or, as an alternative, postulate that risk would peak at a certain time after exposure and then subsequently decrease. In fact, there were no deaths within 0–9 years after first exposure, and all four deaths occurred between 10–19 years, with no deaths in (p.444) the categories of 20–29 years and over 30 years since first exposure. The expected numbers show that there were substantial person-years of follow-up in these categories. The 10–19 years time-frame is compatible with a causal relationship, although the very small number of observed deaths prevents any firm conclusion about the time-specificity of the risk relationship.

10. Is the relationship strong?

This relationship is not particularly strong; the overall effect is a doubling of risk, four observed deaths compared with two expected. Of course, it is likely that the observed SMR, if there is a true causal effect, is an underestimate, because the categorization of exposure must have a great deal of error. The exposure is based on a job history, and for some of the groups of workers is simply employment as no specific job history was available. Within those categorized as exposed there are probably many workers with minimal exposure, and perhaps only a small group with very high levels of exposure. The SMR relates to the average exposure of this overall group. The fact that it is not particularly large means that such a result could occur through a modest amount of error or bias in the observation of death from soft-tissue sarcoma, the effect on an unidentified confounding factor, or, of course, chance variation. As all four deaths occurred in one particular category of year since first exposure, 10–19 years, for this interval there were four observed deaths and 0.66 expected, giving an SMR of 606. This is statistically significant, with a lower 95% confidence interval of 165, but as this is a post hoc choice of category, undue emphasis should not be put on this result.

11. Is there a dose–response relationship?

Dose–response relationships have been assessed in a number of ways in this study. First, results are presented for workers regarded as exposed or probably exposed (combined), and also for those regarded as non-exposed, with the relevant SMRs being 196 and zero. All four deaths in the combined exposure group were in exposed workers. Although this difference could be due to chance, it is consistent with causality. Similarly, those classified as sprayers have an increased SMR, with no increase being seen in production workers. This is consistent with causality if the level of exposure to these chemicals was higher in sprayers than in production workers. It is not clear whether this is true. The authors argue that the sprayers were more likely to be exposed specifically to these chemicals rather than to other chemicals, but they do not comment on the relative intensity of exposure of sprayers and production workers.

The other measure of dose–response is duration of exposure, and we would expect that the risk would increase with duration of exposure. In particular, those workers who had a very modest time of exposure might show little (p.445) increase in risk. Duration of exposure was categorized into four groups, from less than 1 year up to 20 or more years. Two of the deaths occurred in workers with 10–19 years of exposure, giving an SMR of 690, and the other two deaths occurred in workers with less than 1 year of exposure, giving an SMR of 339. Thus these data do not show a dose–response relationship; indeed, the occurrence of two of the four deaths in workers with less than 1 year of exposure tends to argue against these chemicals being the cause of all four deaths.

12. Are the results consistent within the study?

Consistency in this study is assessed by the factors regarded as indicating high levels of exposure; these have been discussed. The number of outcome events is too small to assess other relevant aspects of consistency, for example by country, or in men and women.

13. Is there any specificity within the study?

A particular association with soft-tissue sarcomas would be more convincing if it were strong, and if there were no other excesses from other causes of death. In the workers categorized as exposed, there are several causes of death with SMRs higher than 200, and they are a mixed group. They include cancers of unspecified digestive organs, nose or nasal cavities, the male breast, the testes, the thyroid gland, and other endocrine glands, and there is a high SMR for benign and unspecified neoplasms. There are high SMRs for several other cancer sites in workers classified as probably exposed. Therefore the association with sarcoma is not specific in terms of the outcome. Specificity in terms of the exposure has been discussed; it appears to be specific to phenoxy herbicides.

Conclusions with regard to internal validity

So what can we conclude about the internal validity of this study with regard to the association with sarcoma? The study seems unlikely to be open to systematic observation bias, but unsystematic error in the outcome could influence the study results. The classification of exposure is open to considerable error, reducing the power of the study to show a real association. Confounding by exposures to other chemicals seems a possible explanation of the results, and so little is known about the causation of this disease that unrecognized confounding factors cannot be ruled out. Chance variation is a quite likely alternative explanation of the results seen, as the overall result is not significant at conventional levels. Certain subcategorizations of the results are significant, and the authors put considerable stress on the SMR of 606 obtained by only assessing 10–19 years from first exposure, and the SMR of 882 by using (p.446) the same criterion plus restricting to sprayers rather than production workers. While both of these examples of specificity can be justified, these justifications are essentially after the fact. Specific associations at this time period and in this particular group of workers were not clearly specified as a priori hypotheses to be tested, and so the statistical significance of these selected results has to be viewed with caution. In general, therefore, this study on its own cannot be regarded as pointing unequivocally towards a causal explanation, as the possibilities of observation bias, confounding, and chance variation all remain.

D. External validity: generalization of the results

14. Can the study results be applied to the eligible population?

Generalization of the results from the participant population, those who were followed up so that mortality could be determined, to the eligible population, those employees in workplaces cooperating with the international registry, seems without problems; the participation rate was 98.4 per cent (Ex. 13.1). The main issue is whether large numbers of workers within these cohorts were lost to follow-up, and so did not contribute to the results for the whole time period. In the methods section, the loss to follow-up overall is given as about 5 per cent of the total cohort, and did not exceed 10 per cent in any particular constituent cohort. This is a very high follow-up rate.

15. Can the study results be applied to the source population?

In this study the source and eligible populations are basically the same, if we can accept that all eligible male workers in the work forces in the study were enrolled in the documented cohorts. Problems would have arisen if workers for whom exposure could not be categorized were excluded, but this was not done; such workers were maintained in the study, and results for this ‘unknown exposure category’ are also shown.

16. Can the study results be applied to other relevant populations?

The generalization of the results to other target populations is more difficult. We need to review the objective of the study, and we can consider it as either pragmatic or explanatory. A pragmatic objective is to assess if occupational exposure to these chemicals is associated with an increase in the mortality from soft-tissue sarcoma, which clearly has implications for management, control, and industrial compensation. The relevant issue is whether these (p.447) exposed workers are representative of a wider population of exposed workers. If the workplaces contributing to this international register had particularly high levels of exposure, the association seen, even if true, might not apply to workers in other workplaces with considerably lower levels of exposure. If the workplaces studied had particularly low exposures, perhaps by having particularly good industrial hygiene standards and practices, the results could underestimate the problem. As production of these compounds has been restricted or banned in recent years (as stated in the introduction) current industrial exposures may be more stringently controlled and exposure levels may be lower than they were in the workplaces studied here.

We could argue that the purpose of the study is explanatory—to assess whether there is a causal relationship between these chemicals and soft-tissue sarcoma in general scientific terms. In this argument, we view these occupational groups not as representing occupationally exposed subjects in general, but as a natural experiment to observe the results of in human subjects. Whereas the pragmatic objective would be best fulfilled if the exposures of these workers were typical of a wider workforce, this is not relevant to an explanatory objective; indeed, selection of workplaces with the highest exposure levels would maximize the power of the study.

The other limitations, applying to both interpretations, are in gender and geography. The participants were mainly men, and so the association, if valid, might not apply to women. The workforces represented are predominantly of European ethnic backgrounds; the results might not apply to other ethnic groups. These issues would need to be explored if the results from this study were much more striking.

E. Comparison of the results with other evidence

On its own, this study is rather unconvincing. Its generalizability, at least to adult males and with regard to the levels of exposure experienced in workplaces, are probably quite reasonable. The great weakness is the internal validity of the study. The association found is of modest size and could well occur through observational bias or error, confounding, or chance variation; none of these non-causal explanations can be ruled out confidently. Therefore the consistency with other evidence is critical.

17. Are the results consistent with other evidence, particularly evidence from studies of similar or more powerful study design?

We would expect any individual study of this question to be weak, because of the difficulties of studying a rare disease and a particular chemical exposure which is (p.448) usually accompanied by other chemical exposures. Therefore the consistency of a large number of studies of different designs and different situations is crucial. Indeed, a meta-analysis of all available studies would give the best guidance on the topic. In the introduction to this paper it is stated that several studies had shown excesses of soft-tissue sarcoma linked to these or closely related chemicals. Given the rarity of soft-tissue sarcoma, a major problem with all studies will be small numbers, and because of this there could be a major problem of publication bias. A small study showing an excess may be published, but a small study showing no excess might not be published or even submitted for publication.

18. Does the total evidence suggest any specificity?

This total evidence does suggest a specific relationship between exposure to these chemicals and soft-tissue sarcoma, although associations with other diseases such as non-Hodgkin’s lymphoma, which is also a tumour of non-epithelial tissue, have been suggested. Of course, a small increase in other more common diseases, such as lung cancer, would be even more difficult to detect.

19. Are the results plausible in terms of a biological mechanism?

Plausibility is quite important. It is relevant to know if these chemicals have carcinogenic actions, as tested on cell systems or animal models. The authors state that this evidence has been considered by the International Agency for Research in Cancer, which has a system of assessing evidence for carcinogenicity, which concluded that there was ‘limited’ evidence of human carcinogenicity. In other words, the relationship is regarded as plausible by a well-respected international multidisciplinary group.

20. If a major effect is shown, is it coherent with the distribution of the exposure and the outcome?

The argument of coherence is relevant only if the exposure in question is a major cause of the outcome, so that the occurrence of exposure and outcome would be linked, and this link would be detectable. A relative risk of the order of 2, as suggested here, means that this is not likely, although of course if real, this risk is likely to be an underestimate. A coherence argument predicts that the mortality from this disease would have increased since the 1950s when these herbicides were introduced. However, a time trend analysis of a rare condition which is difficult to classify is open to changes in diagnostic classification patterns, as well as to the influences of unidentified causal factors. It could also be argued that these sarcomas should be more common in populations with heavier exposures, which might include rural or agricultural populations, (p.449) or specific workforces, and these issues could be explored in descriptive data sets looking at mortality by occupation or place of residence. It is unlikely that any of these types of analyses would add greatly to the arguments for and against causality, and they are much weaker methods than the cohort study presented here.


Assessed on its own, this study has produced results which are consistent with causality, but are not convincing, as other explanations cannot be excluded (Ex. 13.3). Taken with the background of several other studies which also suggest an increase in this rare condition with this particular exposure, the overall evidence becomes much more persuasive. It appears from the literature reviewed in this paper that there are a considerable number of studies supporting this association. However, the balance of these reports may be influenced by publication bias. The association is plausible on the grounds of the potential carcinogenicity of these chemicals in non-human systems. Further consideration would have to assess the other published studies with a similar critical appraisal approach. The best method to assess the human evidence would be a systematic meta-analysis, in which considerable efforts would have to be made to identify relevant sets of data which might not have been published in easily accessible sources.

Subsequent development

When published, this study was the most powerful human study of exposure to these chemicals. Even before this study, in 1986, the International Agency for Research on Cancer (IARC) had classified chlorophenoxy herbicides as having ‘limited evidence of carcinogenicity in humans’, group 2B in their classification, on the basis of case–control studies and limited cohort studies, together with strong animal and laboratory evidence of carcinogenicity of these compounds and the frequent contaminant TCDD [3]. This ruling still applies.

The cohort study has been expanded and continued, and a further report shows an SMR of 2.0 for soft-tissue sarcoma, based on six deaths, and of 1.12 for all cancers, both significantly increased, and also shows increases, although not significant, in mortality from non-Hodgkin’s lymphoma and lung cancer [4]. The major new finding was that these increased risks appeared to be restricted to workers who were exposed to TCDD or similar contaminants; those exposed to phenoxy herbicides with minimal or no contamination showed only a small excess of soft-tissue sarcoma (SMR 1.35) and no excess of (p.450)

                      Critical appraisal of a retrospective cohort study

Ex. 13.3. Summary of assessment of the retrospective cohort study of workers exposed to chlorophenoxy herbicides and chlorophenols [1].

(p.451) other cancers. However, these results are based on small numbers, with only two deaths from soft-tissue sarcomas.

A study of the female workers in this international cohort population has also been carried out [5], which is of particular interest because the animal experimental work on TCDD has suggested that it is a much more potent carcinogen in female than in male rats. No overall excess cancer risk was found in the 701 female workers followed up, but there was a significantly increased incidence (standardized incidence ratio 222) for total cancer in the subgroup exposed to herbicides contaminated with TCDD. There were no cases of soft-tissue sarcoma or non-Hodgkin’s lymphoma in the female workers. Non-cancer mortality has also been studied in this international cohort of workers, finding an increased rate of ischaemic heart disease in workers exposed to TCDD and similar contaminants [6].

Another study used record linkage methods to assess cancer in the children of workers at Canadian sawmills who may have been exposed to chlorophenol fungicides [7]. Over 19 000 children were identified, but there were only 40 cases of cancer, and no significant increased risk was shown.

Case–control studies have shown associations between chlorophenol exposure and cancers of the nasal sinuses [8] and nasopharynx [9], and have suggested some specificity in subtypes of sarcomas associated with chlorophenols [10].


Bibliography references:

1. Saracci R, Kogevinas M, Bertazzi P-A, et al. Cancer mortality in workers exposed to chlorophenoxy herbicides and chlorophenols. Lancet 1991; 338: 1027–1032.

2. Checkoway H, Pearce NE, Kriebel D. Research Methods in Occupational Epidemiology (2nd edn). Oxford: Oxford University Press, 2004.

3. International Agency for Research on Cancer. Occupational exposures to chlorophenoxy herbicides. IARC Monogr Eval Carcinog Risk Chem Hum 1986; 41: 357–406.

4. Kogevinas M, Becher H, Benn T, et al. Cancer mortality in workers exposed to phenoxy herbicides, chlorophenols, and dioxins. An expanded and updated international cohort study. Am J Epidemiol 1997; 145: 1061–1075.

5. Kogevinas M, Saracci R, Winkelmann R, et al. Cancer incidence and mortality in women occupationally exposed to chlorophenoxy herbicides, chlorophenols, and dioxins. Cancer Causes Control 1993; 4: 547–553.

6. Vena J, Boffetta P, Becher H, et al. Exposure to dioxin and nonneoplastic mortality in the expanded IARC international cohort study of phenoxy herbicide and chlorophenol production workers and sprayers. Environ Health Perspect 1998; 106 (Suppl 2): 645–653.

7. Heacock H, Hertzman C, Demers PA, et al. Childhood cancer in the offspring of male sawmill workers occupationally exposed to chlorophenate fungicides. Environ Health Perspect 2000; 108: 499–503.

(p.452) 8. Zhu K, Levine RS, Brann EA, Hall HI, Caplan LS, Gnepp DR. Case–control study evaluating the homogeneity and heterogeneity of risk factors between sinonasal and nasopharyngeal cancers. Int J Cancer 2002; 99: 119–123.

9. Mirabelli MC, Hoppin JA, Tolbert PE, Herrick RF, Gnepp DR, Brann EA. Occupational exposure to chlorophenol and the risk of nasal and nasopharyngeal cancers among U.S. men aged 30 to 60. Am J Ind Med 2000; 37: 532–541.

10. Hoppin JA, Tolbert PE, Flanders WD, et al. Occupational risk factors for sarcoma subtypes. Epidemiology 1999; 10: 300–306.