'Representative' - its too "too too" to put a finger on

Prove it... just the facts... the confidential

This case, this case, this case that I...

I've been workin' on so long...

Chirpchirp...the birds
they're giving you the words
The world is just a feeling you undertook.
Now the rose, it slows

you in such colorless clothes

Fantastic! You lose your sense of human.

Project, Protect

It's warm and it's calm and it's perfect

It's too "too too" to put a finger on

This case is closed

Prove it [Television]

I recently tweeted that a new review of meta-analyses of CBT for various disorders was biased - at least as far as the section on schizophrenia is concerned. The Efficacy of Cognitive Behavioral Therapy: A Review of Meta-analyses (Hofmann, Asnaani, Vonk, Sawyer & Fang), which is published in Cognitive Therapy and Research is undoubtedly an ambitious work born of some fortitude. And to be fair, it deserves more detail - than my original 140 characters - about why I think the review is biased.

Here is the verbatim section from the paper:

Schizophrenia and Other Psychotic Disorders

Meta-analyses examining the efficacy of psychological treatments for schizophrenia revealed a beneficial effect of CBT on positive symptoms (i.e., delusions and/or hallucinations) of schizophrenia (e.g., Gould et al. 2001; Rector and Beck 2001). There was also evidence (e.g., Zimmermann et al.2005) that CBT is a particularly promising adjunct to pharmacotherapy for schizophrenia patients who suffer from an acute episode of psychosis rather than a more chronic condition.

CBT appeared to have little effect on relapse or hospital admission compared to other interventions, such as early intervention services or family intervention (e.g., Bird et al. 2010 ; Alvarez-Jimenez et al. 2011). However, CBT had a beneficial effect on secondary outcomes. For example, a more recent meta-analysis by Wykes et al. (2008) examined controlled trials of CBT for schizophrenia and confirmed findings from previous meta-analyses (e.g., Gould et al. 2001; Rector and Beck 2001), suggesting that CBT had a small to medium effect size as compared to control conditions on both positive and negative symptoms. In addition, this meta-analysis revealed medium effect sizes for improvements in secondary outcomes that were not the direct targets of treatment, including general functioning, mood, and social anxiety.

Why do I think its biased? Largely because it is unrepresentative. In the paper, Hofmann et al state "We identified 269 meta-analytic studies and reviewed of those a representative sample of 106 meta-analyses examining CBT for the following problems:...".
I suppose its conceivable that a review may be 'representative' if it references a minority (<40%) of the meta-analyses? Presumably the 60% majority were somehow unrepresentative?

So, how were the 'representative' meta-analyses selected? The paper is surprisingly remiss on this issue - telling us nothing about how selection occured. Leaving aside problems justifying their choices, it is still possible that Hoffman et al managed to formulate a representative account of the meta analyses examining CBT for schizophrenia? They included reference to meta analyses by: Gould et al. 2001; Rector and Beck 2001; Zimmermann et al (2005); Wykes et al (2008); Bird et al. (2010); and Alvarez-Jimenez et al. (2011) - so. lets take a look at what they did select

Rector & Beck (2001) and Gould et al (2001) are the earliest and hence, the smallest published meta-analyses; and almost completely overlap (with 5 of 7 studies being the same). The early studies in these meta analyses were of poor quality and thus inflated the effect sizes e.g. Rector & Beck refer to:

"...the absence of representative sampling (Drury et al., 1996a, 1996b), the absence of blind ratings (Kuipers et al., 1997; Pinto et al., 1999), the absence of reliable and valid multidimensional symptoms measures (Drury et al., 1996a, 1996b; Tarrier et al., 1993a), the failure to control for potential between-group differences in the use of medications (Kuipers et al., 1997; Tarrier et al., 1993a), and the failure to ensure treatment adherence and competence of CBT therapists (Drury et al., 1996a, 1996b; Kuipers et al., 1997)."

Regarding the next, larger meta-analysis of Zimmerman et al (2005), which contained better studies and therefore revealed a much smaller effect size (two and half times smaller than Rector & Beck), Hoffman et al write "There was also evidence (e.g., Zimmermann et al.2005) that CBT is a particularly promising adjunct to pharmacotherapy for schizophrenia patients who suffer from an acute episode of psychosis rather than a more chronic condition." This was not true in the Zimmerman et al meta-analysis, which states that for acute and chronic patients "...this difference was not statistically significant"(p.7)

Wykes et al (2008) - Hofmann et al say "In addition, this meta-analysis (Wykes et al 2008) revealed medium effect sizes for improvements in secondary outcomes that were not the direct targets of treatment, including general functioning, mood, and social anxiety."

Wykes et al calculated effect sizes for these variables according to whether they rated studies as having high or low methodological quality - Social anxiety was calculated from 2 studies, both of low quality; and for studies with acceptable levels of methodological quality, the Effect sizes for Functioning were .147 and for Mood just .084 - not medium (not even small), not representative and not significant (see Table below) - Any effect of CBT on these outcomes is a direct consequence of running low quality studies.

Hoffman et al also say "Wykes et al. (2008) examined controlled trials of CBT for schizophrenia and confirmed findings from previous meta-analyses (e.g., Gould et al. 2001; Rector and Beck 2001), suggesting that CBT had a small to medium effect size as compared to control conditions on both positive and negative symptoms" Again, as the table shows, the effect size for negative symptoms was only significant for low quality studies and not for high quality studies; and true, the effect size for positive symptoms was significant for high quality studies albeit small and barely significant at .22 (with a low end 95% CI of .016 meaning it could become nonsignificant with 1 or 2 more negative studies being reported)

Turning to Bird et al. (2010) and Alvarez-Jimenez et al. (2011), it is not explicitly stated in the article, but these two meta-analyses assess CBT for first episode patients - although they contrast with Wykes et al (2008) which did not examine first episode. Anyway, Alvarez-Himinez found "no statistically significant advantages in favor of CBT [and] CBT showed no clinical benefits on relapse rates compared with either supportive counseling or TAU." And Bird et al reported that "the present review failed to find any evidence that CBT reduced relapse rates in early psychosis, which suggests that the main benefits of this intervention are likely to be a reduction in symptom" Bird et al found no significant reduction in positive and negative symptoms at the end of the trials, but surprisingly found significant effects occur at up to 2 years after the trials finish - but the sample here is just 3 studies

Finally, returning to representativeness -Hoffman et al failed to mention at least three meta analyses published since Wykes (2008), which have been more explicitly negative about the effects of CBT on psychotic symptoms and relapse, including:

Lynch, Laws McKenna (2010) who said "CBT is no better than non-specific control interventions in the treatment of schizophrenia and does not reduce relapse rates."

Newton-Howes & Woods (2011) who said "Theoretically based CBT therapies, although proving effective, may not out perform more accessible and simpler forms of therapy for patients with schizophrenia in reducing psychopathology. Consideration of supportive therapy should be made for patients with psychotic mental disorder"

Jones et al (2012) most recent Cochrane meta analysis concludes "Trial-based evidence suggests no clear and convincing advantage for cognitive behavioural therapy over other - and sometime much less sophisticated - therapies for people with schizophrenia."

To conclude, I stand by my tweet that the review by Dr Hofmann and colleagues represents a biased view of the findings. Given the structured guidelines on how to write meta-analyses, the attention to publication bias, detailing the search process and providing reasons for excluding studies and so on - the raison d'etre for meta analysis would seem to be undermined by reviews of meta-analyses that do not adhere to the same or similar principles.