In a Manner of Speaking

In a Manner of speaking
I just want to say
That I could never forget the way
You told me everything
By saying nothing

In a Manner of Speaking (by Tuxedo Moon)

 

I was recently asked to write an article on some of the problems associated with psychological research (e.g. publication bias, authors 'spinning' their results and so on). We often hear complaints from scientists that journalists put a false spin on their findings; however, I am interested in how authors present or spin their own findings.

Sometimes this spinning in a paper seems to starkly contrast with results of the paper. I can only conclude that some reviewers or editors suffer from what I would call the 'Daylight Robbery Syndrome' - from the adage that some burlgars are unlikey to be caught because they rob you so blatantly.  Here are my current top five quotes from papers on CBT for psychosis published in high impact journals ...For me, the quotes are the academic equivalent of someone kicking down your front door in broad daylight and then carrying off your 42 inch plasma TV screen under their arm past all of your friends and neighbours...

Therapeutic work is rife with vague statements, bias and misrepresentations; however, I am sure comparable examples exist in other areas and that readers may readily be able quote examples from their own research areas.

 

In a Manner of Speaking (covered by Nouvelle Vague)

1. " Interestingly, psychosocial treatments—such as cognitive behavioural therapy (CBT) and, more recently, arts therapies (music therapy, art therapy, and body movement or dance therapy)—have shown more promise than drug treatments in reducing negative symptoms and their impact, and the National Institute for Health and Clinical Excellence (NICE) has recently recommended these treatments."
 

The editorial goes on to discuss the largest, best and most recent study of art therapy for addressing the symptoms of people with schizophrenia (the so-called Matisse trial by Crawford et al 2012)

"The findings of the Matisse trial unfortunately suggest that art therapy, as currently practised in the UK, is unlikely to be of clinical benefit for people with negative or other symptoms of schizophrenia—a conclusion that the profession of art therapy will no doubt find unsettling. However, arts therapies, because they rely on creative expression rather than verbal communication, and some cognitive behavioural approaches, still have the greatest potential for success in the treatment of negative symptoms."

Kendall, T. (2012). Treating negative symptoms of schizophrenia. BMJ-British Medical Journal, 344(7847), 8.

In additon to the elephantine self-contradiction of the latter paragraph - two other things are wrong here. First, CBT does not significantly reduce negative symptoms - Wykes et al (2008) meta-analysis shows that studies with "acceptable" levels of quality find no significant impact of CBT on negative symptoms. Second arts therapy does not appear to reduce psychotic symptoms nor does it outperform medication in reducing negative symptoms. The fact is that no study has compared medication versus any of these therapies ...and so, no data even exist on the issue. 

 

Perhaps Editorials - unlike this one in the BMJ - really ought to be peer-reviewed!

 

2. Although we failed to show a statistically significant effect of the intervention we cannot rule out a beneficial effect of the cognitive therapy on transition rate (although it could be argued that the sample size required to show such an effect, and the small effect sizes reported here, would make such an endeavour unfeasible in practical terms and unwarranted in clinical terms).

Morrison AP, Stewart SL, French P, Bentall RP, Birchwood M, Byrne R, Davies LM, Fowler D, Gumley AI, Jones PB, Lewis SW, Murray GK, Patterson P, Dunn G. Early detection and intervention evaluation for people at risk of psychosis: multisite randomised controlled trial. British Medical Journal, 344

The failure to find a significant effect does not eliminate possibility of an effect! I have to ask, what could then eliminate the possibility of an effect...especially with a very small effect size and a huge sample required? Extremely torturous admission of CBT failure...or is it?

 

3. Patients receiving either CBT or supportive counselling in combination with usual treatment demonstrated better symptomatic recovery but no significant reduction in relapse compared with those receiving usual treatment alone... We suggest that the optimum psychosocial management of early schizophrenia would include a combination of CBT and family intervention.

Tarrier, N., Lewis, S., Haddock, G., Bentall, R., Drake, R., Kinderman, P., ... & Dunn, G. (2004). Cognitive-behavioural therapy in first-episode and early schizophrenia 18-month follow-up of a randomised controlled trial. The British Journal of Psychiatry, 184(3), 231-239.

How to ignore one whole strand of the main findings and also to make a nonsignificant result sound as if it were significant! CBT was no better than Supportive Counselling and yet the authors advocate CBT alongside family intervention - and the latter was not examined at all in this study...a strange conclusion-cocktail! 

4. We did not find significant between-group differences on symptom reduction, indicating no significant benefit of CBT over PE.” [psycho-education]...The use of an active rather than passive control intervention created a more stringent comparison for CBT, which may have further reduced power to detect the hypothesized changes”

Cather, C., Penn, D., Otto, M. W., Yovel, I., Mueser, K. T., & Goff, D. C. (2005). A pilot study of functional cognitive behavioral therapy (fCBT) for schizophrenia. Schizophrenia Research, 74(2), 201-209.

The finding of no significant benefit of CBT is blamed on a pesky control group - those cheeky controls reducing power! If only we hadn't used controls, our results, we would have had a significant result .. hey diddle diddle...
 

 

5. In the linear regression, faster resolution of symptoms in the groups allocated to either psychological treatment condition was seen, compared with routine care alone, but not at statistically significant levels...In summary, for auditory hallucinations, CBT is an improvement on routine care (but the effect is not statistically significant at the α =0.05 level)

Lewis, S., Tarrier, N., Haddock, G., Bentall, R., Kinderman, P., Kingdon, D., ... & Dunn, G. (2002). Randomised controlled trial of cognitive—behavioural therapy in early schizophrenia: acute-phase outcomes. The British Journal of Psychiatry, 181(43), s91-s97.

It doesn't mater how many times one says 'faster', 'better', improvement' - 'No'significant effect' means no significant effect!

Blind commitment to a theory is not an intellectual virtue: it is an intellectual crime.

Imre Lakatos (1970)

 

CBT: She's Lost Control(s) Again

 

And of a voice that told her when and where to act,
She said I've lost
control again

Joy Division (Shes Lost Control Again)

 

Cognitive therapy for people with a schizophrenia spectrum diagnosis not taking antipsychotic medication: an exploratory trial

Morrison, Hutton, Spencer, Barratt, Brabban, Callcott, Christodoulides, Dudley, French, Lunley, Tai & Turkington

 

This study, published recently in Psychological Medicine marks a departure from previous CBT for psychosis studies as it involves administering CBT to individuals who are unmedicated - and alongside the media attention (All in the Mind), it is important to look more closely at the study itself.

 

Morrison and 11 colleagues examined a small sample of 20 participants with schizophrenia spectrum disorders (actually 18 and one also didn't complete the therapy - so 17). All were outpatients - who had not been taking antipsychotic medication for at least 6 months. Morrison et al measured the impact of CBT on symptomatic outcome measures: Positive and Negative Syndromes Scale (PANSS), which was administered at baseline, 9 months (end of treatment) and 15 months (follow-up). Secondary outcomes were dimensions of hallucinations and delusions, self-rated recovery and social functioning. Rather than dismiss missing data, the authors chose to impute the missing data in the following analyses:

 

Their main findings were stated as:

"significant beneficial effects on all primary and secondary outcomes at end of treatment and follow-up, with the exception of self-rated recovery at end of treatment. Cohen’s d effect sizes were moderate to large [for PANSS total, d=0.85, 95% confidence interval (CI) 0.32–1.35 at end of treatment; d=1.26, 95% CI 0.66–1.84 at follow-up]."

The authors conclude that the

"study provides preliminary evidence that CT is an acceptable and effective treatment for people with psychosis who choose not to take antipsychotic medication. An adequately powered randomized controlled trial is warranted."

Now, I do think this is important because some individuals may see this study and attendant media as a basis to decide not to take antipsychotic medication. I have no qualms about personal choice when based on evidence - so let us examine the evidence.

 

Joy Division: Shes Lost Control

 

 

So, what does the study show? Well, the answer comes from the design of the study. It is a pretest - posttest design i.e. there is no control group. So, a within-group analysis with no control group. Without a control group of any description, we cannot know if any change is a generalised consequence of the added interaction (rather than anything about CBT itself). And without even a Treatment as Usual (TAU) control group, we cannot exclude the possibility that any change would have occurred regardless of therapy e.g. regression to the mean.

 

Second, the trial is an Open Trial i.e. the outcome measures are not made blind. All participants are evaluated by members of the team who were involved in the administering the trial. The lack of blinding is the biggest drawback to the evaluation of therapy and is well documented by all meta-analyses in this area (Lynch, Laws & McKenna 2010; Wykes et al 2008).

 

So, the study suffers from a host of threats to validity: a) regression to the mean i.e. extreme scores at start simply move toward the (lower) mean on second testing (with no control we cant estimate or eliminate this); b) Hawthorne effect i.e. any change is due to the special circumstances patients find themselves in (with no control, we cant eliminate this); c) lack of blinding i.e. those measuring outcome were involved in the study and aware that all patients had received CBT - the authors don't provide detail on this, but presumably some or all of the 8 different therapists administering the CBT to the participants and/or those who designed the study

 

Losing Control adds up!

 

To summarise - no control group and no blinding of outcome measures.

Can we see what happens in such designs with medicated patients? Well, below is a table from a meta-analysis by the CBT guru Aaron Beck  looking at effect size for pretes-posttest analyses in medicated patients (Rector & Beck 2002; so 'good' the journal republished it in 2012). Now Morrison et al found d=.85 for overall symptoms -Rector and Beck report a much larger mean effect size of d=1.31 for pre-post comparison (CBT-RC). So, the effect with unmedicated is substantially less - consider the size in some studies here - Pinto is almost three times that reported by Morrison et al!

 

 

This doesn't alter the fact that Morrison et al do report a substantial effect in unmedicated patients of .8+. However, the second thing to note about the table above is the comparison of CBT with a so-called active control (ST-RC) i.e. a condition that controls for the generalised impact of just interacting with another, receiving attention etc - here the controls are supportive therapy, befriending and so on. This shows that an effect size of .63 emerges in pretest postest designs for something as simple as befriending - not that much smaller than Morrison et al claim to attribute to CBT.

 

The final and key point though concerns the lack of blind evaluation. As Morrison et al note in their discussion:

"CBT for psychosis trials that attempt masking were reported to be associated with a reduction of effect sizes of nearly 60% (Wykes et al 2008)"

Actually, what Wykes et al say is

"There is a tendency for the unmasked studies to be overoptimistic about the effects of CBTp, with effect sizes of 50%–100% higher than those found in masked studies."

60% may be an average, but the reduction could range from 50-100%!

 

Crucially, the patients may not feel better. At the end of the study, the patients rated themselves as experiencing no recovery (pre and post); although they did report a minor improvement at the follow-up. In other words, the non-blind researchers perceived a much greater change than the patients themselves - approximately double the effect size!

 

To conclude: the use of a pretest-posttest design with no blinding means the authors are unable to draw conclusions about the efficacy of CBT in the study. The effect size is much smaller than in pretest-postest comparisons with medicated patients; of the .85 reported, how much is attributable to generic effects of interaction (and nothing to do with CBT) - well it could be .63; and if we assume a minimum of 60% of this may reflect lack of blinding - then what are we left with? An effect size of possibly less than .1 - in other words - (next to) nothing! And patients are not even convinced!

 

So, their conclusions that the "study provides preliminary evidence that CT is an acceptable and effective treatment for people with psychosis who choose not to take antipsychotic medication" seems unwarranted. Some might say, well its called 'exploratory' - but that is no excuse for a design that leaves the data uninterpretable and may even lead to some people changing their behaviour (withdrawing from their medication). I also understand that an RCT with blind assessment is being conducted - indeed, much of the press around this study has been about the uncompleted study - a bit cart before the horse! But at the moment - no data published exist to show that CBT reduces psychotic symptoms in unmedicated individuals

 

 

References

 

Lynch, D., Laws, K. R., & McKenna, P. J. (2010). Cognitive behavioural therapy for major psychiatric disorder: does it really work? A meta-analytical review of well-controlled trials. Psychological medicine, 40(01), 9-24.

 

Wykes, T., Steel, C., Everitt, B., & Tarrier, N. (2008). Cognitive behavior therapy for schizophrenia: effect sizes, clinical models, and methodological rigor. Schizophrenia Bulletin, 34(3), 523-537.