Satellites of Love: uninterpretable results

In llama land there's a one-man band
And he'll toot his flute for you
Come on fly with me, let's take off in the blue

Frank Sinatra (Come Fly with Me)
 

Cognitive Behavioral Therapy for negative symptoms in psychotic disorders: a pilot study

Staring, Hurne & van der Gaag (2013 in press)

 

Another month and another CBT for psychosis study that is uninterpretable. In previous months, I have commented here and on Twitter about CBT for psychosis studies often failing to use a control group and/or blinding of outcome assessments. As an example, I recently blogged on: Cognitive therapy for people with a schizophrenia spectrum diagnosis not taking antipsychotic medication: an exploratory trial by Morrison et al

Here we have another, in press at Journal of Behavior Therapy and Experimental Psychiatry. While Morrison et al referred to theirs as 'exploratory', Staring et al provide us with the ubiquitous .... 'pilot'

Satellite of Love by Lou Reed

 

 

What did Staring et al do?

In an open trial 21 adult outpatients with a schizophrenia spectrum disorder with negative symptoms received an average of 17,5 sessions of CBT-n (CBT for negative symptoms). At baseline and end-of-treatment, they assessed negative symptoms. They claim that "Intention-to-treat analyses showed a within group effect size of 1.26 on negative symptoms"

An effect size of 1.26 is, of course, massive - if only it meant something. As noted previously nonblind (open) studies are prone to large bias and without a control group, who knows why they changed. To be fair to the authors, they do have a limitations section - and it is quite large! Here is part of it:

"This was an uncontrolled study. Therefore the efficacy findings are biased. Patients may have improved over the course of the study by self-initiated change or because other treatments were helpful. The lack of a control group means that this effect was not controlled for."

"Second, also as a consequence of the uncontrolled design, measurements were not blind. We mostly used self-report measurements, and patients were fully aware that they had received an active treatment for their negative symptoms. This may have caused efficacy to be overestimated"

 

One thing we may be sure of is that the effect is overestimated...why? Because studies meeting acceptable quality i.e. RCTs that are blind and with controls suggest an effect size of at most .20 and nonsignificant (Wykes et al 2008) - Seems an overestimation is likely - by about 6 fold!

Anyway, my point here is that we need to have some monitoring of  studies that are essentially uninterpretable. To borrow a phrase from Lou Reed, I see them as "Satellites of Love" - autopilot research - probably done out of love - but ultimately clutter-up the science atmosphere.

The authors finish with:
"Randomized controlled trials with sufficient statistical power will need to be performed in order to confirm or refute our results"

This is somewhat odd because, as noted, many RCTs have already shown CBT to be ineffective for negative symptoms - as such... the study was refuted at conception

...All such satellites should engage auto-pilot and return to 'Llama land'

p.s. ask youself - would the study have been published if the results were negative?